Data migration method and device in large data volume scene

Abstract

The embodiment of the invention provides a data migration method and device under a large data volume scene, and the method comprises the steps: determining a to-be-migrated index of target data in a distributed search engine and fragmentation setting of the to-be-migrated index, and determining a corresponding migration task data volume and a migration task priority according to the size of the to-be-migrated index and hotspot distribution in the fragmentation setting, determining a corresponding migration strategy according to the migration task data volume and the migration task priority; reading target data in the index to be migrated in batches according to a migration strategy, writing the target data into an immigration directory in a local file system, and inputting the real-time migration speed and the real-time resource utilization rate of the local file system into a set linear regression model, dynamically updating the data volume read during each migration operation, the maximum file number of each storage directory and the immigration directory according to real-time adjustment parameters output by the linear regression model; according to the method, the data migration efficiency and accuracy in a large data volume scene can be effectively improved.

Description

Technical Field

[0001] The present application relates to the field of data processing, and specifically to a data migration method and device in a large data volume scenario. Background Art

[0002] In the current era of big data, data migration has become a common and critical operation, especially in distributed search engines and large-scale data processing systems. However, existing data migration technologies still face many challenges and limitations when facing large data volume scenarios.

[0003] First, traditional data migration methods usually adopt a one-time batch migration strategy. This method works well when processing small-scale data, but often encounters performance bottlenecks and system resource exhaustion problems when facing large-scale data scenarios. One-time migration of large-scale data may cause excessive system load, affect normal business operations, and even cause system crashes.

[0004] Second, existing migration technologies often lack a deep understanding and utilization of data characteristics. In a distributed search engine, different indexes and shards may have different access frequencies and importance. Ignoring these characteristics may lead to unreasonable resource allocation during the migration process, affecting migration efficiency and system performance. Summary of the invention

[0005] In response to the problems in the prior art, the present application provides a data migration method and device in a large data volume scenario, which can effectively improve the data migration efficiency and accuracy in a large data volume scenario.

[0006] In order to solve at least one of the above problems, the present application provides the following technical solutions:

[0007] In a first aspect, the present application provides a data migration method in a large data volume scenario, comprising:

[0008] Determine the target data's to-be-migrated index and its sharding settings in the distributed search engine, determine the corresponding migration task data volume and migration task priority according to the size of the to-be-migrated index and the hotspot distribution in the sharding settings, and determine the corresponding migration strategy according to the migration task data volume and the migration task priority;

[0009] Determine the amount of data read in each migration operation and the maximum number of files in each storage directory according to the business requirements sent by the user and the storage performance of the local file system used to store the migration data, and determine the corresponding migration directory according to the amount of data and the maximum number of files;

[0010] Read the target data in the to-be-migrated index in batches according to the migration strategy, and write the target data into the migration-in directory in the local file system, input the real-time migration speed and the real-time local file system resource utilization rate into a set linear regression model, and dynamically update the amount of data read in each migration operation, the maximum number of files in each storage directory, and the migration-in directory according to the real-time adjustment parameters output by the linear regression model, wherein the linear regression model is trained by historical data migration data;

[0011] After the write operation is completed, multiple threads are started to delete the corresponding data in the distributed search engine, and the next batch of data is read after all threads complete the deletion operation.

[0012] Further, the step of determining the index to be migrated of the target data in the distributed search engine and its sharding setting, and determining the corresponding migration task data volume and migration task priority according to the size of the index to be migrated and the hotspot distribution in the sharding setting, includes:

[0013] Determine an index to be migrated in a distributed search engine where the target data to be migrated is located, wherein the index to be migrated includes at least one shard, and determine a shard setting of the shard according to a storage location and a data volume of each shard;

[0014] The corresponding hot spot distribution is determined according to the storage location and data size of each shard, the corresponding migration task data volume is determined according to the size of the index to be migrated, and the corresponding migration task priority is determined according to the hot spot distribution of the shard.

[0015] Further, the determining a corresponding migration strategy according to the migration task data volume and the migration task priority includes:

[0016] Dividing the data volume and priority of the migration task into different levels in advance and constructing a migration strategy matrix, wherein each migration strategy in the migration strategy matrix includes specific migration operation parameters;

[0017] A matching search is performed in a migration strategy matrix according to the migration task data volume and the migration task priority to determine a corresponding migration strategy.

[0018] Further, the method of determining the amount of data to be read in each migration operation and the maximum number of files in each storage directory according to the business requirements sent by the user and the storage performance of the local file system used to store the migration data includes:

[0019] The migration speed is calculated based on the expected completion time in the business requirements sent by the user and the available storage space of the local file system used to store the migration data.

[0020] The amount of data read in each migration operation and the maximum number of files in each storage directory are determined according to the maximum value of the migration speeds.

[0021] Further, determining the corresponding migration directory according to the data volume and the maximum number of files includes:

[0022] Determine the directory hierarchy and storage space distribution of the local file system;

[0023] A directory with sufficient available space and a small number of files is determined as a migration directory according to the data volume and the maximum number of files, and the migration data is written to multiple subdirectories under the directory in a dispersed manner.

[0024] Further, the reading the target data in the to-be-migrated index in batches according to the migration strategy and writing the target data into the migration directory in the local file system includes:

[0025] According to the migration strategy, the target data in the index to be migrated is read in batches using multiple threads;

[0026] Different sub-directories are created according to the category or time period to which the target data belongs, and the target data are scattered and written into different sub-directories of the migration directory.

[0027] Furthermore, the real-time migration speed and the real-time local file system resource utilization are input into a set linear regression model, and the amount of data read during each migration operation, the maximum number of files in each storage directory, and the migration directory are dynamically updated according to the real-time adjustment parameters output by the linear regression model, wherein the linear regression model is trained by historical data migration data, and includes:

[0028] Continuously monitor and collect real-time migration speed and local file system resource utilization data and input them into the pre-trained linear regression model;

[0029] The amount of data read during each migration operation, the maximum number of files in each storage directory and the migrated directory are dynamically updated according to the data read adjustment ratio output by the linear regression model, the change in the maximum number of files in the storage directory and the priority adjustment of the migrated directory.

[0030] In a second aspect, the present application provides a data migration device in a large data volume scenario, including:

[0031] A migration strategy determination module is used to determine the index to be migrated of the target data in the distributed search engine and its sharding settings, determine the corresponding migration task data volume and migration task priority according to the size of the index to be migrated and the hot spot distribution in the sharding settings, and determine the corresponding migration strategy according to the migration task data volume and the migration task priority;

[0032] A migration directory determination module is used to determine the amount of data read in each migration operation and the maximum number of files in each storage directory according to the business requirements sent by the user and the storage performance of the local file system used to store the migration data, and determine the corresponding migration directory according to the amount of data and the maximum number of files;

[0033] A migration parameter update module, used to read the target data in the to-be-migrated index in batches according to the migration strategy, and write the target data into the migration-in directory in the local file system, input the real-time migration speed and the real-time local file system resource utilization rate into a set linear regression model, and dynamically update the amount of data read in each migration operation, the maximum number of files in each storage directory, and the migration-in directory according to the real-time adjustment parameters output by the linear regression model, wherein the linear regression model is trained by historical data migration data;

[0034] The multi-threaded deletion module is used to start multiple threads to delete the corresponding data in the distributed search engine after the write operation is completed, and wait for all threads to complete the deletion operation before continuing to read the next batch of data.

[0035] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the data migration method in a large data volume scenario are implemented.

[0036] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the data migration method in the large data volume scenario.

[0037] In a fifth aspect, the present application provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the data migration method in a large data volume scenario.

[0038] It can be seen from the above technical scheme that the present application provides a data migration method and device in a large data volume scenario, by determining the index to be migrated of the target data in the distributed search engine and its sharding setting, determining the corresponding migration task data volume and migration task priority according to the size of the index to be migrated and the hot spot distribution in the sharding setting, and determining the corresponding migration strategy according to the migration task data volume and the migration task priority; reading the target data in the index to be migrated in batches according to the migration strategy, and writing the target data into the migration directory in the local file system, inputting the real-time migration speed and the real-time local file system resource utilization into the set linear regression model, and dynamically updating the data volume read during each migration operation, the maximum number of files in each storage directory, and the migration directory according to the real-time adjustment parameters output by the linear regression model, thereby effectively improving the data migration efficiency and accuracy in large data volume scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0040] Figure 1 This is one of the flow diagrams of the data migration method in a large data volume scenario in an embodiment of the present application;

[0041] Figure 2 This is a second flow chart of a data migration method in a large data volume scenario in an embodiment of the present application;

[0042] Figure 3 This is a third flow chart of a data migration method in a large data volume scenario in an embodiment of the present application;

[0043] Figure 4 This is a fourth flow chart of a data migration method in a large data volume scenario in an embodiment of the present application;

[0044] Figure 5 This is a fifth flow chart of a data migration method in a large data volume scenario in an embodiment of the present application;

[0045] Figure 6 This is a sixth flow chart of a data migration method in a large data volume scenario in an embodiment of the present application;

[0046] Figure 7 This is the seventh flow chart of the data migration method in a large data volume scenario in the embodiment of the present application;

[0047] Figure 8This is a structural diagram of a data migration device in a large data volume scenario in an embodiment of the present application;

[0048] Fig. 9 It is a schematic diagram of the structure of an electronic device in an embodiment of the present application.

[0049] Reference numerals:

[0050] Electronic device 9600, central processing unit 9100, memory 9140, communication module 9110, input unit 9120, audio processor 9130, display 9160, power supply 9170, buffer memory 9141, application / function storage unit 9142, data storage unit 9143, driver program storage unit 9144, antenna 9111, speaker 9131, microphone 9132. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0052] The acquisition, storage, use, and processing of data in the technical solution of this application comply with the relevant provisions of national laws and regulations.

[0053] In view of the problems existing in the prior art, the present application provides a data migration method and device in a large data volume scenario, by determining the index to be migrated of the target data in a distributed search engine and its sharding setting, determining the corresponding migration task data volume and migration task priority according to the size of the index to be migrated and the hot spot distribution in the sharding setting, and determining the corresponding migration strategy according to the migration task data volume and the migration task priority; reading the target data in the index to be migrated in batches according to the migration strategy, and writing the target data into the migration directory in the local file system, inputting the real-time migration speed and the real-time local file system resource utilization into a set linear regression model, and dynamically updating the data volume read in each migration operation, the maximum number of files in each storage directory, and the migration directory according to the real-time adjustment parameters output by the linear regression model, thereby improving the data migration efficiency and accuracy in large data volume scenarios.

[0054] In order to effectively improve the data migration efficiency and accuracy in large data volume scenarios, the present application provides an embodiment of a data migration method in large data volume scenarios. Figure 1 The data migration method in the large data volume scenario specifically includes the following contents:

[0055] Step S101: determining the index to be migrated of the target data in the distributed search engine and its sharding setting, determining the corresponding migration task data volume and migration task priority according to the size of the index to be migrated and the hot spot distribution in the sharding setting, and determining the corresponding migration strategy according to the migration task data volume and the migration task priority;

[0056] Optionally, in this embodiment, step S101 is the primary step of the data migration method in a large data volume scenario, which lays the foundation for the entire migration process. The core goal of this step is to determine the data to be migrated and formulate an appropriate migration strategy to ensure the efficiency and reliability of the migration process.

[0057] In the specific technical implementation process, you first need to determine the index of the target data to be migrated in the distributed search engine. This usually involves interaction with the search engine, and you may need to use specific APIs or query statements to obtain index information. Once the index to be migrated is determined, the next step is to analyze its sharding settings. Sharding is the basic unit of data distribution in a distributed system, and understanding sharding settings is crucial to developing an effective migration strategy.

[0058] Analyzing the sharding setup involves determining the size, location, and data distribution of each shard. This can be done by querying the metadata of a search engine or using specialized analysis tools. Special attention should be paid to the identification of hotspot distribution, which is usually achieved by analyzing the access frequency and data update rate of each shard. Hotspot shards often mean more frequent data access and higher business importance, so they need to be given priority during the migration process.

[0059] Based on the overall size of the index to be migrated and the amount of data in each shard, the system can calculate the total amount of data for the migration task. This amount of data directly affects the time and resources required for migration. At the same time, based on the distribution of hot spots identified previously, the system will assign a priority to each shard or index subset. There are many strategies for assigning priorities, such as sorting based on access frequency, manual definition based on business importance, or a combination of the two.

[0060] With the two key parameters of migration task data volume and priority, the next step is to determine the specific migration strategy. Migration strategy usually includes the order of migration, concurrency, amount of data for each migration, migration time window, etc. The formulation of the strategy needs to consider multiple factors, such as available computing resources, network bandwidth, performance characteristics of the source system and the target system, etc. A common practice is to predefine a series of policy templates and then select the most suitable strategy based on the characteristics of the current task.

[0061] This step solves several key technical problems. First, it effectively identifies the key points and difficulties of data migration by analyzing the sharding settings and hotspot distribution. This solves the resource allocation problem in large-scale data migration and ensures that important data is given priority. Second, by formulating targeted migration strategies, it solves the problem of how to maximize migration efficiency while ensuring business continuity. Finally, this step also provides a basis for subsequent dynamic adjustments, allowing the entire migration process to respond flexibly according to actual conditions.

[0062] The technical effect achieved by this step is significant. First, it realizes the intelligent planning of migration tasks, ensuring that important data and hot data can be processed in a timely and prioritized manner. Second, by formulating appropriate migration strategies, it can maximize the use of available resources and improve migration efficiency while ensuring data integrity and consistency. Third, this strategy formulation method based on data characteristics and system status greatly improves the reliability and robustness of the migration process and reduces the risk of failure caused by resource competition or system overload.

[0063] This example uses a simplified example to illustrate the application of this step. Assume that we need to migrate the product data index of an e-commerce platform. The index contains 10 million product records, distributed on 10 shards. Through analysis, we find that 3 of the shards contain data on hot-selling products, and the access frequency is 5 times that of other shards.

[0064] In this example, the system first determined that the total size of the index to be migrated was approximately 100GB. Then, by analyzing the access logs of each shard, three hot shards were identified. Based on this information, the system assigned the highest migration priority to the hot shards, and the remaining shards were sorted by data size.

[0065] Considering the total data volume and priority distribution, the system selected a concurrent migration strategy: for hot shards, 3 concurrent threads are used for migration, each time 1GB of data is migrated; for non-hot shards, 2 concurrent threads are used, each time 2GB of data is migrated. In addition, the strategy also stipulates that the concurrency and the amount of data migrated at a time should be increased during off-peak business hours (such as 2 am to 6 am).

[0066] The formulation of this strategy fully considers the importance of data, the load capacity of the system, and the time characteristics of the business, ensuring the rapid migration of hot data while avoiding excessive interference with normal business. In this way, the system can significantly improve migration efficiency and reduce overall migration time while ensuring data availability.

[0067] In general, step S101 develops an intelligent, efficient and reliable execution plan for the entire migration process by deeply analyzing the data characteristics and system status. This not only improves the efficiency and reliability of the migration, but also lays a solid foundation for the smooth progress of subsequent steps. In the scenario of large data volume, this intelligent planning method based on data characteristics and system status plays a key role in ensuring the smooth progress and ultimate successful completion of the migration process.

[0068] Step S102: determining the amount of data to be read in each migration operation and the maximum number of files in each storage directory according to the business requirements sent by the user and the storage performance of the local file system used to store the migration data, and determining the corresponding migration directory according to the amount of data and the maximum number of files;

[0069] Optionally, in this embodiment, step S102 is a key step in the data migration method in a large data volume scenario, which mainly focuses on how to optimize data reading and storage during the migration process according to actual conditions. The core goal of this step is to fully utilize the storage performance of the local file system while meeting user business needs to achieve efficient data migration.

[0070] In the specific technical implementation process, the business requirements sent by users need to be parsed first. These requirements usually include the expected migration completion time, the acceptable business impact, data availability requirements, etc. The system needs to convert these high-level requirements into specific technical parameters, such as the amount of data that can be migrated per second, the proportion of system resources that can be occupied, etc. This process may involve complex calculations and trade-offs, and it is necessary to consider the interaction between multiple factors.

[0071] At the same time, the system needs to evaluate the performance of the local file system used to store the migrated data. This includes disk read and write speed, IOPS (input / output operations per second), available storage space, and file system characteristics (such as the maximum number of files supported, directory depth, etc.). This information can be obtained through system calls, specialized performance testing tools, or historical performance data analysis.

[0072] Based on the analysis of business needs and storage performance, the system can calculate the amount of data that is suitable for reading during each migration operation. The determination of this amount of data requires balancing multiple factors: on the one hand, a larger amount of data can reduce the number of I / O operations and improve overall efficiency; on the other hand, too large an amount of data may increase memory pressure and affect system stability. Usually, the system will set an initial value, and then dynamically adjust it according to actual conditions during the subsequent migration process.

[0073] At the same time, the system also needs to determine the maximum number of files in each storage directory. The setting of this parameter mainly considers the performance characteristics of the file system. In most file systems, when there are too many files in a single directory, the efficiency of file search, read and write operations will drop significantly. Therefore, setting the maximum number of files reasonably can avoid performance bottlenecks and maintain efficient file operations.

[0074] With the two key parameters of data volume and maximum number of files, the next step is to determine the appropriate directory to migrate to. This process needs to consider factors such as the directory structure of the local file system, the available space of each directory, the current number of files, etc. The system usually selects a directory with sufficient available space and a small number of current files as the migration directory. If there is no existing directory that meets the conditions, the system may create a new directory structure.

[0075] This step solves several key technical problems. First, it solves the problem of how to convert high-level business requirements into specific technical parameters so that the migration process can truly meet user needs. Second, it solves the problem of how to ensure migration efficiency while avoiding excessive pressure on the system by reasonably setting data read volume and file number limits. Finally, by intelligently selecting the migration directory, it solves the problem of how to maximize the use of storage resources and avoid performance bottlenecks of a single directory.

[0076] This step achieves many technical effects. First, it implements intelligent settings for migration parameters, ensuring that the migration process can maximize the use of system resources while meeting business needs. Second, by properly controlling the amount of data read each time and the number of files in each directory, it effectively balances the migration speed and system load, improving overall migration efficiency. Third, by intelligently selecting the migration directory, it optimizes the distribution of data in the local file system, avoids possible performance bottlenecks, and lays a good foundation for subsequent data access and management.

[0077] This embodiment uses a simplified embodiment to illustrate the application of this step. Assume that we are migrating product catalog data for a medium-sized e-commerce company. The user requires that the migration be completed within 48 hours, and the system response time during the migration process must not exceed 120% of the normal level. The local file system uses a high-performance SSD array with a total capacity of 10TB and currently 5TB of available space.

[0078] In this example, the system first calculates the average amount of data that needs to be migrated per hour based on user needs. Assuming the total data volume is 4TB, then an average of about 83GB of data needs to be migrated per hour. Considering the system performance requirements, the system decides to set the data volume for each migration operation to 2GB. This value can ensure sufficient migration speed without causing excessive pressure on the system.

[0079] Regarding the maximum number of files in a storage directory, the system has found through testing that when the number of files in a single directory exceeds 10,000, the efficiency of file operations will drop significantly. Therefore, the system sets the maximum number of files in each storage directory to 8,000, leaving some room for future growth.

[0080] When determining the directory to migrate to, the system scanned the existing directory structure and found a directory called "product_catalog" with multiple sub-directories corresponding to different product categories. The system decided to create new sub-directories under this directory to store the migrated data. For example, for products in the "electronics" category, the system might create the following directory structure: " / product_catalog / electronics / 2023 / Q2 / " and store the relevant migration data under this directory.

[0081] In this way, the system not only ensures that the migration process can be completed within the specified time, but also ensures the reasonable distribution of data in the local file system, avoiding performance problems caused by too many files in a single directory. At the same time, this directory structure also facilitates subsequent data management and query.

[0082] In general, step S102 sets reasonable parameters and storage strategies for the data migration process by intelligently analyzing user needs and system performance. This method not only improves the efficiency and reliability of migration, but also lays a good foundation for subsequent data management and access. In large data volume scenarios, this parameter optimization and storage planning method based on actual conditions plays a key role in ensuring the smooth progress and ultimate successful completion of the migration process. It demonstrates how to balance multiple requirements in a complex technical environment to achieve efficient and reliable data migration.

[0083] Step S103: Read the target data in the to-be-migrated index in batches according to the migration strategy, and write the target data into the migration-in directory in the local file system, input the real-time migration speed and the real-time local file system resource utilization rate into a set linear regression model, and dynamically update the amount of data read in each migration operation, the maximum number of files in each storage directory, and the migration-in directory according to the real-time adjustment parameters output by the linear regression model, wherein the linear regression model is trained by historical data migration data;

[0084] Optionally, in this embodiment, step S103 is the core execution step in the data migration method in a large data volume scenario, which not only realizes the actual migration of data, but also introduces a dynamic adjustment mechanism to adapt to various situations that may occur during the migration process. The core goal of this step is to achieve adaptive optimization of the migration process while ensuring data migration efficiency and system stability.

[0085] In the specific technical implementation process, first, according to the previously formulated migration strategy, the system starts to read the target data in the index to be migrated in batches. This process usually involves interaction with the source database or search engine, and may require the use of specific APIs or query statements to obtain data. Data is usually read in paging or cursors to ensure that large amounts of data can be processed without running out of memory.

[0086] The read data is then written to the designated migration directory of the local file system. The writing process needs to consider factors such as data format conversion and compression to optimize storage space and subsequent reading efficiency. At the same time, the system needs to maintain metadata information, recording the source, target location, timestamp and other information of each batch of migrated data to facilitate subsequent verification and rollback operations.

[0087] While data is being migrated, the system monitors and records the migration speed and resource utilization of the local file system in real time. The migration speed can be measured by calculating the amount of data successfully migrated per unit time. File system resource utilization includes indicators such as disk usage, I / O load, and number of files. These real-time data are fed into a pre-set linear regression model as input.

[0088] The linear regression model is a statistical learning method used to analyze the linear relationship between independent variables (here, migration speed and resource utilization) and dependent variables (adjustment parameters). This model is trained with historical data migration data, and it can predict the optimal adjustment parameters based on the current migration situation. The model training process involves the collection, cleaning, and analysis of a large amount of historical data, which may need to be completed using machine learning frameworks such as TensorFlow or PyTorch.

[0089] Based on the real-time adjustment parameters output by the linear regression model, the system dynamically updates three key parameters: the amount of data read during each migration operation, the maximum number of files per storage directory, and the directory to migrate to. This dynamic adjustment mechanism enables the system to adaptively optimize the migration process based on the current operating conditions. For example, if the system load is detected to be low, the amount of data read each time can be increased; if the number of files in a directory is close to the upper limit, the maximum number of files can be adjusted or a new directory to migrate to can be selected.

[0090] This step solves several key technical problems. First, it solves the problem of how to maintain high efficiency and system stability during large-scale data migration. By reading and writing in batches and dynamically adjusting parameters, the system can maintain good performance under different load conditions. Second, it solves the problem of how to optimize the migration process in real time. By introducing a linear regression model, the system can make intelligent decisions based on the current situation instead of relying on preset static rules. Finally, it also solves the problem of how to use historical experience to improve the current migration process. By training the model, the system can learn from past migration tasks and apply them to the current task.

[0091] The technical effect achieved by this step is significant. First, it realizes adaptive optimization of the migration process, allowing the system to maintain high efficiency in a variety of different situations. Second, through real-time monitoring and dynamic adjustment, it significantly improves the stability and reliability of the migration process and reduces the risk of failure due to system overload or resource exhaustion. Third, by utilizing machine learning models, it enables the system to continuously learn and improve, and the migration efficiency will gradually increase over time.

[0092] This example uses a simplified example to illustrate the application of this step. Assume that we are migrating course data and student records for a large online education platform. The initial setting is to read 1GB of data each time, store a maximum of 10,000 files in each storage directory, and the migration directories are " / data / courses / " and " / data / students / ".

[0093] During the migration process, the system continuously monitors the migration speed and resource utilization. Assume that at a certain point in time, the system detects that the migration speed is 100MB / s, the disk utilization is 60%, and the I / O load is 40%. These data are input into the linear regression model.

[0094] After analyzing these inputs, the model outputs a set of tuning parameters. For example, the model may recommend increasing the amount of data read each time to 1.2GB because the current system load is low and can handle more data. At the same time, the model may recommend increasing the maximum number of files in the " / data / courses / " directory to 12,000 because the files in this directory are growing faster than expected.

[0095] The system dynamically adjusts parameters based on these suggestions. It increases the amount of data read each time to 1.2GB and updates the maximum number of files in the " / data / courses / " directory to 12,000. At the same time, the system may decide to create a new migration directory " / data / courses_2023 / " to distribute storage pressure.

[0096] This dynamic adjustment process continues, allowing the system to continuously optimize the migration process based on actual conditions. For example, if the system subsequently detects that the I / O load begins to increase, it may reduce the amount of data read each time to avoid affecting other system operations.

[0097] In general, step S103 realizes an intelligent and adaptive data migration process by combining real-time monitoring, machine learning models and dynamic adjustment mechanisms. This method not only improves the efficiency and reliability of migration, but also demonstrates how to apply advanced technologies to solve practical problems in complex big data environments. It reflects the transformation of data migration from static planning to dynamic optimization, and provides an effective method for handling large-scale and complex data migration tasks. In this way, the system can maintain efficient and stable operation in a variety of environments and conditions, greatly improving the success rate and efficiency of large-scale data migration.

[0098] Step S104: after the write operation is completed, multiple threads are started to perform a deletion operation on the corresponding data in the distributed search engine, and the next batch of data is read after all threads complete the deletion operation.

[0099] Optionally, in this embodiment, step S104 is a key step in the data migration method in a large data volume scenario, which mainly handles the cleanup work after data migration to ensure that the data in the source system can be deleted in a timely and efficient manner, thereby maintaining data consistency and system cleanliness. The core goal of this step is to complete the deletion operation as quickly as possible while ensuring data security so as not to affect the overall migration progress.

[0100] In the specific implementation process, the system first needs to confirm that the write operation has been completely successful. This usually involves verifying the write results, which may include file size checks, verification and comparison methods. After confirming that the write is successful, the system immediately starts the multi-threaded deletion operation. The use of multi-threading technology is to improve the deletion efficiency, especially when dealing with a large number of small files or data in a distributed system.

[0101] The implementation of multi-threaded deletion operations usually involves the creation and management of thread pools. The system dynamically determines the optimal number of threads based on the currently available computing resources and network bandwidth. Each thread is responsible for deleting a portion of data, and the allocation of this data can be based on a hash algorithm or a simple sequence number allocation.

[0102] When performing deletion operations in a distributed search engine, the distribution characteristics of the data need to be taken into account. Usually, data is stored in multiple nodes, so the deletion operation needs to coordinate the behavior of multiple nodes. This may involve distributed transactions or two-phase commit protocols to ensure the atomicity and consistency of the deletion operation.

[0103] The execution of the deletion operation is usually completed by calling the API provided by the distributed search engine. These APIs may support batch deletion, thereby reducing the number of network interactions and improving efficiency. During the deletion process, the system needs to handle various possible abnormal situations, such as network interruption, node failure, etc., and implement corresponding retry and recovery mechanisms.

[0104] The system continuously monitors the execution status of each thread, including the amount of data deleted, the deletion speed, the number of failures, etc. This information can be used to dynamically adjust the deletion strategy, such as increasing or decreasing the number of threads, adjusting the batch size, etc.

[0105] After all threads have completed the deletion operation, the system will perform a comprehensive verification to ensure that all data marked for deletion has indeed been removed from the distributed search engine. This verification process may involve sampling queries or full scans of the search engine.

[0106] Only after confirming that all deletion operations have been successfully completed will the system continue to read the next batch of data and start a new round of migration. This strict synchronization mechanism ensures data consistency and avoids duplicate migration or missing data.

[0107] This step solves several key technical problems. First, it solves the problem of how to efficiently clean up source data during large-scale data migration. Through multi-threaded parallel deletion, the system can complete the deletion of large amounts of data in a relatively short period of time. Second, it solves the problem of how to ensure data consistency. Through strict write confirmation and deletion verification mechanisms, data synchronization between the source system and the target system is ensured. Finally, it also solves the problem of how to coordinate complex operations in a distributed environment, and achieves efficient distributed deletion through reasonable task allocation and status monitoring.

[0108] This step achieves many technical effects. First, it implements real-time cleanup during data migration, avoiding long-term accumulation of data in the source system, reducing storage pressure and potential data inconsistency risks. Second, through multi-threaded parallel processing, it significantly improves the efficiency of deletion operations and reduces the overall migration time. Third, the strict synchronization mechanism ensures the integrity and consistency of the data and improves the reliability of the migration process.

[0109] From the above description, it can be seen that the data migration method in a large data volume scenario provided by the embodiment of the present application can determine the index to be migrated of the target data in the distributed search engine and its sharding setting, determine the corresponding migration task data volume and migration task priority according to the size of the index to be migrated and the hot spot distribution in the sharding setting, and determine the corresponding migration strategy according to the migration task data volume and the migration task priority; read the target data in the index to be migrated in batches according to the migration strategy, and write the target data into the migration directory in the local file system, input the real-time migration speed and the real-time local file system resource utilization into the set linear regression model, and dynamically update the data volume read during each migration operation, the maximum number of files in each storage directory, and the migration directory according to the real-time adjustment parameters output by the linear regression model, thereby improving the data migration efficiency and accuracy in large data volume scenarios.

[0110] In one embodiment of the data migration method in the large data volume scenario of the present application, see Figure 2 , and can also include the following:

[0111] Step S201: determining an index to be migrated in a distributed search engine where target data to be migrated is located, wherein the index to be migrated includes at least one shard, and determining a shard setting of the shard according to a storage location and a data volume of each shard;

[0112] Step S202: determining the corresponding hotspot distribution according to the storage location and data size of each of the shards, determining the corresponding migration task data volume according to the size of the index to be migrated, and determining the corresponding migration task priority according to the hotspot distribution of the shards.

[0113] Optionally, in this embodiment, steps S201 and S202 are key preparation stages in the data migration method in a large data volume scenario, and together they constitute the process of formulating a migration strategy. The main goal of these two steps is to conduct a comprehensive analysis and evaluation of the data to be migrated before starting the actual data migration, so as to formulate the optimal migration strategy.

[0114] In step S201, it is first necessary to determine the specific index in the distributed search engine where the target data to be migrated is located. This usually involves communication with the source system administrator and querying the system metadata. After determining the index to be migrated, the system needs to deeply analyze the internal structure of the index, especially its sharding. In a distributed search engine, an index is usually divided into multiple shards to achieve distributed storage and parallel processing of data.

[0115] For each shard, the system needs to determine two key pieces of information: storage location and data size. Storage location refers to the physical location of the shard in the cluster, which may involve specific server nodes or storage devices. The data size is directly related to the difficulty and time required to migrate the shard. This information can be obtained by querying the management API of the distributed search engine or directly accessing the underlying storage system.

[0116] Based on the information obtained, the system needs to determine the corresponding shard settings for each shard. These settings may include migration priority, migration method (such as overall migration or incremental migration), target storage location, etc. The determination of shard settings needs to consider multiple factors, such as data volume, current storage location load, network topology, etc.

[0117] Step S202 further deepens the analysis of the migrated data. First, the system needs to determine the corresponding hotspot distribution according to the storage location and data size of each shard obtained in step S201. Hotspot distribution reflects the frequency and mode of data access and is an important factor in determining the migration strategy.

[0118] Determining hotspot distribution usually involves analyzing system access logs, and may require the use of big data analysis tools such as Hadoop or Spark to process massive amounts of log data. The analysis results may show that some shards are accessed significantly more frequently than other shards, and these are the so-called "hot" shards.

[0119] Next, the system needs to determine the corresponding migration task data volume based on the overall size of the index to be migrated. This step involves the evaluation of system resources, including network bandwidth, storage capacity, processing power, etc. The determination of the migration task data volume requires finding a balance between migration efficiency and system stability.

[0120] Finally, based on the previously determined shard hotspot distribution, the system assigns a corresponding priority to each migration task. Usually, hot shards are given a higher migration priority to ensure that the most frequently accessed data can be migrated as quickly as possible, thereby minimizing the impact on the business.

[0121] These two steps solve several key technical problems. First, they solve the problem of how to fully understand and evaluate the data to be migrated in a complex distributed environment. By analyzing the sharding situation and hotspot distribution in detail, the system can have a comprehensive understanding of the complexity of the migration task. Second, they solve the problem of how to formulate the optimal migration strategy. By considering multiple factors such as data volume, storage location, access mode, etc., the system can formulate personalized strategies for each shard and migration task. Finally, they also solve the problem of how to maximize migration efficiency while ensuring system stability, and achieve optimal resource utilization by reasonably allocating the data volume and priority of migration tasks.

[0122] The technical effects achieved by these two steps are significant. First, they realize the intelligence and personalization of the migration process, enabling the system to formulate the most suitable migration strategy according to the actual situation. Second, by giving priority to hot data, the impact of the migration process on normal business is greatly reduced. Third, through reasonable task division and priority allocation, the overall migration efficiency is improved and the time required for migration is shortened.

[0123] In one embodiment of the data migration method in the large data volume scenario of the present application, see Figure 3 , and can also include the following:

[0124] Step S301: pre-dividing the data volume and priority of the migration task into different levels and constructing a migration strategy matrix, wherein each migration strategy in the migration strategy matrix includes specific migration operation parameters;

[0125] Step S302: performing a match search in a migration strategy matrix according to the migration task data volume and the migration task priority, and determining a corresponding migration strategy.

[0126] Optionally, in this embodiment, steps S301 and S302 are key strategy formulation stages in the data migration method in large data volume scenarios, and together they constitute an intelligent and automated migration strategy selection mechanism. The main goal of these two steps is to quickly and accurately select the most suitable migration strategy in complex and changeable data migration scenarios, thereby improving migration efficiency and reliability.

[0127] In step S301, the system first needs to quantify and classify the data volume and priority of the migration task. This process needs to take into account the actual situation of the system and historical experience. For example, the data volume may be divided into small (<100GB), medium (100GB-1TB), large (1TB-10TB), and extra large (>10TB). The priority may be divided into low, medium, high, and urgent levels. This division should be detailed enough to cover various possible situations, but not too complicated to be difficult to manage.

[0128] After the division is completed, the system needs to build a migration strategy matrix. This matrix is ​​essentially a multidimensional table, in which each cell represents a specific combination of data volume and priority, corresponding to a specific migration strategy. Each migration strategy contains a series of specific migration operation parameters, which may include:

[0129] Concurrency: The number of threads or processes performing migration simultaneously.

[0130] Batch size: The amount of data read and written each time.

[0131] Network bandwidth restriction: To avoid affecting other services, you may need to limit the network usage of the migration task.

[0132] Error retry policy: The number and interval of retries when encountering network errors or other problems.

[0133] Data consistency check frequency: the frequency of data verification during the migration process.

[0134] Resource allocation strategy: such as CPU usage limit, memory allocation, etc.

[0135] The specific values ​​of these parameters should be determined based on the system's historical experience, performance test results, and theoretical analysis. For example, for tasks with large data volumes but low priority, lower concurrency and larger batch sizes may be selected to reduce the occupation of system resources; while for tasks with small data volumes but high priority, higher concurrency and smaller batch sizes may be selected to speed up the migration.

[0136] Step S302 uses the migration strategy matrix constructed in step S301 in the actual migration process. When the system needs to process a specific migration task, it will first search in the migration strategy matrix based on the data volume and priority of the task. This process is essentially a multi-dimensional index operation, which can be implemented through data structures such as hash tables or decision trees to ensure fast search efficiency.

[0137] Once a matching migration strategy is found, the system will obtain all migration operation parameters defined in the strategy. These parameters will be used to configure specific migration tasks and guide the execution of the entire migration process.

[0138] These two steps solve several key technical problems. First, they solve the problem of how to quickly select the most suitable migration strategy in complex and changing migration scenarios. By pre-building a policy matrix, the system can make decisions quickly at runtime without the need for complex calculations each time. Second, they solve the problem of how to effectively integrate expert experience and historical data into the automated decision-making process. Through a carefully designed policy matrix, the system is able to encode the knowledge and experience of human experts into the automated process. Finally, they also solve the problem of how to balance system stability and other business needs while ensuring migration efficiency, and achieve refined management of the migration process through fine-grained parameter control.

[0139] The technical effects achieved by these two steps are significant. First, they realize the intelligent and automated selection of migration strategies, greatly reducing the need for manual intervention and improving the efficiency of the entire migration process. Second, through the pre-defined strategy matrix, the consistency and predictability of the migration strategy are ensured, reducing problems caused by human misjudgment. Third, through fine-grained parameter control, the system can better balance migration efficiency and system stability, maximizing resource utilization while minimizing the impact on other businesses.

[0140] In one embodiment of the data migration method in the large data volume scenario of the present application, see Figure 4 , and can also include the following:

[0141] Step S401: calculating the migration speed during the migration operation according to the expected completion time in the business requirements sent by the user and the available storage space of the local file system for storing the migration data;

[0142] Step S402: determining the amount of data to be read in each migration operation and the maximum number of files in each storage directory according to the maximum value of the migration speeds.

[0143] Optionally, in this embodiment, steps S401 and S402 constitute a key performance optimization stage in the data migration method in a large data volume scenario. The main goal of these two steps is to dynamically calculate and adjust key parameters of the migration operation under the premise of considering user business requirements and system resource limitations to achieve optimal migration performance.

[0144] In step S401, the system first needs to obtain two key inputs: the expected completion time specified by the user and the available storage space of the local file system for storing the migrated data. The expected completion time is usually provided by the user based on business needs, which reflects the urgency of data migration. The available storage space needs to be obtained through system calls or file system APIs.

[0145] After obtaining these two inputs, the system needs to calculate the migration speed during the migration operation. This calculation process needs to consider multiple factors:

[0146] Total data volume: This can be obtained through the previous steps (such as S201 and S202).

[0147] Expected Completion Time: A time constraint provided by the user.

[0148] Available storage space: The limits of the local file system.

[0149] Network bandwidth: The network conditions between the source and target systems.

[0150] System processing capacity: including the limitations of hardware resources such as CPU and memory.

[0151] A simple way to calculate the migration speed is to divide the total data volume by the expected completion time to get an ideal migration speed. The system then needs to consider other factors that constrain this ideal speed. For example, if there is insufficient available storage space, the system may need to reduce the migration speed to avoid storage overflow. Similarly, if network bandwidth or system processing power becomes a bottleneck, the migration speed needs to be adjusted accordingly.

[0152] This calculation process may involve complex performance models and optimization algorithms. For example, the system may use a machine learning model to predict the actual migration speed under different parameter settings based on historical migration data. Or, the system may use a heuristic algorithm to find the optimal migration speed that meets all constraints through multiple iterations.

[0153] Step S402 is to determine specific operation parameters based on the migration speed calculated in step S401. The key here is to determine two important parameters: the amount of data read during each migration operation and the maximum number of files in each storage directory.

[0154] When determining the amount of data to read each time, the system needs to find a balance between efficiency and stability. A larger read amount can reduce the number of I / O operations and improve efficiency, but it may also increase memory pressure and affect system stability. Generally speaking, this value can be set to a small fraction of the migration speed, such as 1 / 10 or 1 / 100 of the migration speed per second.

[0155] The maximum number of files per storage directory is determined to avoid file system performance degradation. Most file systems will significantly degrade performance when a single directory contains a large number of files, so it is necessary to limit the number of files per directory. The setting of this value needs to take into account the characteristics of the target file system, and the optimal value can usually be determined through performance testing.

[0156] These two steps solve several key technical problems. First, they solve the problem of how to dynamically determine the optimal migration speed while considering multiple constraints. By comprehensively considering user needs, system resources, and performance characteristics, the system is able to find a balance that not only meets user time requirements but also does not over-consume system resources. Second, they solve the problem of how to translate high-level performance goals (such as migration speed) into specific operating parameters. By scientifically setting limits on the amount of data to be read and the number of files, the system is able to achieve the expected performance goals at the practical operational level. Finally, they also solve the problem of how to avoid common performance bottlenecks (such as file system performance degradation) during the migration process.

[0157] The technical effects achieved by these two steps are significant. First, they achieve dynamic optimization of migration performance, enabling the system to adaptively adjust migration parameters according to actual conditions and maximize the use of available resources. Second, by precisely controlling the amount of data and the number of files read, the system can avoid excessive pressure on the source and target systems while ensuring performance, thus improving the stability of the entire migration process. Third, this approach can better meet users' time requirements and improve user satisfaction.

[0158] In one embodiment of the data migration method in the large data volume scenario of the present application, see Figure 5 , and can also include the following:

[0159] Step S501: Determine the directory hierarchy and storage space distribution of the local file system;

[0160] Step S502: determining a directory with sufficient available space and a small number of files as a migration directory according to the data volume and the maximum number of files, and writing the migration data to multiple subdirectories under the directory in a dispersed manner.

[0161] Optionally, in this embodiment, steps S501 and S502 constitute the key storage optimization stage in the data migration method in a large data volume scenario. The main goal of these two steps is to reasonably select and utilize storage space under the premise of considering the characteristics and limitations of the local file system to achieve efficient and balanced data storage, thereby improving the overall migration performance and system stability.

[0162] In step S501, the system first needs to conduct a comprehensive analysis and evaluation of the local file system. This process involves multiple aspects of investigation:

[0163] Directory hierarchy: The system needs to traverse the entire file system and build a complete directory tree. This process may involve recursive algorithms and needs to take into account the situation of deeply nested directories.

[0164] Storage space distribution: The system needs to count the used and available space of each directory and its subdirectories. This includes not only the size of files directly stored in the directory, but also the recursive statistics of subdirectories.

[0165] File count statistics: In addition to space usage, the system also needs to count the number of files in each directory. This is because the performance of some file systems will drop significantly when a single directory contains a large number of files.

[0166] I / O performance evaluation: The system may need to perform I / O performance tests on different directories because storage performance may vary in different locations in some storage systems.

[0167] File system feature analysis: The system needs to understand the characteristics of the target file system, such as whether sparse files are supported and whether there is a single file size limit.

[0168] This analysis process may consume a certain amount of time and system resources, especially on large file systems. Therefore, the system may need to adopt some optimization strategies, such as parallel scanning, incremental updates, etc., to improve analysis efficiency.

[0169] Step S502 is based on the analysis results of S501, combined with the data volume and maximum number of files determined in the previous step (such as S402), to select the most suitable migration directory and plan the data storage method. This step involves a complex decision-making process:

[0170] Directory selection: The system needs to select the most suitable directory from all available directories as the migration directory. This selection needs to take into account factors such as available space, current number of files, I / O performance, etc. The system may use a weighted scoring system to sort the directories.

[0171] Subdirectory creation: Once the migration directory is selected, the system needs to create multiple subdirectories under it to disperse the migration data. The number and structure of subdirectories need to be determined based on the total data volume and the maximum number of files. For example, if the total data volume is 10TB and the maximum number of files in each directory is 1000, then the system may need to create up to 10,000 subdirectories.

[0172] Data distribution strategy: The system needs to design a strategy to decide how to write the migrated data to these subdirectories in a dispersed manner. This strategy needs to take into account factors such as load balancing and query efficiency. Common strategies include hash distribution and range partitioning.

[0173] Dynamic adjustment: During the migration process, the system may need to dynamically adjust the storage policy. For example, if the write speed of a subdirectory is significantly slower than that of other directories, the system may need to reduce the write speed to this directory and use other directories with better performance instead.

[0174] These two steps solve several key technical problems. First, they solve the problem of how to find the optimal storage location in a complex file system. Through comprehensive analysis and intelligent decision-making, the system can make full use of available storage resources and avoid excessive local storage pressure. Second, they solve the problem of how to avoid file system bottlenecks while ensuring performance. Through reasonable directory structure design and decentralized data storage, the system can avoid performance degradation caused by too many files in a single directory. Finally, they also solve the problem of how to implement a flexible and scalable storage strategy so that the system can adapt to data migration needs of different sizes and types.

[0175] The technical effects achieved by these two steps are significant. First, they achieve optimal utilization of storage resources and improve overall storage efficiency. Second, through reasonable data distribution, the system can achieve better load balancing and avoid storage hotspots, thereby improving overall I / O performance. Third, this approach improves the scalability and flexibility of the system, enabling it to adapt to various complex data migration scenarios.

[0176] In one embodiment of the data migration method in the large data volume scenario of the present application, see Figure 6 , and can also include the following:

[0177] Step S601: Read the target data in the to-be-migrated index in batches using multiple threads according to the migration strategy;

[0178] Step S602: creating different sub-directories according to the category or time period to which the target data belongs, and writing the target data in different sub-directories of the migration directory in a dispersed manner.

[0179] Optionally, in this embodiment, steps S601 and S602 constitute the core execution phase of the data migration method in a large data volume scenario. The main goal of these two steps is to achieve efficient and orderly data reading and writing, while ensuring the reasonable organization and distribution of data in the target system. This process involves multiple technical fields such as concurrent processing, data classification and storage optimization.

[0180] In step S601, the system first needs to organize multi-threaded data reading operations according to the migration strategy determined in the previous step. This process involves the following key points:

[0181] Thread pool management: The system needs to create and maintain a thread pool for concurrent execution of data reading tasks. The size of the thread pool needs to be dynamically adjusted based on system resources (such as the number of CPU cores, available memory) and network bandwidth to achieve the best concurrency effect.

[0182] Data sharding: The system needs to divide the index data to be migrated into multiple small data shards. The size of each shard needs to be set according to the amount of data read each time determined in the previous step (such as S402). This process may involve complex data sharding algorithms, which need to consider the distribution characteristics of the data and query efficiency.

[0183] Task scheduling: The system needs to implement an efficient task scheduling mechanism to allocate data shards to threads in the thread pool. This scheduling mechanism needs to consider load balancing to ensure that each thread can get a reasonable amount of work.

[0184] Error handling and retry mechanism: In a distributed environment, network failures or service interruptions are common problems. The system needs to implement a robust error handling mechanism that can detect failed read operations and perform appropriate retries.

[0185] Progress tracking: The system needs to track the reading progress of each thread in real time so as to adjust the strategy in time or provide progress feedback to the user.

[0186] Step S602 is to perform intelligent data classification and storage based on the read target data. This step involves the following key points:

[0187] Data classification: The system needs to analyze the attributes of the target data (such as category or timestamp) and classify the data based on these attributes. This process may involve complex classification algorithms or rule engines.

[0188] Directory structure design: The system needs to dynamically create a suitable sub-directory structure based on the data classification results. This structure needs to balance query efficiency and storage balance.

[0189] Write strategy: The system needs to design an efficient write strategy to write data to different subdirectories. This strategy needs to consider load balancing to avoid the situation where some directories are too dense and other directories are too sparse.

[0190] Concurrent write control: Since multiple threads may perform write operations at the same time, the system needs to implement a concurrent control mechanism, such as a lock mechanism or CAS (Compare-And-Swap) operation, to ensure data consistency and integrity.

[0191] Storage monitoring and dynamic adjustment: The system needs to monitor the storage status of each sub-directory in real time and make dynamic adjustments when necessary, such as creating new sub-directories or adjusting write policies.

[0192] These two steps solve several key technical problems. First, they solve the problem of how to achieve efficient data reading and writing in a big data environment. Through multi-threaded concurrent processing and intelligent data sharding, the system can fully utilize hardware resources and greatly improve data processing speed. Second, they solve the problem of how to reasonably organize and distribute data in the target system. Through classification and storage strategies based on data attributes, the system can achieve balanced storage distribution while ensuring query efficiency. Finally, they also solve the problem of how to ensure data consistency and integrity in a complex distributed environment.

[0193] The technical effects achieved by these two steps are significant. First, they achieve high efficiency and high concurrency in the data migration process, greatly shortening the time required for migration. Second, through intelligent data classification and storage strategies, the system is able to build a data organization form in the target system that is well-structured, easy to manage and query. Third, this approach improves the scalability and fault tolerance of the system, enabling it to better cope with various abnormal situations that may occur in large-scale data migration.

[0194] In one embodiment of the data migration method in the large data volume scenario of the present application, see Figure 7 , and can also include the following:

[0195] Step S701: continuously monitor and collect real-time migration speed and local file system resource utilization data and input them into a pre-trained linear regression model;

[0196] Step S702: dynamically update the amount of data read during each migration operation, the maximum number of files in each storage directory, and the migrated directory according to the data read amount adjustment ratio output by the linear regression model, the change in the maximum number of files in the storage directory, and the priority of the migrated directory.

[0197] Optionally, in this embodiment, steps S701 and S702 constitute the intelligent optimization stage in the data migration method in a large data volume scenario. The main goal of these two steps is to dynamically adjust the migration strategy through real-time monitoring and machine learning technology to achieve adaptive optimization of the migration process. This process involves multiple technical fields such as data collection, machine learning, resource management, and dynamic strategy adjustment.

[0198] In step S701, the system first needs to establish a real-time monitoring mechanism to continuously collect key performance indicators during the migration process. This process involves the following key points:

[0199] Data collection: The system needs to design and implement an efficient data collection mechanism that can collect key indicators such as migration speed and local file system resource utilization in real time. This may involve system-level performance counters, file system API calls, and custom performance indicators.

[0200] Data preprocessing: The original collected data may contain noise or outliers. The system needs to implement data cleaning and preprocessing mechanisms to ensure the quality of data input into the machine learning model.

[0201] Feature Engineering: The system needs to extract meaningful features from the raw data, which may include complex feature engineering techniques such as moving average, rate of change, peak detection, etc.

[0202] Model training: Before actual deployment, the system needs to pre-train the linear regression model using historical data or simulated data. This process may involve machine learning techniques such as model selection and hyperparameter tuning.

[0203] Real-time prediction: The system needs to input the preprocessed real-time data into the pre-trained linear regression model and obtain the output results of the model.

[0204] Step S702 is to dynamically adjust the migration strategy based on the output of the model. This step involves the following key points:

[0205] Result interpretation: The system needs to correctly interpret the output results of the model and convert them into specific strategy adjustment recommendations.

[0206] Policy adjustment: Based on the output of the model, the system needs to dynamically adjust three key parameters: the amount of data read during each migration operation, the maximum number of files per storage directory, and the priority of the migrated directory.

[0207] Parameter constraints: When adjusting parameters, the system needs to consider various constraints, such as hardware limitations, file system characteristics, etc., to ensure that the adjusted parameters are within a reasonable range.

[0208] Smooth transition: To avoid sudden performance fluctuations caused by policy adjustments, the system may need to implement a smooth transition mechanism to gradually apply new policies.

[0209] Feedback loop: The system needs to continuously monitor the effects of strategy adjustments and feed this information back to the monitoring system to form a closed-loop optimization process.

[0210] These two steps solve several key technical problems. First, they solve the problem of how to achieve adaptive optimization in a complex and ever-changing big data migration environment. Through real-time monitoring and machine learning technology, the system can dynamically adjust strategies according to the current operating conditions to improve migration efficiency. Second, they solve the problem of how to maximize migration performance under limited system resources. Through intelligent resource allocation and strategy adjustment, the system can find the optimal operating parameters under various constraints. Finally, they also solve the problem of how to deal with uncertainty and volatility in the big data migration process, and through continuous monitoring and adjustment, the system can adapt to various complex operating environments.

[0211] The technical effects achieved by these two steps are significant. First, they achieve adaptive optimization of the migration process, allowing the system to achieve high performance under different hardware environments and data characteristics. Second, by dynamically adjusting the strategy, the system can better utilize available resources and improve overall migration efficiency. Third, this approach improves the robustness and reliability of the system, enabling it to better cope with various abnormal situations and performance fluctuations.

[0212] In order to effectively improve the data migration efficiency and accuracy in large data volume scenarios, the present application provides an embodiment of a data migration device in a large data volume scenario for implementing all or part of the content of the data migration method in a large data volume scenario, see Figure 8 The data migration device in the large data volume scenario specifically includes the following contents:

[0213] A migration strategy determination module 10 is used to determine the index to be migrated of the target data in the distributed search engine and its sharding setting, determine the corresponding migration task data volume and migration task priority according to the size of the index to be migrated and the hot spot distribution in the sharding setting, and determine the corresponding migration strategy according to the migration task data volume and the migration task priority;

[0214] The migration directory determination module 20 is used to determine the amount of data read in each migration operation and the maximum number of files in each storage directory according to the business requirements sent by the user and the storage performance of the local file system used to store the migration data, and determine the corresponding migration directory according to the amount of data and the maximum number of files;

[0215] A migration parameter updating module 30 is used to read the target data in the to-be-migrated index in batches according to the migration strategy, and write the target data into the migration-in directory in the local file system, input the real-time migration speed and the real-time local file system resource utilization rate into a set linear regression model, and dynamically update the amount of data read in each migration operation, the maximum number of files in each storage directory, and the migration-in directory according to the real-time adjustment parameters output by the linear regression model, wherein the linear regression model is trained by historical data migration data;

[0216] The multi-threaded deletion module 40 is used to start multiple threads to perform a deletion operation on the corresponding data in the distributed search engine after the write operation is completed, and wait for all threads to complete the deletion operation before continuing to read the next batch of data.

[0217] From the above description, it can be seen that the data migration device in a large data volume scenario provided by the embodiment of the present application can determine the index to be migrated of the target data in the distributed search engine and its sharding setting, determine the corresponding migration task data volume and migration task priority according to the size of the index to be migrated and the hot spot distribution in the sharding setting, and determine the corresponding migration strategy according to the migration task data volume and the migration task priority; read the target data in the index to be migrated in batches according to the migration strategy, and write the target data into the migration directory in the local file system, input the real-time migration speed and the real-time local file system resource utilization into the set linear regression model, and dynamically update the data volume read during each migration operation, the maximum number of files in each storage directory, and the migration directory according to the real-time adjustment parameters output by the linear regression model, thereby improving the data migration efficiency and accuracy in large data volume scenarios.

[0218] From the hardware level, in order to effectively improve the efficiency and accuracy of data migration in large data volume scenarios, the present application provides an embodiment of an electronic device for implementing all or part of the content of the data migration method in large data volume scenarios, and the electronic device specifically includes the following content:

[0219] Processor, memory, communication interface and bus; wherein the processor, memory and communication interface communicate with each other through the bus; the communication interface is used to realize information transmission between the data migration device in large data volume scenario and related devices such as core business system, user terminal and related database; the logic controller can be a desktop computer, tablet computer and mobile terminal, etc., but the present embodiment is not limited thereto. In the present embodiment, the logic controller can be implemented with reference to the embodiment of the data migration method in large data volume scenario and the embodiment of the data migration device in large data volume scenario in the embodiment, and the contents thereof are incorporated herein, and the repeated parts are not repeated.

[0220] It is understandable that the user terminal may include a smart phone, a tablet electronic device, a network set-top box, a portable computer, a desktop computer, a personal digital assistant (PDA), a vehicle-mounted device, a smart wearable device, etc. Among them, the smart wearable device may include smart glasses, a smart watch, a smart bracelet, etc.

[0221] In practical applications, part of the data migration method in a large data volume scenario can be performed on the electronic device side as described above, or all operations can be completed in the client device. The specific selection can be based on the processing capability of the client device and the limitations of the user's usage scenario. This application does not limit this. If all operations are completed in the client device, the client device may also include a processor.

[0222] The client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side, and other implementation scenarios may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, or a server cluster consisting of multiple servers, or a server structure of a distributed device.

[0223] Fig. 9 FIG. 9 is a schematic block diagram of the system structure of the electronic device 9600 according to an embodiment of the present application. Fig. 9 As shown, the electronic device 9600 may include a central processor 9100 and a memory 9140; the memory 9140 is coupled to the central processor 9100. It is worth noting that Fig. 9 is exemplary; other types of structures may also be used to supplement or replace this structure to implement telecommunication functions or other functions.

[0224] In one embodiment, the data migration method function in a large data volume scenario may be integrated into the central processing unit 9100. The central processing unit 9100 may be configured to perform the following control:

[0225] Step S101: determining the index to be migrated of the target data in the distributed search engine and its sharding setting, determining the corresponding migration task data volume and migration task priority according to the size of the index to be migrated and the hot spot distribution in the sharding setting, and determining the corresponding migration strategy according to the migration task data volume and the migration task priority;

[0226] Step S102: determining the amount of data to be read in each migration operation and the maximum number of files in each storage directory according to the business requirements sent by the user and the storage performance of the local file system used to store the migration data, and determining the corresponding migration directory according to the amount of data and the maximum number of files;

[0227] Step S103: Read the target data in the to-be-migrated index in batches according to the migration strategy, and write the target data into the migration-in directory in the local file system, input the real-time migration speed and the real-time local file system resource utilization rate into a set linear regression model, and dynamically update the amount of data read in each migration operation, the maximum number of files in each storage directory, and the migration-in directory according to the real-time adjustment parameters output by the linear regression model, wherein the linear regression model is trained by historical data migration data;

[0228] Step S104: after the write operation is completed, multiple threads are started to perform a delete operation on the corresponding data in the distributed search engine, and the next batch of data is read after all threads complete the delete operation.

[0229] From the above description, it can be seen that the electronic device provided by the embodiment of the present application determines the index to be migrated of the target data in the distributed search engine and its sharding setting, determines the corresponding migration task data volume and migration task priority according to the size of the index to be migrated and the hot spot distribution in the sharding setting, and determines the corresponding migration strategy according to the migration task data volume and the migration task priority; reads the target data in the index to be migrated in batches according to the migration strategy, and writes the target data into the migration directory in the local file system, inputs the real-time migration speed and the real-time local file system resource utilization into the set linear regression model, and dynamically updates the amount of data read during each migration operation, the maximum number of files in each storage directory, and the migration directory according to the real-time adjustment parameters output by the linear regression model, thereby improving the data migration efficiency and accuracy in large data volume scenarios.

[0230] In another embodiment, the data migration device in a large data volume scenario can be configured separately from the central processing unit 9100. For example, the data migration device in a large data volume scenario can be configured as a chip connected to the central processing unit 9100, and the function of the data migration method in a large data volume scenario can be implemented through the control of the central processing unit.

[0231] like Fig. 9 As shown, the electronic device 9600 may also include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is worth noting that the electronic device 9600 does not necessarily have to include Fig. 9 In addition, the electronic device 9600 may also include Fig. 9 For components not shown, reference may be made to the prior art.

[0232] like Fig. 9As shown, the central processing unit 9100 is sometimes also referred to as a controller or an operation control, and may include a microprocessor or other processor device and / or logic device. The central processing unit 9100 receives input and controls the operation of various components of the electronic device 9600.

[0233] The memory 9140 may be, for example, one or more of a cache, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory or other suitable devices. The above-mentioned information related to the failure may be stored, and a program for executing the relevant information may also be stored. The CPU 9100 may execute the program stored in the memory 9140 to implement information storage or processing, etc.

[0234] The input unit 9120 provides input to the central processing unit 9100. The input unit 9120 is, for example, a key or a touch input device. The power supply 9170 is used to provide power to the electronic device 9600. The display 9160 is used to display display objects such as images and texts. The display may be, for example, an LCD display, but is not limited thereto.

[0235] The memory 9140 may be a solid-state memory, such as a read-only memory (ROM), a random access memory (RAM), a SIM card, etc. It may also be a memory that saves information even when the power is off, can be selectively erased, and is provided with more data, examples of which are sometimes referred to as EPROMs, etc. The memory 9140 may also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application / function storage unit 9142, which is used to store application programs and function programs or processes for executing the operation of the electronic device 9600 through the central processor 9100.

[0236] The memory 9140 may also include a data storage unit 9143 for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 9144 of the memory 9140 may include various drivers for communication functions of the electronic device and / or for executing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0237] The communication module 9110 is a transmitter / receiver that sends and receives signals via the antenna 9111. The communication module 9110 (transmitter / receiver) is coupled to the central processor 9100 to provide input signals and receive output signals, which may be the same as the case of a conventional mobile communication terminal.

[0238] Based on different communication technologies, multiple communication modules 9110 may be provided in the same electronic device, such as a cellular network module, a Bluetooth module and / or a wireless LAN module, etc. The communication module 9110 (transmitter / receiver) is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide an audio output via the speaker 9131 and receive an audio input from the microphone 9132, thereby realizing a common telecommunication function. The audio processor 9130 may include any suitable buffer, decoder, amplifier, etc. In addition, the audio processor 9130 is also coupled to the central processor 9100, so that recording can be performed on the local machine through the microphone 9132, and the sound stored on the local machine can be played through the speaker 9131.

[0239] The embodiments of the present application also provide a computer-readable storage medium capable of implementing all steps of the data migration method in the scenario of large data volume in which the execution subject is a server or a client in the above embodiments. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, all steps of the data migration method in the scenario of large data volume in the above embodiments are implemented. For example, when the processor executes the computer program, the following steps are implemented:

[0240] Step S101: determining the index to be migrated of the target data in the distributed search engine and its sharding setting, determining the corresponding migration task data volume and migration task priority according to the size of the index to be migrated and the hot spot distribution in the sharding setting, and determining the corresponding migration strategy according to the migration task data volume and the migration task priority;

[0241] Step S102: determining the amount of data to be read in each migration operation and the maximum number of files in each storage directory according to the business requirements sent by the user and the storage performance of the local file system used to store the migration data, and determining the corresponding migration directory according to the amount of data and the maximum number of files;

[0242] Step S103: Read the target data in the to-be-migrated index in batches according to the migration strategy, and write the target data into the migration-in directory in the local file system, input the real-time migration speed and the real-time local file system resource utilization rate into a set linear regression model, and dynamically update the amount of data read in each migration operation, the maximum number of files in each storage directory, and the migration-in directory according to the real-time adjustment parameters output by the linear regression model, wherein the linear regression model is trained by historical data migration data;

[0243] Step S104: after the write operation is completed, multiple threads are started to perform a deletion operation on the corresponding data in the distributed search engine, and the next batch of data is read after all threads complete the deletion operation.

[0244] From the above description, it can be seen that the computer-readable storage medium provided in the embodiment of the present application determines the index to be migrated of the target data in the distributed search engine and its sharding setting, determines the corresponding migration task data volume and migration task priority according to the size of the index to be migrated and the hot spot distribution in the sharding setting, and determines the corresponding migration strategy according to the migration task data volume and the migration task priority; reads the target data in the index to be migrated in batches according to the migration strategy, and writes the target data into the migration directory in the local file system, inputs the real-time migration speed and the real-time local file system resource utilization into the set linear regression model, and dynamically updates the amount of data read during each migration operation, the maximum number of files in each storage directory, and the migration directory according to the real-time adjustment parameters output by the linear regression model, thereby improving the data migration efficiency and accuracy in large data volume scenarios.

[0245] The embodiments of the present application also provide a computer program product capable of implementing all steps of the data migration method in a large data volume scenario in which the execution subject is a server or a client in the above embodiments. When the computer program / instruction is executed by a processor, the steps of the data migration method in the large data volume scenario are implemented. For example, the computer program / instruction implements the following steps:

[0246] Step S101: determining the index to be migrated of the target data in the distributed search engine and its sharding setting, determining the corresponding migration task data volume and migration task priority according to the size of the index to be migrated and the hot spot distribution in the sharding setting, and determining the corresponding migration strategy according to the migration task data volume and the migration task priority;

[0247] Step S102: determining the amount of data to be read in each migration operation and the maximum number of files in each storage directory according to the business requirements sent by the user and the storage performance of the local file system used to store the migration data, and determining the corresponding migration directory according to the amount of data and the maximum number of files;

[0248] Step S103: Read the target data in the to-be-migrated index in batches according to the migration strategy, and write the target data into the migration-in directory in the local file system, input the real-time migration speed and the real-time local file system resource utilization rate into a set linear regression model, and dynamically update the amount of data read in each migration operation, the maximum number of files in each storage directory, and the migration-in directory according to the real-time adjustment parameters output by the linear regression model, wherein the linear regression model is trained by historical data migration data;

[0249] Step S104: after the write operation is completed, multiple threads are started to perform a deletion operation on the corresponding data in the distributed search engine, and the next batch of data is read after all threads complete the deletion operation.

[0250] From the above description, it can be seen that the computer program product provided by the embodiment of the present application determines the index to be migrated of the target data in the distributed search engine and its sharding setting, determines the corresponding migration task data volume and migration task priority according to the size of the index to be migrated and the hot spot distribution in the sharding setting, and determines the corresponding migration strategy according to the migration task data volume and the migration task priority; reads the target data in the index to be migrated in batches according to the migration strategy, and writes the target data into the migration directory in the local file system, inputs the real-time migration speed and the real-time local file system resource utilization into the set linear regression model, and dynamically updates the amount of data read during each migration operation, the maximum number of files in each storage directory, and the migration directory according to the real-time adjustment parameters output by the linear regression model, thereby improving the data migration efficiency and accuracy in large data volume scenarios.

[0251] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0252] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (apparatus), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0253] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0254] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0255] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A data migration method in a large data volume scenario, characterized in that: The method comprises: Determine the target data's to-be-migrated index and its sharding settings in the distributed search engine, determine the corresponding migration task data volume and migration task priority according to the size of the to-be-migrated index and the hotspot distribution in the sharding settings, and determine the corresponding migration strategy according to the migration task data volume and the migration task priority; Determine the amount of data read in each migration operation and the maximum number of files in each storage directory according to the business requirements sent by the user and the storage performance of the local file system used to store the migration data, and determine the corresponding migration directory according to the amount of data and the maximum number of files; Read the target data in the to-be-migrated index in batches according to the migration strategy, and write the target data into the migration-in directory in the local file system, input the real-time migration speed and the real-time local file system resource utilization rate into a set linear regression model, and dynamically update the amount of data read in each migration operation, the maximum number of files in each storage directory, and the migration-in directory according to the real-time adjustment parameters output by the linear regression model, wherein the linear regression model is trained by historical data migration data; After the write operation is completed, multiple threads are started to delete the corresponding data in the distributed search engine, and the next batch of data is read after all threads complete the deletion operation.

2. The data migration method in a large data volume scenario according to claim 1 is characterized in that: The step of determining the index of the target data to be migrated in the distributed search engine and its sharding setting, and determining the corresponding migration task data volume and migration task priority according to the size of the index to be migrated and the hotspot distribution in the sharding setting, includes: Determine an index to be migrated in a distributed search engine where the target data to be migrated is located, wherein the index to be migrated includes at least one shard, and determine a shard setting of the shard according to a storage location and a data volume of each shard; The corresponding hot spot distribution is determined according to the storage location and data size of each shard, the corresponding migration task data volume is determined according to the size of the index to be migrated, and the corresponding migration task priority is determined according to the hot spot distribution of the shard.

3. The data migration method in a large data volume scenario according to claim 1 is characterized in that: The determining a corresponding migration strategy according to the migration task data volume and the migration task priority includes: Dividing the data volume and priority of the migration task into different levels in advance and constructing a migration strategy matrix, wherein each migration strategy in the migration strategy matrix includes specific migration operation parameters; A matching search is performed in a migration strategy matrix according to the migration task data volume and the migration task priority to determine a corresponding migration strategy.

4. The data migration method in a large data volume scenario according to claim 1 is characterized in that: The method of determining the amount of data to be read in each migration operation and the maximum number of files in each storage directory according to the business requirements sent by the user and the storage performance of the local file system used to store the migration data includes: The migration speed is calculated based on the expected completion time in the business requirements sent by the user and the available storage space of the local file system used to store the migration data. The amount of data read in each migration operation and the maximum number of files in each storage directory are determined according to the maximum value of the migration speeds.

5. The data migration method in a large data volume scenario according to claim 1 is characterized in that: The step of determining the corresponding migration directory according to the data volume and the maximum number of files includes: Determine the directory hierarchy and storage space distribution of the local file system; A directory with sufficient available space and a small number of files is determined as a migration directory according to the data volume and the maximum number of files, and the migration data is written to multiple subdirectories under the directory in a dispersed manner.

6. The data migration method in a large data volume scenario according to claim 1 is characterized in that: The step of reading the target data in the to-be-migrated index in batches according to the migration strategy and writing the target data into the migration directory in the local file system includes: According to the migration strategy, the target data in the index to be migrated is read in batches using multiple threads; Different sub-directories are created according to the category or time period to which the target data belongs, and the target data are scattered and written into different sub-directories of the migration directory.

7. The data migration method in a large data volume scenario according to claim 1 is characterized in that: The real-time migration speed and the real-time local file system resource utilization are input into a set linear regression model, and the amount of data read during each migration operation, the maximum number of files in each storage directory, and the migration directory are dynamically updated according to the real-time adjustment parameters output by the linear regression model, wherein the linear regression model is trained by historical data migration data, and includes: Continuously monitor and collect real-time migration speed and local file system resource utilization data and input them into the pre-trained linear regression model; The amount of data read during each migration operation, the maximum number of files in each storage directory and the migrated directory are dynamically updated according to the data read adjustment ratio output by the linear regression model, the change in the maximum number of files in the storage directory and the priority adjustment of the migrated directory.

8. A data migration device in a large data volume scenario, characterized in that: The device comprises: A migration strategy determination module is used to determine the index to be migrated of the target data in the distributed search engine and its sharding settings, determine the corresponding migration task data volume and migration task priority according to the size of the index to be migrated and the hot spot distribution in the sharding settings, and determine the corresponding migration strategy according to the migration task data volume and the migration task priority; A migration directory determination module is used to determine the amount of data read in each migration operation and the maximum number of files in each storage directory according to the business requirements sent by the user and the storage performance of the local file system used to store the migration data, and determine the corresponding migration directory according to the amount of data and the maximum number of files; A migration parameter update module, used to read the target data in the to-be-migrated index in batches according to the migration strategy, and write the target data into the migration-in directory in the local file system, input the real-time migration speed and the real-time local file system resource utilization rate into a set linear regression model, and dynamically update the amount of data read in each migration operation, the maximum number of files in each storage directory, and the migration-in directory according to the real-time adjustment parameters output by the linear regression model, wherein the linear regression model is trained by historical data migration data; The multi-threaded deletion module is used to start multiple threads to delete the corresponding data in the distributed search engine after the write operation is completed, and wait for all threads to complete the deletion operation before continuing to read the next batch of data.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the data migration method in a large data volume scenario described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the data migration method in a large data volume scenario described in any one of claims 1 to 7 are implemented.

Images