Data migration method and device, storage medium and computer program product

Abstract

The invention discloses a data migration method and device, a storage medium and a computer program product. The method comprises the following steps: acquiring an initial data set of a plurality of service modules in a service system to be migrated, the plurality of service modules being function modules obtained by splitting according to a plurality of functions provided by the service system; determining the similarity among the service modules according to the initial data set of each service module, and clustering the plurality of service modules according to the similarity to obtain a plurality of clusters; and for each clustering cluster, performing dimension reduction processing on the initial data set of each service module in the clustering cluster to obtain a target data set of each service module, and migrating the target data set of each service module in the clustering cluster. According to the method and the device, the technical problem of data missing and inconsistency of dimension-reduced data due to the fact that the dependency relationship between the business modules to which the data belong is not considered when dimension-reduced migration is carried out on a large-scale data set in the prior art is solved.

Description

Technical Field

[0001] The present application relates to the field of cloud computing technology, and more specifically, to a data migration method and device, a storage medium, and a computer program product. Background Art

[0002] With the continuous development of cloud computing technology, data migration has become increasingly important in cloud environments. Especially for large-scale data sets, effective and efficient migration is one of the key elements of cloud services. Traditional data migration methods often ignore the dependencies between applications, which may lead to problems such as data consistency and data integrity during the migration process, thus affecting the normal operation of applications and the efficiency of data use.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present application provide a data migration method and device, a storage medium, and a computer program product to at least solve the technical problem that the related technology does not consider the dependencies between the business modules to which the data belongs when performing dimensionality reduction migration on large-scale data sets, resulting in missing and inconsistent data in the data after dimensionality reduction.

[0005] According to one aspect of an embodiment of the present application, a data migration method is provided, comprising: obtaining initial data sets of multiple business modules in a business system to be migrated, wherein the multiple business modules are functional modules obtained by splitting according to multiple functions provided by the business system; determining the similarity between the business modules based on the initial data sets of the business modules, and clustering the multiple business modules according to the similarity to obtain multiple clustering clusters; for each clustering cluster, performing dimensionality reduction processing on the initial data sets of each business module in the clustering cluster to obtain a target data set of each business module, and migrating the target data set of each business module in the clustering cluster.

[0006] Optionally, the similarity between each business module is determined based on the initial data set of each business module, including: for any two target business modules in multiple business modules, using a preset distance algorithm to determine the distance between the initial data sets of the two target business modules, and judging the relationship between the distance and a preset threshold value; when the distance is lower than the threshold value, using a Gaussian kernel function to adjust the distance to obtain the similarity between the two target business modules; when the distance is not lower than the threshold value, using a Gaussian kernel function and an exponential function to adjust the distance to obtain the similarity between the two target business modules.

[0007] Optionally, multiple business modules are clustered according to similarity to obtain multiple clusters, including: judging the relationship between the similarity and a preset similarity threshold; when the similarity is not lower than the preset similarity threshold, determining that the two target business modules belong to the same type; and aggregating multiple business modules of the same type to obtain multiple clusters.

[0008] Optionally, after obtaining multiple clusters, the method also includes: for each cluster, determining the value of the cluster under a preset evaluation dimension, wherein the evaluation dimension includes at least one of the following: the number of business modules within the cluster, the importance of the business modules within the cluster in the business system, and the amount of data; weightedly summing the values ​​under the preset evaluation dimension and the corresponding dimension weights to obtain the migration priority of the cluster.

[0009] Optionally, the initial data set of each business module in the clustering cluster is subjected to dimensionality reduction processing to obtain the target data set of each business module, including: constructing a data matrix with each business module in the clustering cluster as a row and a preset dimension as a column, wherein the preset dimension includes at least one of the following: link abnormality frequency, bandwidth peak ratio, business cycle duration, interactive data volume in the business cycle, transmission delay, and each matrix element in the data matrix represents the initial data set of the corresponding business module under the corresponding preset dimension; determining the matrix product result of the data matrix and the corresponding transposed matrix, and dividing the matrix product result by the difference obtained by subtracting 1 from the number of business modules in the clustering cluster to obtain a covariance matrix; performing eigendecomposition on the covariance matrix to obtain multiple eigenvalues ​​and corresponding multiple eigenvectors; selecting a preset number of eigenvectors from the multiple eigenvectors as column vectors based on the eigenvalues ​​to construct a projection matrix; using the projection matrix to perform dimensionality reduction processing on the data matrix to obtain the target data set of each business module in the clustering cluster.

[0010] Optionally, the method also includes: determining the migration duration and network resource utilization rate of the target data set of each business system in the clustering cluster during the migration process; judging the relationship between the migration duration and a preset first range threshold, and between the network resource utilization rate and a preset second range threshold; when the migration duration is within the first range threshold, and the network resource utilization rate is within the second range threshold, determining that the migration process of the target data set of each business module in the clustering cluster is in a normal state; when the migration duration is not within the first range threshold, and / or the network resource utilization rate is not within the second range threshold, determining that the migration process of the target data set of each business module in the clustering cluster is in an abnormal state, and performing corresponding processing operations on the migration process of the target data set of each business module in the clustering cluster, wherein the processing operations include at least one of the following: increasing network resources, adjusting migration priority.

[0011] Optionally, determining the migration duration and resource occupancy rate of the target data set of each business system in the cluster during the migration process includes: determining the data volume and network resource information of the target data set of each business system in the cluster, wherein the network resource information includes at least one of the following: network bandwidth occupancy rate, memory occupancy rate, bandwidth transmission rate, and snapshot creation rate; determining the data transmission duration based on the data transmission rate and data volume, and determining the mirror creation duration based on the snapshot creation rate and data volume, and determining the migration duration based on the data transmission duration and the mirror creation duration.

[0012] According to another aspect of an embodiment of the present application, a data migration device is also provided, including: an acquisition module, used to acquire an initial data set of multiple business modules in a business system to be migrated, wherein the multiple business modules are functional modules obtained by splitting according to multiple functions provided by the business system; a determination module, used to determine the similarity between each business module based on the initial data set of each business module, and cluster the multiple business modules according to the similarity to obtain multiple clustering clusters; a migration module, used to perform dimensionality reduction processing on the initial data set of each business module in the clustering cluster for each clustering cluster, obtain a target data set of each business module, and migrate the target data set of each business module in the clustering cluster.

[0013] According to another aspect of an embodiment of the present application, a non-volatile storage medium is further provided, the non-volatile storage medium including a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-mentioned data migration method by running the computer program.

[0014] According to another aspect of an embodiment of the present application, a computer program product is further provided. The computer program product includes a stored computer program, wherein the computer program implements the above-mentioned data migration method when executed by a processor.

[0015] In an embodiment of the present application, the similarity between each business module is determined based on the initial data set of each business module in the business system, so as to cluster the business modules of the same type and obtain multiple clusters to identify and separate business modules with similar characteristics and dependencies, which provides a basis for subsequent dimensionality reduction migration; then, the initial data set of each business module in each cluster is subjected to dimensionality reduction processing, which effectively reduces the data dimension. The data set after dimensionality reduction is called the target data set, which is more compact and efficient while maintaining the integrity of the information, reducing the overhead of data storage and transmission; finally, the target data set of each business module in the cluster is migrated, which greatly improves the migration efficiency. The technical effect of efficient and secure migration of multiple business modules in the business system in the cloud environment is achieved, and the technical problem that the related technology does not consider the dependency between the business modules to which the data belongs when performing dimensionality reduction migration on large-scale data sets, resulting in data missing and inconsistent data after dimensionality reduction is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0017] Figure 1 It is a hardware structure block diagram of an optional computer terminal (or mobile device) for implementing a data migration method according to an embodiment of the present application;

[0018] Figure 2 is a flowchart of an optional data migration method according to an embodiment of the present application;

[0019] Figure 3 is a schematic diagram of an optional clustering cluster according to an embodiment of the present application;

[0020] Figure 4 is a schematic structural diagram of an optional data migration device according to an embodiment of the present application;

[0021] Figure 5 It is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0023] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0024] In addition, the relevant information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. For example, an interface is set up between this system and the relevant user or organization. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or organization through the interface, and obtain relevant information after receiving the consent information fed back by the aforementioned user or organization.

[0025] Example 1

[0026] According to an embodiment of the present application, an embodiment of a data migration method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0027] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal (or mobile device) for implementing a data migration method. Figure 1As shown, the computer terminal 10 (or mobile device 10) may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0028] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0029] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the data migration method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the data migration method of the above-mentioned business module is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0030] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0031] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0032] In the above operating environment, Figure 2 is a flow chart of an optional data migration method according to an embodiment of the present application, such as Figure 2 As shown, the method at least includes steps S202-S206, wherein:

[0033] Step S202: obtaining initial data sets of multiple business modules in the business system to be migrated.

[0034] In the technical solution provided in step S202, the above-mentioned business system refers to a software system or application program that supports specific business processes and functions of an enterprise or organization, such as a financial system, a customer relationship management system (CRM), a supply chain management system (SCM), a human resource management system (HRM), etc. As the basic unit for building a business system, the business module can be understood as a functional module obtained by splitting according to multiple functions provided by the business system, so it usually exists in the form of a subsystem or a functional unit. Generally, each business module is responsible for executing a specific function of the system or a part of the business process. For example, in a CRM system, the business module may include but is not limited to: a customer information management module, a sales management module, a marketing activity management module, etc. The initial data set of the above-mentioned business module may be all data related to the business module, which includes but is not limited to business data, user data, configuration files, log records, database content, cache information, and any other data related to the operation and status of the business module. It should be noted that the above-mentioned business system to be migrated is one or more within the same project, and its number can be set according to the actual application scenario.

[0035] Step S204 , determining the similarity between each business module according to the initial data set of each business module, and clustering the multiple business modules according to the similarity to obtain multiple clusters.

[0036] In the technical solution provided in step S204, the above similarity is used to measure the similarity of the data content, consistency of the data structure or the similarity of the business functions of the initial data sets of different business modules. According to the multiple clusters divided by similarity, the business modules in each cluster have a high degree of similarity, while the similarity between the business modules in different clusters is lower. This can avoid the "one-size-fits-all" dimensionality reduction processing of all data sets in the business system, resulting in poor data integrity and consistency.

[0037] As an optional implementation, in the technical solution provided in the above step S204, the similarity between the business modules may be determined by the following method, including:

[0038] For any two target business modules in the multiple business modules, a preset distance algorithm is used to determine the distance between the initial data sets of the two target business modules, and the relationship between the distance and a preset threshold value is determined.

[0039] When the distance is lower than the threshold, the Gaussian kernel function is used to adjust the distance to obtain the similarity between the two target business modules;

[0040] When the distance is not less than the threshold value, the distance is adjusted using the Gaussian kernel function and the exponential function to obtain the similarity between the two target business modules.

[0041] In this embodiment, a preset distance algorithm, such as Euclidean distance, cosine similarity or Jaccard distance, is first selected to measure the distance between the initial data sets of any two target business modules. This step is intended to quantify the difference or similarity between the data sets. Next, the calculated distance is compared with a preset threshold value, wherein the threshold value can be set as a threshold value reflecting the significant difference between the business module data sets. Then, if the calculated distance is lower than the threshold value, it means that the data sets of the two target business modules are very close. At this time, the distance can be adjusted using the Gaussian kernel function to obtain the similarity between the two target business modules; and if the calculated distance is not lower than the threshold value, it means that the data sets of the two target business modules are very different. At this time, in order to keep the dissimilar business modules away from each other, the Gaussian kernel function and the exponential function can be used to adjust the distance to obtain the similarity between the two target business modules, so that the distinction between the two target business modules can be enhanced. Among them, due to the nature of the exponential function, the larger the distance (close to zero), the adjusted similarity value will tend to infinitesimal, which means that the similarity between the business modules is very low.

[0042] For example, for application module M i and application module M j For example, the Euclidean distance calculation application module Mi and application module M j The similarity between them is expressed as: ||M i ,M j ||. Then, if ||M i ,M j ||<ω, then the Gaussian kernel function is used to adjust the distance to obtain the similarity between the two target business modules, which can be expressed as: N ij =θ = exp(-||M i ,M j || 2 / δ), where δ represents a scaling factor, which is used to control the decay rate of the exponential function, and can be the mean square of the Euclidean distance between the initial data sets of all business systems in the business system. i ,M j ||≥ω, then the distance is adjusted using the Gaussian kernel function and the exponential function to obtain the similarity between the two target business modules, which can be expressed as: N ij =θexp(θ-1).

[0043] Furthermore, after obtaining the similarity of the above-mentioned various business modules, the multiple business modules may be clustered according to the following method, including:

[0044] Determine the relationship between the similarity and a preset similarity threshold;

[0045] When the similarity is not less than a preset similarity threshold, determining that the two target business modules are of the same type;

[0046] Multiple business modules of the same type are aggregated to obtain multiple clusters.

[0047] Specifically, the above clustering process can be understood as follows: first, a similarity threshold is pre-set, which is used to determine whether two business modules are similar enough to be classified into the same cluster. Among them, the setting of the similarity threshold depends on the specific business scenario and data characteristics, and the present application does not impose specific restrictions on the setting of this threshold; then, the calculated similarity between any two target business modules is compared with the preset similarity threshold; if the similarity between the two target business modules is not lower than the similarity threshold, it means that they have something in common in terms of business logic, data processing or user interaction, and the two modules can be considered to be of the same type, and all business modules of the same type are aggregated to form a cluster. Repeat the above process until all business modules in the business system are assigned to the corresponding cluster. In this way, the entire business system is subdivided into multiple clusters with similar characteristics, and the business modules in each cluster share similar business attributes or functions.

[0048] For example, Figure 3 is a schematic diagram of an optional business module clustering according to the present application, such as Figure 3 As shown in the figure, the business modules of the same type are clustered according to the similarity between the business modules to achieve the purpose of clustering the same type. In addition, when the Euclidean distance between the initial data sets of the business modules is large, the distance is adjusted through the exponential function to keep the heterogeneous modules away, so that different types of business modules can be better distinguished.

[0049] Furthermore, after obtaining the above-mentioned multiple clusters, in order to more reasonably allocate network resources, storage resources and computing resources during the migration process, the migration priority of each cluster can be calculated first. This can be achieved through the following steps:

[0050] For each cluster, determine the value of the cluster under a preset evaluation dimension, wherein the evaluation dimension includes at least one of the following: the number of business modules within the cluster, the importance of the business modules within the cluster in the business system, and the amount of data; perform weighted summation on the value under the preset evaluation dimension and the corresponding dimension weight to obtain the migration priority of the cluster.

[0051] The above process can be understood as follows: first, collect the values ​​of the current cluster under multiple evaluation dimensions, such as the number of business modules in the cluster, the importance of the business modules in the cluster in the business system, the amount of data (which can reflect the complexity of migration), etc.; then, formulate a scoring standard and weight for each preset scoring indicator, where the weight can reflect its importance in the migration priority decision, which can generally be determined based on expert scoring and historical data analysis; finally, calculate the migration priority based on the scoring standard and the corresponding weight value corresponding to the values ​​of the current cluster under multiple evaluation dimensions, which can be specifically calculated using the following formula:

[0052]

[0053] In the formula, n represents the evaluation dimension, and the score i Indicates the score and weight corresponding to the value of the i-th evaluation dimension i Represents the weight value of the i-th evaluation dimension.

[0054] This application uses the above process to scientifically and reasonably determine the migration priority of each cluster, and in the subsequent process, performs migration operations on each cluster in turn according to the migration priority, which has the following advantages:

[0055] (1) By migrating key or high-priority clusters first, network resources, storage resources, and computing resources can be allocated more reasonably during the migration process. This means that under limited resources, the continuity and efficiency of important businesses can be prioritized to avoid resource waste.

[0056] (2) High-priority clusters can be migrated first to allocate more resources to them, which can speed up their migration process and shorten the completion time of the entire migration project. This is especially important for migration of large systems or in emergency situations, and can quickly achieve business launch or data switching.

[0057] (3) Processing the most critical and complex business modules at the early stage of migration or during a period with more resources can effectively reduce the risks of data loss and data inconsistency during the migration process. Once the migration of high-priority clusters is completed and confirmed to be correct, the subsequent migration planning and execution of low-priority clusters will be smoother and the risks will be relatively reduced.

[0058] (4) During the migration process, the clusters with the greatest impact on the business are processed first, which can minimize the interruption of user services. This can ensure that important businesses can still operate normally during the migration and improve user experience.

[0059] Step S206 , for each cluster, perform dimensionality reduction processing on the initial data set of each business module in the cluster to obtain the target data set of each business module, and migrate the target data set of each business module in the cluster.

[0060] In the technical solution provided in step S206, for each cluster in the multiple clusters, a dimensionality reduction algorithm is applied to perform dimensionality reduction processing on the initial data set of each business module in the cluster. The purpose of dimensionality reduction is to reduce the dimension of the data while retaining the key information and features of the data set as much as possible to optimize the storage and transmission efficiency of the data in the cloud environment. After the dimensionality reduction processing, the target data set of each business module in the cluster can be obtained, and the target data set is migrated as a migration object. Since all the original data in the same cluster are reduced in dimension, not only the business relevance and key features of the initial data set can be maintained, but also the migration object can have a lower dimension and a lower volume, so that while ensuring the consistency and integrity of the data during the migration process, the transmission cost can be greatly reduced, and the migration process will be more efficient, so as to achieve better management of storage resources in the cloud environment.

[0061] As an optional implementation, in the technical solution provided in the above step S206, the method may include the following steps:

[0062] Step S2061 , constructing a data matrix with each business module in the cluster as a row and the preset dimensions as a column.

[0063] The preset dimensions include at least one of the following: link abnormality frequency, bandwidth peak ratio, service cycle duration, interactive data volume within the service cycle, and transmission delay. The data matrix can be denoted as X, where each matrix element X mn Represents the initial data set of the corresponding business module m under the corresponding preset dimension n.

[0064] Step S2062, determine the matrix product result of the data matrix and the corresponding transposed matrix, and divide the matrix product result by the difference obtained by subtracting 1 from the number of business modules in the cluster to obtain a covariance matrix. Therefore, the expression of the covariance matrix can be written as: Where m represents the number of business modules in the cluster.

[0065] Step S2063, perform eigendecomposition on the covariance matrix to obtain multiple eigenvalues ​​and corresponding multiple eigenvectors. The eigendecomposition process can be expressed as: Cv = λv, where λ represents the eigenvalue and v represents the eigenvector.

[0066] Step S2064, select a preset number of eigenvectors from the multiple eigenvectors as column vectors according to the eigenvalues ​​to construct a projection matrix. The projection matrix can be understood as: according to the size of the eigenvalue, select the eigenvectors corresponding to the first k largest eigenvalues ​​to form the principal component. Therefore, the expression of the projection matrix can be written as: P = [v1, v2, ..., v k ].

[0067] Step S2065, use the projection matrix to perform dimensionality reduction processing on the data matrix to obtain the target data set of each business module in the cluster. The dimensionality reduction process can be expressed by the following formula: Y = P T X.

[0068] In addition, in order to enhance the reliability and integrity of data migration during the migration process, technologies such as redundant backup, data error checking (such as CRC check, MD5 hash, etc.) and error correction codes (such as Reed-Solomon, Turbo code, etc.) can also be introduced. In the migration process, considering that the original data may be lost or damaged due to hardware failure, network interruption, software error, etc., therefore, by saving multiple copies of the data at multiple locations or multiple systems at the same time, even if one of the copies has a problem, there are other copies that can be used, thereby ensuring the security and high availability of the migrated data; in addition, considering that the data may have errors during the migration process due to network noise, hardware failure, etc., the error detection mechanism can be used to check the integrity of the data in real time, and by adding additional error correction codes to the migration data, the received migration data can be self-repaired within a certain range. Even if the data encounters errors during the migration process, the correct original data can be restored to ensure data quality, greatly improve the reliability of data transmission, and reduce the delay and bandwidth consumption caused by retransmission.

[0069] As an optional implementation, in the technical solution provided in the above step S206, after the target data set of each business module in the cluster is migrated, the migration process can be evaluated and managed, including:

[0070] First, the migration time and network resource usage of the target data set of each business system in the cluster are determined during the migration process.

[0071] Specifically, the process of determining the migration duration and resource occupancy rate includes:

[0072] Determine the data volume and network resource information of the target data set of each business system in the cluster, wherein the network resource information includes at least one of the following: network resource utilization rate (including network bandwidth utilization rate, memory utilization rate, CPU utilization rate), bandwidth transmission rate, snapshot creation speed;

[0073] Determine the data transmission duration based on the data transmission rate and data volume (i.e. ), and determine the mirroring time based on the snapshot creation rate and data volume (i.e. ), the migration duration is determined by the data transmission duration and the image creation duration.

[0074] For example, the total amount of data to be migrated is 10240MB, the bandwidth transmission rate is 1.25MB / s, and the snapshot creation rate is 100MB / s. Then the data transmission time is 8192 seconds, or 2.27 hours, and the image creation time is 102 seconds, or 0.03 hours. Add the calculated data transmission time and image creation time together, and the migration time is 2.3 hours.

[0075] Next, the relationship between the migration duration and the preset first range threshold, and the relationship between the network resource usage rate and the preset second range threshold are determined, wherein:

[0076] When the migration duration is within the first range threshold and the network resource usage rate is within the second range threshold, it is determined that the migration process of the target data set of each service module in the cluster is in a normal state, that is, the cluster completes the migration process normally.

[0077] When the migration duration is not within the first range threshold, and / or the network resource utilization rate is not within the second range threshold, it is determined that the migration process of the target data set of each business module in the cluster is in an abnormal state, that is, the cluster has not completed the migration process. At this time, in order to ensure that the cluster can be migrated normally, it is necessary to take timely measures to intervene and adjust the migration process to ensure that the migration process can proceed smoothly. Specifically, the migration process of the target data set of each business module in the cluster can be processed accordingly. Generally, if the migration duration is too long, network resources can be appropriately increased, such as allocating more bandwidth or using higher-speed network equipment to accelerate data transmission. If the network resource utilization rate is too high, it may be necessary to adjust the migration priority, migrate some non-critical business system data to the secondary priority, and ensure that the data migration network resources of the critical business system are sufficient. In addition, if the network resource utilization rate is too low, this may mean that the current network resources are inefficiently utilized, and the network strategy can be adjusted according to the actual situation, such as increasing the number of concurrent migration tasks to improve resource utilization efficiency.

[0078] For example, if the first range threshold is set to 2-4 hours, and the second range threshold is set to 50%-80% network bandwidth utilization. Then, if the migration time of the cluster to which the trading system belongs is monitored to be 3 hours, and the network resource utilization rate is 65%, because these indicators are within the preset normal range threshold, it can be judged that the migration process of the cluster is normal and no additional intervention is required; and if the migration time of the cluster to which the risk management platform belongs is monitored to reach 5 hours, and the network resource utilization rate reaches 85%, this exceeds the preset range threshold, indicating that the migration process is abnormal. At this time, the processing operations taken can be: Increase network resources: such as by adjusting the network configuration, allocate additional bandwidth resources for the migration of the risk management platform to reduce the migration time. Adjust the migration priority: lower the migration priority of the risk management platform, migrate the business modules with lower dynamics first, and ensure that the highly dynamic business modules are migrated under more stable network conditions.

[0079] It should be noted that the first range threshold and the second range threshold are the maximum and minimum values ​​of the normal range of migration duration and network resource usage analyzed based on expert experience and performance data of historical migration processes (such as disk I / O performance). Therefore, this application does not impose specific restrictions on the setting of the first range threshold and the second range threshold, and the specific parameters need to be determined in combination with actual application scenarios and data feature analysis.

[0080] Based on the scheme defined by the above steps S202 to S206, it can be known that in the embodiment, by determining the similarity between each business module according to the initial data set of each business module in the business system, the business modules of the same type are clustered to obtain multiple clustering clusters, so as to identify and separate business modules with similar characteristics and dependencies, which provides a basis for subsequent dimensionality reduction migration; then, considering multiple dimensions such as link abnormality frequency, bandwidth peak ratio, data volume ratio in the business cycle, business cycle, delay and loss impact, the initial data set of each business module in each clustering cluster is reduced in dimension, which effectively reduces the data dimension and improves the migration efficiency. The reduced-dimensional data set is called the target data set, which is more compact and efficient while maintaining the integrity of information, and reduces the overhead of data storage and transmission; finally, the target data set of each business module in the clustering cluster is migrated, and the monitoring and optimization strategy of the migration duration and network resource utilization is combined to ensure the smooth progress of the migration process. The technical effect of efficient and secure migration of multiple business modules in the business system in the cloud environment is achieved, and the purpose of optimizing the migration process, reducing resource consumption, improving data processing efficiency and ensuring business continuity is achieved.

[0081] Example 2

[0082] Based on Embodiment 1 of the present application, an embodiment of a data migration device is also provided, and the device executes the data migration method of the above embodiment when it is running. Figure 4 is a schematic diagram of the structure of an optional data migration device according to an embodiment of the present application, such as Figure 4 As shown, the data migration device at least includes an acquisition module 42, a determination module 44 and a migration module 46, wherein:

[0083] An acquisition module 42 is used to acquire an initial data set of multiple business modules in the business system to be migrated, wherein the multiple business modules are functional modules obtained by splitting according to multiple functions provided by the business system;

[0084] A determination module 44 is used to determine the similarity between each business module according to the initial data set of each business module, and cluster the multiple business modules according to the similarity to obtain multiple clusters;

[0085] The migration module 46 is used for, for each cluster, performing dimensionality reduction processing on the initial data set of each business module in the cluster to obtain the target data set of each business module, and migrating the target data set of each business module in the cluster.

[0086] It should be noted that the various modules in the above-mentioned data migration device can be program modules (for example, a set of program instructions that implement a certain specific function) or hardware modules. For the latter, it can be expressed in the following forms, but is not limited to this: the expression form of each of the above-mentioned modules is a processor, or the functions of each of the above-mentioned modules are implemented by a processor.

[0087] Example 3

[0088] According to an embodiment of the present application, a non-volatile storage medium is further provided, in which a program is stored, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the data migration method in Example 1.

[0089] Optionally, the device where the non-volatile storage medium is located implements the following steps by running the program: obtaining initial data sets of multiple business modules in the business system to be migrated, wherein the multiple business modules are functional modules obtained by splitting according to multiple functions provided by the business system; determining the similarity between each business module based on the initial data sets of each business module, and clustering the multiple business modules according to the similarity to obtain multiple clustering clusters; for each clustering cluster, performing dimensionality reduction processing on the initial data sets of each business module in the clustering cluster to obtain a target data set of each business module, and migrating the target data set of each business module in the clustering cluster.

[0090] According to an embodiment of the present application, a computer program product is also provided, which includes a stored computer program, wherein the computer program implements the data migration method in Example 1 when executed by a processor.

[0091] Optionally, the computer program executes the following steps: obtaining initial data sets of multiple business modules in the business system to be migrated, wherein the multiple business modules are functional modules obtained by splitting according to multiple functions provided by the business system; determining the similarity between the business modules based on the initial data sets of each business module, and clustering the multiple business modules according to the similarity to obtain multiple clustering clusters; for each clustering cluster, performing dimensionality reduction processing on the initial data sets of each business module in the clustering cluster to obtain the target data sets of each business module, and migrating the target data sets of each business module in the clustering cluster.

[0092] According to an embodiment of the present application, a processor is also provided, which is used to run a program, wherein the data migration method in Example 1 is executed when the program is running.

[0093] Optionally, the following steps are executed when the program is running: obtaining initial data sets of multiple business modules in the business system to be migrated, wherein the multiple business modules are functional modules obtained by splitting according to multiple functions provided by the business system; determining the similarity between the business modules based on the initial data sets of each business module, and clustering the multiple business modules according to the similarity to obtain multiple clustering clusters; for each clustering cluster, performing dimensionality reduction processing on the initial data sets of each business module in the clustering cluster to obtain the target data sets of each business module, and migrating the target data sets of each business module in the clustering cluster.

[0094] According to an embodiment of the present application, an electronic device is also provided, wherein: Figure 5 is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present application, such as Figure 5 As shown, the electronic device includes one or more processors; a memory for storing one or more programs, which, when the one or more programs are executed by the one or more processors, enables the one or more processors to run the programs, wherein the programs are configured to execute the data migration method in the above-mentioned embodiment 1 when running.

[0095] Optionally, the processor is configured to execute the following steps through a computer program: obtaining initial data sets of multiple business modules in the business system to be migrated, wherein the multiple business modules are functional modules obtained by splitting according to multiple functions provided by the business system; determining the similarity between the business modules based on the initial data sets of each business module, and clustering the multiple business modules according to the similarity to obtain multiple clustering clusters; for each clustering cluster, performing dimensionality reduction processing on the initial data sets of each business module in the clustering cluster to obtain the target data sets of each business module, and migrating the target data sets of each business module in the clustering cluster.

[0096] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0097] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0098] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0099] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed over multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0100] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0101] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the relevant technology or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc. Various media that can store program codes.

[0102] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A data migration method, characterized in that: include: Acquire initial data sets of multiple business modules in the business system to be migrated, wherein the multiple business modules are functional modules obtained by splitting according to multiple functions provided by the business system; Determining the similarity between each of the business modules according to the initial data sets of each of the business modules, and clustering the multiple business modules according to the similarity to obtain multiple clusters; For each of the clusters, the initial data sets of each business module in the cluster are subjected to dimensionality reduction processing to obtain the target data sets of each business module, and the target data sets of each business module in the cluster are migrated.

2. The method according to claim 1, characterized in that Determining the similarity between the business modules according to the initial data sets of the business modules includes: For any two target business modules in the multiple business modules, a preset distance algorithm is used to determine the distance between the initial data sets of the two target business modules, and a relationship between the distance and a preset threshold value is determined; When the distance is lower than the threshold value, the distance is adjusted by using a Gaussian kernel function to obtain the similarity between the two target service modules; When the distance is not less than the threshold value, the distance is adjusted using a Gaussian kernel function and an exponential function to obtain the similarity between the two target service modules.

3. The method according to claim 2, characterized in that The multiple business modules are clustered according to the similarity to obtain multiple clusters, including: Determining the magnitude relationship between the similarity and a preset similarity threshold; When the similarity is not lower than a preset similarity threshold, determining that the two target service modules are of the same type; A plurality of business modules of the same type are aggregated to obtain a plurality of the clusters.

4. The method according to claim 1, characterized in that: After obtaining a plurality of clusters, the method further includes: For each of the clusters, determine the value of the cluster under a preset evaluation dimension, wherein the evaluation dimension includes at least one of the following: the number of business modules in the cluster, the importance of the business modules in the cluster in the business system, and the amount of data; The values ​​under the preset evaluation dimensions and the corresponding dimension weights are weighted and summed to obtain the migration priority of the clustering cluster.

5. The method according to claim 1, characterized in that: Performing dimensionality reduction processing on the initial data set of each business module in the cluster to obtain the target data set of each business module includes: A data matrix is ​​constructed with each business module in the cluster as a row and a preset dimension as a column, wherein the preset dimension includes at least one of the following: link abnormality frequency, bandwidth peak ratio, business cycle duration, interactive data volume in the business cycle, and transmission delay, and each matrix element in the data matrix represents an initial data set of the corresponding business module under the corresponding preset dimension; Determine a matrix product result of the data matrix and the corresponding transposed matrix, and divide the matrix product result by a difference obtained by subtracting 1 from the number of business modules in the cluster to obtain a covariance matrix; Performing eigendecomposition on the covariance matrix to obtain a plurality of eigenvalues ​​and a corresponding plurality of eigenvectors; Selecting a preset number of eigenvectors from the plurality of eigenvectors as column vectors according to the eigenvalues ​​to construct a projection matrix; The projection matrix is ​​used to perform dimensionality reduction processing on the data matrix to obtain a target data set of each business module in the cluster.

6. The method according to claim 1, characterized in that The method further comprises: Determine the migration duration and network resource usage rate of the target data set of each business system in the cluster during the migration process; Determine the magnitude relationship between the migration duration and a preset first range threshold, and between the network resource usage rate and a preset second range threshold; When the migration duration is within the first range threshold and the network resource usage rate is within the second range threshold, determining that the migration process of the target data set of each service module in the cluster is in a normal state; When the migration duration is not within the first range threshold, and / or the network resource utilization rate is not within the second range threshold, it is determined that the migration process of the target data set of each business module in the cluster is in an abnormal state, and corresponding processing operations are performed on the migration process of the target data set of each business module in the cluster, wherein the processing operations include at least one of the following: increasing network resources, adjusting migration priority.

7. The method according to claim 6, characterized in that Determining the migration duration and resource occupancy rate of the target data set of each business system in the cluster during the migration process includes: Determine the data volume and network resource information of the target data set of each business system in the cluster, wherein the network resource information includes at least one of the following: network bandwidth occupancy rate, memory occupancy rate, bandwidth transmission rate, snapshot creation rate; The data transmission duration is determined according to the data transmission rate and the data volume, the mirroring duration is determined according to the snapshot creation rate and the data volume, and the migration duration is determined by the data transmission duration and the mirroring duration.

8. A data migration device, characterized in that: include: An acquisition module, used to acquire initial data sets of multiple business modules in the business system to be migrated, wherein the multiple business modules are functional modules obtained by splitting according to multiple functions provided by the business system; A determination module, configured to determine the similarity between each of the business modules according to the initial data sets of each of the business modules, and cluster the multiple business modules according to the similarity to obtain multiple clusters; The migration module is used to perform dimensionality reduction processing on the initial data set of each business module in each cluster to obtain the target data set of each business module, and migrate the target data set of each business module in the cluster.

9. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a computer program, wherein the device where the non-volatile storage medium is located executes the data migration method according to any one of claims 1 to 7 by running the computer program.

10. A computer program product, characterized in that include: A computer program, wherein when the computer program is executed by a processor, the data migration method according to any one of claims 1 to 7 is implemented.

Images