Data migration method, electronic equipment and storage medium

Abstract

The invention discloses a data migration method in a cloud storage system, an electronic device and a storage medium, the cloud storage system comprises a first storage device and a second storage device, the performance of the first storage device is better than that of the second storage device, and the method comprises the following steps: writing shot current picture data into the first storage device; in response to the fact that the residence duration of the current picture data in the first storage device is equal to the preset duration, migrating the current picture data from the first storage device to a second storage device; and in response to the fact that the access frequency of the current picture data in the second storage device is greater than the preset frequency, returning the current picture data from the second storage device to the first storage device. Hierarchical storage of data in the cloud storage system is realized, and resources of different storage devices are reasonably utilized.

Description

Data migration methods, electronic devices, and storage media Technical Field

[0001] This application relates to the field of distributed data storage technology, and in particular to a data migration method, electronic device, and storage medium. Background Technology

[0002] In recent years, image data has experienced explosive growth, and image cloud storage technology has been widely applied and is developing rapidly. Image data is no longer just about meeting storage space requirements; the demands for image data storage performance and quality are also gradually increasing.

[0003] In cloud storage applications, storing all image data on a single high-performance storage medium is costly; storing all image data on a single low-performance storage medium results in low real-time write performance for the business. Summary of the Invention

[0004] To address the aforementioned issues, this application provides at least one data migration method, electronic device, and storage medium to enable dynamic migration of written image data for storage on different storage devices.

[0005] In a first aspect, this application provides a data migration method in a cloud storage system, the cloud storage system including a first storage device and a second storage device, wherein the performance of the first storage device is superior to that of the second storage device, the method comprising: writing captured current image data to the first storage device; migrating the current image data from the first storage device to the second storage device in response to the current image data residing in the first storage device for a duration greater than a preset duration; and migrating the current image data back from the second storage device to the first storage device in response to the current image data being accessed more frequently in the second storage device than a preset frequency.

[0006] In some embodiments, migrating the current image data from the first storage device to the second storage device includes: in response to the current remaining storage space of the first storage device being greater than the reserved storage space in the first storage device, determining a current loss value corresponding to migrating the current image data to the second storage device; obtaining a previous loss value corresponding to migrating the previous image data to the second storage device; adjusting the initial migration rate corresponding to determining the migration of the current image data to the second storage device based on the current loss value and the previous loss value to obtain a target migration rate; and migrating the current image data from the first storage device to the second storage device according to the target migration rate.

[0007] In some embodiments, adjusting the initial migration rate corresponding to migrating the current image data to the second storage device based on the current loss value and the previous loss value to obtain a target migration rate includes: reducing the initial migration rate to obtain the target migration rate in response to the difference between the current loss value and the previous loss value being greater than or equal to a preset value; and increasing the initial migration rate to obtain the target migration rate in response to the difference between the current loss value and the previous loss value being less than a preset value.

[0008] In some embodiments, determining the initial migration rate includes: acquiring historical indicator data of the cloud storage system and performing feature processing on the historical indicator data to obtain a historical indicator data feature stream; performing feature processing on the current indicator data corresponding to the current image data to obtain a current indicator data feature stream; determining the current business state when the current image data is written to the first storage device based on the historical indicator data feature stream and the current indicator data feature stream; and determining the initial migration rate based on the current business state; wherein the current business state includes a first business state, a second business state, and a third business state, and the initial migration rate corresponding to the first business state, the second business state, and the third business state increases sequentially.

[0009] In some embodiments, the initial migration rate corresponding to the first service state is the product of a first coefficient and a basic migration rate, the initial migration rate corresponding to the second service state is the product of a second coefficient and the basic migration rate, and the initial migration rate corresponding to the third service state is the product of a third coefficient and the basic migration rate; reducing or increasing the initial migration rate includes: keeping the basic migration rate unchanged, and reducing or increasing the first coefficient, the second coefficient, or the third coefficient.

[0010] In some embodiments, determining the preset value includes: obtaining the current input / output load corresponding to the current image data; in response to the current input / output load being greater than a preset input / output load, the preset value being less than a base value; in response to the current input / output load being less than the preset input / output load, the preset value being greater than the base value.

[0011] In some embodiments, migrating the current image data from the first storage device to the second storage device includes: in response to the current remaining storage space of the first storage device being less than or equal to the reserved storage space in the first storage device, using the reserved bandwidth of the cloud storage system, migrating other image data in the first storage device and the current image data to the second storage device in descending order of their residence time in the first storage device; wherein, during the migration of other image data in the first storage device and the current image data to the second storage device, the residence time corresponding to the image data written to the first storage device is simultaneously reduced to a preset proportion of the preset duration, and the residence time is gradually increased to be less than or equal to the preset duration according to a preset period.

[0012] In some embodiments, the reserved storage space in the first storage device is the maximum value among the following: the product of the estimated storage space corresponding to the first storage device and the first preset ratio for writing the current image data; and the product of the total storage space in the first storage device and the second preset ratio.

[0013] Secondly, this application provides a data migration apparatus, comprising: a data acquisition module configured to write captured current image data into a first storage device; a first migration module configured to migrate the current image data from the first storage device to a second storage device in response to the current image data residing in the first storage device for a duration equal to a preset duration; and a second migration module configured to migrate the current image data back from the second storage device to the first storage device in response to the current image data being accessed more frequently in the second storage device than a preset frequency.

[0014] Thirdly, this application provides an electronic device including a memory and a processor. The memory stores program instructions, and the processor executes the program instructions in the memory to implement the data migration method described in the first aspect above.

[0015] Fourthly, this application provides a non-volatile computer-readable storage medium having program instructions stored thereon, which, when executed by a processor, implement the data migration method described in the first aspect above.

[0016] The beneficial effects of this application are as follows: based on the status of the image data written to the first storage device, if the residence time of the current image data in the first storage device is equal to a preset time, the current image data is migrated from the first storage device to the second storage device; and based on the status of the image data written to the second storage device, if the access frequency of the current image data in the second storage device is greater than a preset frequency, the current image data is migrated back from the second storage device to the first storage device, thereby realizing dynamic migration of the written image data to store it on different storage devices according to different needs, balancing the cost and performance of storage devices, and improving resource utilization.

[0017] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0019] Figure 1 is a simplified structural diagram of the cloud storage system involved in this application; Figure 2 is a flowchart of a data migration method according to some embodiments of this application; Figure 3 is an architectural diagram of a network model according to some embodiments of this application; Figure 4 is a schematic diagram of a dual-path neural network architecture according to some embodiments of this application; Figure 5 is a partial flowchart of a data migration method according to some embodiments of this application; Figure 6 is a partial flowchart of a data migration method according to some embodiments of this application; Figure 7 is a schematic diagram of an application scenario according to some embodiments of this application; Figure 8 is a structural diagram of a data migration device according to some embodiments of this application; Figure 9 is a schematic diagram of the framework of an electronic device according to some embodiments of this application; Figure 10 is a schematic diagram of the framework of a non-volatile computer-readable storage medium according to some embodiments of this application. Detailed Implementation

[0020] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0021] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0022] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this document means two or more. Moreover, the term "at least one" in this document means any combination of at least two of any one or more of a plurality of objects. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0023] To enable those skilled in the art to better understand the technical solution of this application, the technical solution of this application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0024] To facilitate understanding of this application, the cloud storage system involved in this application will be described below.

[0025] Please refer to Figure 1, which is a simplified structural diagram of the cloud storage system involved in this application. The cloud storage system 10 includes a first storage device 11 and a second storage device 12. The first storage device 11 and the second storage device 12 can be connected by a wired connection, for example, through a fiber optic cable. The performance of the first storage device 11 is superior to that of the second storage device 12. For example, the sequential read / write speed and / or random read / write IOPS of the first storage device 11 are superior to those of the second storage device 12. The first storage device 11 can be a high-performance storage medium, and the second storage device 12 can be a low-performance storage medium. High-performance storage media include, for example, SSDs (Solid State Disks). Low-performance storage media include, for example, HHDs (Hybrid Hard Drives). In one application scenario, storing image data in a high-performance storage medium provides better real-time write performance. Storing image data in a low-performance storage medium allows for the storage of large amounts of image data, but real-time write performance cannot be guaranteed. The cost of high-performance storage media is relatively higher than that of low-performance storage media.

[0026] Furthermore, image data can be categorized into two types based on different preset conditions: one type is stored in high-performance storage media, and the other type is stored in low-performance storage media. In one example, image data is divided into hot data and cold data, with hot data stored in high-performance storage media and cold data stored in low-performance storage media. Hot data includes, for example, data written in real-time and data accessed frequently. Cold data includes, for example, data whose write time exceeds a preset duration and data accessed infrequently.

[0027] Please refer to Figure 2, which is a flowchart of a data migration method according to some embodiments of this application, including the following steps: Step S21: Write the captured current image data into a first storage device.

[0028] The current image data can be captured by a camera and written to the first storage device 11.

[0029] Step S22: In response to the current image data residing in the first storage device for a duration longer than a preset duration, the current image data is migrated from the first storage device to the second storage device.

[0030] The dwell time is the duration for which the current image data has remained in the first storage device 11 since it was written. For example, the current image data has been written to the first storage device 11 and has been dwelling for 1 hour. The preset duration is a preset value, such as 8 hours.

[0031] In one example, if the current image data resides in the first storage device 11 for a longer than a preset duration, the current image data is migrated from the first storage device 11 to the second storage device 12. In other words, the current image data is transformed from hot data to cold data, thus migrating it from a high-performance storage medium to a low-performance storage medium.

[0032] In another example, if the current image data resides in the first storage device 11 for a shorter period than a preset duration, the current image data is still considered hot data and continues to be stored in the first storage device 11.

[0033] Step S23: In response to the current image data being accessed more frequently in the second storage device than a preset frequency, the current image data is migrated back from the second storage device to the first storage device.

[0034] Access frequency refers to the number of times an image is accessed within a certain period of time. For example, the current image data is accessed 200 times in one hour. The preset frequency is a preset value, such as 200 times per hour.

[0035] In one example, the access frequency of the current image data stored in the second storage device 12 is greater than the preset frequency. The current image data is then migrated back from the second storage device 12 to the first storage device 11, that is, the storage of cold data in a low-performance storage medium is converted to the storage of hot data in a high-performance storage medium.

[0036] In another example, if the access frequency of the current image data in the second storage device 12 is less than the preset frequency, then the current image data is still considered cold data and continues to be stored in the second storage device 12.

[0037] In this embodiment, the status of the image data written to the first storage device 11 is used to determine whether the current image data resides in the first storage device 11 for a preset duration. If so, the current image data is migrated from the first storage device 11 to the second storage device 12. Furthermore, the status of the image data written to the second storage device 12 is used to determine whether the access frequency of the current image data in the second storage device 12 is greater than a preset frequency. If so, the current image data is migrated back from the second storage device 12 to the first storage device 11. This achieves dynamic migration of the written image data, allowing it to be stored on different storage devices according to different needs, balancing the cost and performance of the storage devices, and improving resource utilization.

[0038] As mentioned above, the residence time of the current image data in the first storage device 11 is equal to the preset duration. To facilitate understanding, the network model involved in determining whether to migrate the current image data from the first storage device 11 to the second storage device 12 will be described below.

[0039] As shown in Figure 3, the network model 30 includes an indicator monitoring layer 31, a migration decision layer 32, and a migration execution layer 33. The network model 30 can be deployed in the control section inside the storage device or in an external control device connected to the storage device.

[0040] The indicator monitoring layer 31 includes an indicator collector 311, which is used to collect relevant indicator data for subsequent determination of whether to perform migration.

[0041] The migration decision layer 32 determines whether to perform migration based on the corresponding loss value, and performs the migration of image data through the migration execution layer 33. The loss value is calculated using the loss function shown in formula (1).

[0042] (1) Where Loss is the loss value, , , and This represents the relevant metrics data when image data is migrated. The network bandwidth required for the migration is measured in MB / s. This refers to real-time input / output load, measured in IOPS (Input / Output Per Second). The written capacity of the first storage device 11 The total writable capacity in the first storage device 11 is collected by the indicator collector 311 in the indicator monitoring layer 31. The indicator collector 311 can capture the indicator data of the cloud storage system 10 in real time and store it in time-series form. The indicator data includes the indicator data of the first storage device 11 and the second storage device 12, or the indicator data of each storage device in the cluster. The cluster includes at least two first storage devices 11 and two second storage devices 12. The indicator collector 311 also collects CPU (Central Processing Unit) utilization, etc. , , These are the corresponding weighting coefficients, weighting coefficients. , , Initially, each factor is assigned an initial value. During the learning process of network model 30, as the loss function converges, the weight coefficients... , , Update using gradient descent.

[0043] Specifically, please refer to Figure 3. The transfer decision layer 32 may include a feature engine 321, a time decision maker 322, and a model monitor 323. The feature engine 321 can perform feature standardization processing on the raw data collected and stored in time series form, and construct corresponding time series feature engineering, including converting timestamps into multi-dimensional periodic features of sin (hour) and cos (hour), and constructing features by taking the mean and variance of different system indicators according to 1min, 5min, and 60min. Among them, the multi-dimensional periodic features are used to convert the hour information in the timestamp into a periodic pattern that the model can understand. The mean and variance of the system indicators are calculated from 1min, 5min, and 60min, i.e., at different time scales, to provide short-term memory for the model and understand whether the indicator data of the most recent 1min, 5min, and 60min fluctuate drastically.

[0044] The time decision-maker 322 includes a dual-path neural network architecture as shown in Figure 4 to determine the current business state. Specifically, the dual-path neural network architecture may include two input ends. For example, the first input end is used to input the historical indicator data feature stream, and the second end is used to input the current indicator data feature stream. In one example, the historical indicator data feature stream is input into a Long Short-Term Memory (LSTM) network, and the current indicator data feature stream is input into a fully connected layer. After attention weighting of the output data at the output end of the LSM network, it is fused with the current indicator data feature stream from the output end of the fully connected layer. The fused data is then integrated through the fully connected layer to make a decision, which can determine the current business state when the current image data is written to the first storage device 11.

[0045] Furthermore, the parameters in the network architecture can be optimized using the action probability policy and state value function as shown in Figure 4. For example, during the model training phase, the decision results can be fed back through the action probability policy and state value function to optimize the Long Short-Term Memory network. The action probability policy aims to maximize the cumulative reward, while the state value function aims to minimize the difference between the current value estimate and the target value estimate.

[0046] Model monitor 323 can monitor the decision results of the dual-path neural network architecture, including the corresponding prediction confidence. Based on the monitored prediction confidence, model monitor 323 provides the final prediction result. In one example, if the prediction confidence of the current business state corresponding to the monitored decision result is greater than a preset threshold, the current business state is determined. In another example, if the prediction confidence of the monitored decision result is less than a preset threshold, the current decision result is discarded, and the decision result from the previous decision when the prediction confidence was greater than the preset threshold is selected as the current decision result. The dual-path neural network architecture is then retrained, and the parameters in network model 30 are corrected, for example, using the PPO (Proximal Policy Optimization) algorithm. The preset threshold can be set according to actual design requirements and is not limited here.

[0047] In some embodiments, please refer to FIG5, which is a partial flowchart of a data migration method according to some embodiments of the present application. Specifically, migrating current image data from a first storage device 11 to a second storage device 12 includes the following steps: Step S51: In response to the fact that the current remaining storage space of the first storage device is greater than the reserved storage space in the first storage device, determine the current loss value corresponding to migrating the current image data to the second storage device.

[0048] The current remaining storage space of the first storage device 11 is the storage space remaining after the current image data is written. Reserved storage space is the storage space pre-reserved by the first storage device 11. When the current remaining storage space of the first storage device 11 is greater than the reserved storage space in the first storage device 11, the current loss value corresponding to migrating the current image data to the second storage device 12 is determined. The current loss value is calculated using the loss function shown in formula (1).

[0049] Step S52: Obtain the previous loss value corresponding to migrating the previous image data to the second storage device.

[0050] The previous image data is the image data before the current image data, and the previous image data is continuous with the current image data. The previous loss value is the previous loss value of the current loss value. That is to say, before each image data migration, the loss value of the image data migration is calculated, and the previous loss value has been determined before the previous image data migration. The previous loss value can be obtained, that is, the previous loss value is calculated by the loss function shown in formula (1).

[0051] Step S53: Based on the current loss value and the previous loss value, adjust the initial migration rate corresponding to the determination of the migration of the current image data to the second storage device to obtain the target migration rate.

[0052] The initial migration rate can be a preset migration rate for migrating image data, for example, 100MB / s. The target migration rate can be the actual rate at which the image data is migrated. Based on the current loss value and the previous loss value, the initial migration rate corresponding to the migration of the current image data to the second storage device 12 is adjusted to obtain the target migration rate. The first migration in the cloud storage system 10 is mandatory; that is, during the first migration, there is no previous loss value, and the migration is performed according to the initial migration rate.

[0053] Step S54: According to the target migration rate, migrate the current image data from the first storage device to the second storage device.

[0054] In some embodiments, based on the current loss value and the migration loss value, the initial migration rate corresponding to the determination of the migration of the current image data to the second storage device 12 is adjusted to obtain the target migration rate, including: in response to the difference between the current loss value and the previous loss value being greater than or equal to a preset value, the initial migration rate is reduced to obtain the target migration rate; in response to the difference between the current loss value and the previous loss value being less than a preset value, the initial migration rate is increased to obtain the target migration rate.

[0055] The difference can be an absolute value, which can characterize the degree of impact or cost of this migration on the cloud storage system 10, such as the bandwidth usage of the cloud storage system 10 and the load of the first storage device 11. The preset value can be set to an initial value according to the design requirements, and can be updated by the corresponding threshold update function in the migration decision layer 32, such as the threshold update function shown in formula (2).

[0056] (2) Among them, As a preset value, For real-time acquisition of input and output load. To preset the input and output load, during the first calculation using formula (2), It can be the initial value set according to the design requirements. When performing the second and subsequent calculations using formula (2), the preset value is continuously updated using formula (2) to adapt to the actual input and output load.

[0057] In one example, if the absolute value of the difference between the current loss value and the previous loss value is greater than or equal to a preset value, it indicates that the migration has a significant impact on the cloud storage system 10. For example, this migration may require more bandwidth from the cloud storage system 10, affecting the writing of real-time image data. Appropriately reducing the initial migration rate to obtain the target migration rate can reduce the cost of this migration in the cloud storage system 10.

[0058] In another example, if the absolute value of the difference between the current loss value and the previous loss value is less than a preset value, it indicates that the impact of this migration on the cloud storage system 10 is relatively small. For example, this migration requires less bandwidth from the cloud storage system 10 and will not affect the writing of real-time image data. The initial migration rate can be appropriately increased to obtain the target migration rate, allowing for faster migration.

[0059] Please refer to Figure 6, which is a partial flowchart of a data migration method according to some embodiments of this application. Determining the initial migration rate includes the following steps: Step S61: Obtain historical indicator data of the cloud storage system and perform feature processing on the historical indicator data to obtain a historical indicator data feature stream.

[0060] Historical indicator data can be the historical indicator data of the cloud storage system 10 when writing and migrating image data, collected by the indicator collector 311. Feature processing is performed on the historical indicator data, such as through the feature engine 321 shown in Figure 3, to obtain the historical indicator data feature stream.

[0061] Step S62: Perform feature processing on the current indicator data corresponding to the current image data to obtain the feature stream of the current indicator data.

[0062] For example, the feature engine 321 shown in Figure 3 performs feature processing on the current indicator data corresponding to the current image data to obtain the current indicator data feature stream.

[0063] Step S63: Based on the historical indicator data feature stream and the current indicator data feature stream, determine the current business status when the current image data is written to the first storage device.

[0064] Specifically, the historical indicator data feature stream is input into the Long Short-Term Memory network in the time decision maker 322, and the current image data is input into the fully connected layer to determine the current business state when the current image data is written to the first storage device 11. The current business state includes a first business state, a second business state, and a third business state. In one example, the first business state is a peak period, the second business state is a non-peak period, and the third business state is a low-peak period. A peak period indicates high current image data write traffic, a low-peak period indicates low current image data write traffic, and a non-peak period indicates that the current image data traffic falls between the two states.

[0065] In another example, the dual-path neural network architecture shown in Figure 4 determines the probability of the first business state as 0.6, the probability of the second business state as 0.2, and the probability of the third business state as 0.1. Since the probability of the first business state is the highest, the decision result is that the current cloud storage system 10 is in the first business state, i.e., the peak period.

[0066] Step S64: Determine the initial migration rate based on the current business status.

[0067] The initial migration rates for the first, second, and third business states increase sequentially. In one example, the first business state corresponds to the first initial migration rate, the second business state to the second initial migration rate, and the third business state to the third initial migration rate, with the first, second, and third initial migration rates increasing sequentially. This achieves low-speed migration during peak periods, medium-speed migration during off-peak periods, and high-speed migration during low-peak periods, thereby improving resource utilization.

[0068] Furthermore, the residence time of image data in the first storage device 11 varies depending on the business state. In one example, when the current business state is the first business state (peak period), the residence time of image data in the first storage device 11 is 8 hours; when the current business state is the second business state (off-peak period), the residence time of image data in the first storage device 11 is 6 hours; and when the current business state is the third business state (low-peak period), the residence time of image data in the first storage device 11 is 4 hours. This allows for setting different residence times under different business states to optimize resource utilization.

[0069] In some embodiments, the initial migration rate corresponding to the first service state is the product of a first coefficient and a basic migration rate, the initial migration rate corresponding to the second service state is the product of a second coefficient and a basic migration rate, and the initial migration rate corresponding to the third service state is the product of a third coefficient and a basic migration rate.

[0070] Wherein, the base migration rate is a preset value; the product of the first coefficient and the base migration rate is the first initial migration rate, corresponding to the first service state; the product of the second coefficient and the base migration rate is the second initial migration rate, corresponding to the second service state; and the product of the third coefficient and the base migration rate is the third initial migration rate, corresponding to the third service state. Assuming the base migration rate is v (unit: MB / s), the first, second, and third coefficients of the first, second, and third initial migration rates are 0.5, 1, and 1.5 respectively, i.e., the first, second, and third initial migration rates are 0.5v, v, and 1.5v respectively.

[0071] Furthermore, the initial migration rate can be decreased or increased. For example, while keeping the base migration rate constant, the first coefficient, the second coefficient, or the third coefficient can be decreased or increased.

[0072] Specifically, the base migration rate remains unchanged after being set. The initial migration rate is adjusted by decreasing or increasing the first, second, or third coefficient. In one example, the first coefficient is decreased to 0.4, the second to 0.9, and the third to 1.2, thus decreasing the corresponding initial migration rate for the three service states. In another example, the first coefficient is increased to 0.6, the second to 1.1, and the third to 1.6, thus increasing the corresponding initial migration rate for the three service states. Furthermore, in yet another example, in response to a difference between the current loss value and the migration loss value being greater than or equal to a preset value, the coefficient corresponding to the initial migration rate for the current service state is decreased to appropriately reduce the initial migration rate and obtain the target migration rate.

[0073] In some embodiments, determining the preset value includes: obtaining the current input / output load corresponding to the current image data; in response to the current input / output load being greater than the preset input / output load, the preset value being less than the base value; in response to the current input / output load being less than the preset input / output load, the preset value being greater than the base value.

[0074] The current input / output load corresponding to the current image data can be obtained through the indicator collector 311 in the indicator monitoring layer 31 shown in Figure 3.

[0075] The current input / output load is greater than the preset input / output load, indicating that the current business volume of the cloud storage system 10 is high. In one example, in response to... Greater than Then the output result of formula (2) Decrease, meaning the preset value is less than the base value.

[0076] The current input / output load is less than the preset input / output load, indicating that the current business volume of the cloud storage system 10 is high. In one example, in response to... Less than Then the output result of formula (2) Increase, meaning the preset value increases. In another example, in response to equal Then the output result of formula (2) That is, the preset value remains unchanged.

[0077] In some embodiments, migrating the current image data from the first storage device 11 to the second storage device 12 includes: in response to the current remaining storage space of the first storage device 11 being less than or equal to the reserved storage space in the first storage device 11, using the reserved bandwidth of the cloud storage system 10, migrating the other image data in the first storage device 11 and the current image data in the first storage device 11 to the second storage device 12 in descending order of their residence time in the first storage device 11.

[0078] Specifically, the indicator collector 311 collects indicator data from the remaining storage space of the first storage device 11 in real time to obtain the current remaining storage space. Reserved storage space ensures that the first storage device 11 can continuously write upcoming image data. Bandwidth is provided for the cloud storage system 10 to write and migrate image data. Reserved bandwidth means that when migrating data in the cloud storage system 10, the full bandwidth is not used; a portion of the bandwidth is reserved. For example, as shown in Figure 7, when the current storage space is less than the reserved storage space, 40% of the total bandwidth (reserved bandwidth) is used for image data migration. Other image data and the current image data reside in the first storage device 11 in descending order of their residence time; that is, other image data and the current image data have their own residence time recorded in the first storage device 11. For example, the indicator collector 311 records the residence time of different image data in the first storage device 11. The other image data and the current image data in the first storage device 11 are then migrated to the second storage device 12 in descending order of their residence time.

[0079] During the process of migrating other image data and current image data in the first storage device 11 to the second storage device 12, the residence time of the image data written to the first storage device 11 is reduced to a preset proportion of the preset duration, and the residence time is gradually increased to less than or equal to the preset duration according to a preset cycle.

[0080] The dwell time is reduced to a preset percentage of the preset duration. For example, if the preset duration is 8 hours and the preset percentage is half, the dwell time is reduced to 4 hours. A preset period, for example, 1 minute, is used. The dwell time is gradually increased to be less than or equal to the preset duration according to the preset period, for example, increasing by 1 minute every preset period. In one example, when migrating other image data and current image data from the first storage device 11 to the second storage device 12, the dwell time corresponding to the image data written to the first storage device 11 is reduced to half of 8 hours, i.e., 4 hours. During this period, the dwell time is gradually increased by 1 minute every 1 minute, so that the dwell time of the currently written image data is 4 hours, and the dwell time of the image data written when the preset duration is 4 hours and 1 minute is 4 hours and 1 minute. During this period, the dwell time is gradually increased to be less than or equal to the preset duration. If the dwell time is still less than the preset duration after this period, the dwell time continues to increase until it equals the preset duration.

[0081] Furthermore, since the cloud storage system 10 has reserved bandwidth, and the reserved bandwidth is used to migrate data when the current remaining storage space in the first storage device 11 is less than the reserved storage space, that is, when the access frequency of the current image data in the second storage device 12 is greater than the preset frequency, and when the reserved bandwidth in the cloud storage system 10 is not enabled, the current image data needs to be migrated back to the first storage device 11.

[0082] In some embodiments, the reserved storage space in the first storage device 11 is the maximum value among the following: writing the current image data to the product of the estimated storage space corresponding to the first storage device 11 and the first preset ratio; and the product of the total storage space in the first storage device 11 and the second preset ratio.

[0083] The estimated storage space is calculated by forecasting the storage space required for image data written to the first storage device 11 over a future period. A first preset ratio is used, for example, 1.5. In one example, the indicator collector 311 collects indicator data from the first storage device 11, obtaining a read / write rate of 100MB / s. The estimated storage space for the next hour is calculated as 100MB / s × 60 × 60 × 1.5 ≈ 527GB. Although the estimated storage space for the next hour is calculated, to avoid a sharp increase in image data traffic leading to an underestimation of the estimated storage space, a first preset ratio of 1.5 is used to improve the fault tolerance of the cloud storage system 10. In another example, the estimated storage space for the next two hours is calculated using a first preset ratio of 2.5. A second preset ratio is used, for example, 0.2, which is the total storage capacity of the first storage device 11 multiplied by 0.2.

[0084] The data migration method of this application is described below with reference to an application scenario. Please refer to Figure 7 for further details.

[0085] Step S71: Real-time monitoring.

[0086] The metric collector 311 in the cloud storage system 10 continuously monitors the cloud storage system 10, collecting metric data of the cloud storage system 10 every 5 seconds. When the current business state is determined to be the peak period of the first business state, the collector further judges the storage space of the first storage device 11 based on the collected metric data to determine the migration method. It can be understood that the initial migration rate corresponding to the first business state is the product of the basic migration rate and the first coefficient. Step S72: The current storage space is greater than the reserved storage space.

[0087] The current storage space of the first storage device 11 is obtained from the index data collected by the index collector 311 and compared with the reserved storage space. The estimated storage space is 100MB / s × 60 × 60 × 1.5 ≈ 527GB. The product of the total storage space of the first storage device 11 and the second preset ratio is 1000GB × 0.2 = 200GB. Since 527GB > 200GB, the reserved storage space is 527GB. The index data shows that the current storage space is 550GB. Therefore, the current storage space is greater than the reserved storage space.

[0088] Step S73: Calculate the loss value.

[0089] The collected indicator data is input into the loss function shown in formula (1), and the corresponding migration loss value is calculated. For example, the current loss value when writing the current image data and the previous loss value when writing the previous image data.

[0090] Step S74: Threshold update function.

[0091] The indicator collector 311 monitors the system indicators of the cloud storage system 10 in real time. The indicator collector 311 collects the indicator data of the cloud storage system 10 and inputs the input and output loads in the indicator data into the threshold update function shown in formula (2) to obtain the updated preset value. This enables dynamic adjustment of the preset value based on the actual load of the cloud storage system 10 to dynamically adjust the migration rate.

[0092] Step S75: The loss difference is less than the preset value.

[0093] The difference between the loss value calculated before the last migration and the loss value calculated before this migration is taken as the absolute value of the difference.

[0094] In response to the absolute value of the loss difference being greater than a preset value, proceed to step S76, reduce the first coefficient of the initial migration rate by 0.1 to reduce the initial migration rate to obtain the target migration rate, and perform migration according to the target migration rate.

[0095] In response to the absolute value of the loss difference being less than a preset value, proceed to step S77, increase the first coefficient of the initial migration rate by 0.1 to increase the initial migration rate to obtain the target migration rate, and perform migration according to the target migration rate.

[0096] Step S78: Enable reserved bandwidth migration.

[0097] The reserved bandwidth is 40% of the total bandwidth. When the judgment result of step S72 is yes, the bandwidth used for image data migration is the remaining 60% of the total bandwidth. In response to the current storage space being less than the reserved storage space, 40% of the total bandwidth is used to migrate all image data in the current first storage device 11. The migration order is as follows: all image data in the current first storage device 11 with 40% total bandwidth enabled are migrated to the second storage device 12 in descending order of their residence time. Furthermore, the residence time of newly written image data in the first storage device 11 is changed to half of the previous time, and the residence time of newly written image data is increased by 5 minutes every 1 minute until the previous preset residence time is restored. Note that S98 does not mean that it is executed last in this process, but rather that it is executed in response to the judgment result of step 92 being no.

[0098] Please refer to Figure 8, which is a schematic diagram of the structure of a data migration apparatus according to some embodiments of this application. The data migration apparatus 80 includes: a data acquisition module 81 configured to write captured current image data into a first storage device 11; a first migration module 82 configured to migrate the current image data from the first storage device 11 to a second storage device 12 in response to the current image data residing in the first storage device 11 for a duration equal to a preset duration; and a second migration module 83 configured to migrate the current image data back from the second storage device 12 to the first storage device 11 in response to the current image data being accessed more frequently in the second storage device 12 than a preset frequency.

[0099] The functions performed by each module are described in the same way as the data migration method, and will not be repeated here.

[0100] In this embodiment, the status of the image data written to the first storage device 11 is used to determine whether the current image data resides in the first storage device 11 for a preset duration. If so, the current image data is migrated from the first storage device 11 to the second storage device 12. Furthermore, the status of the image data written to the second storage device 12 is used to determine whether the access frequency of the current image data in the second storage device 12 is greater than a preset frequency. If so, the current image data is migrated back from the second storage device 12 to the first storage device 11. This achieves dynamic migration of the written image data, allowing it to be stored on different storage devices according to different needs, balancing the cost and performance of the storage devices, and improving resource utilization.

[0101] Please refer to Figure 9, which is a schematic diagram of the framework of an electronic device according to some embodiments of this application. The electronic device 90 includes a memory 91 and a processor 92 coupled to each other. The processor 92 is used to execute program instructions stored in the memory 91 to implement the steps in any of the above-described data synchronization method embodiments.

[0102] In a specific implementation scenario, electronic device 90 may include, but is not limited to, microcomputers and servers. In addition, electronic device 90 may also include mobile devices such as laptops and tablets, without limitation.

[0103] Specifically, processor 92 controls itself and memory 91 to implement the steps in any of the above data synchronization embodiments. Processor 92 can also be referred to as a CPU (Central Processing Unit). Processor 92 may be an integrated circuit chip with signal processing capabilities. Processor 92 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 92 can be implemented using integrated circuit chips.

[0104] In this embodiment, the status of the image data written to the first storage device 11 is used to determine whether the current image data resides in the first storage device 11 for a preset duration. If so, the current image data is migrated from the first storage device 11 to the second storage device 12. Furthermore, the status of the image data written to the second storage device 12 is used to determine whether the access frequency of the current image data in the second storage device 12 is greater than a preset frequency. If so, the current image data is migrated back from the second storage device 12 to the first storage device 11. This achieves dynamic migration of the written image data, allowing it to be stored on different storage devices according to different needs, balancing the cost and performance of the storage devices, and improving resource utilization.

[0105] Please refer to Figure 10, which is a schematic diagram of the framework of a non-volatile computer-readable storage medium according to some embodiments of this application. The non-volatile computer-readable storage medium 100 stores program instructions 101 that can be executed by the processor 92. The program instructions 101 are used to implement the steps of any of the above-described data migration method embodiments.

[0106] In this embodiment, the status of the image data written to the first storage device 11 is used to determine whether the current image data resides in the first storage device 11 for a preset duration. If so, the current image data is migrated from the first storage device 11 to the second storage device 12. Furthermore, the status of the image data written to the second storage device 12 is used to determine whether the access frequency of the current image data in the second storage device 12 is greater than a preset frequency. If so, the current image data is migrated back from the second storage device 12 to the first storage device 11. This achieves dynamic migration of the written image data, allowing it to be stored on different storage devices according to different needs, balancing the cost and performance of the storage devices, and improving resource utilization.

[0107] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0108] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0109] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0110] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0111] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A data migration method in a cloud storage system, characterized in that, The cloud storage system includes a first storage device and a second storage device, wherein the performance of the first storage device is superior to that of the second storage device. The system includes: writing captured current image data to the first storage device; migrating the current image data from the first storage device to the second storage device in response to the current image data residing in the first storage device for a duration exceeding a preset duration; and migrating the current image data back from the second storage device to the first storage device in response to the current image data being accessed more frequently in the second storage device than a preset frequency.

2. The data migration method according to claim 1, characterized in that, The step of migrating the current image data from the first storage device to the second storage device includes: in response to the current remaining storage space of the first storage device being greater than the reserved storage space in the first storage device, determining a current loss value corresponding to migrating the current image data to the second storage device; obtaining a previous loss value corresponding to migrating the previous image data to the second storage device; adjusting the initial migration rate corresponding to determining the migration of the current image data to the second storage device based on the current loss value and the previous loss value to obtain a target migration rate; and migrating the current image data from the first storage device to the second storage device according to the target migration rate.

3. The data migration method according to claim 2, characterized in that, Based on the current loss value and the previous loss value, the initial migration rate corresponding to the migration of the current image data to the second storage device is adjusted to obtain a target migration rate, including: in response to the difference between the current loss value and the previous loss value being greater than or equal to a preset value, the initial migration rate is reduced to obtain the target migration rate; in response to the difference between the current loss value and the previous loss value being less than a preset value, the initial migration rate is increased to obtain the target migration rate.

4. The data migration method according to claim 3, characterized in that, Determining the initial migration rate includes: acquiring historical indicator data of the cloud storage system and performing feature processing on the historical indicator data to obtain a historical indicator data feature stream; performing feature processing on the current indicator data corresponding to the current image data to obtain a current indicator data feature stream; determining the current business state when the current image data is written to the first storage device based on the historical indicator data feature stream and the current indicator data feature stream; and determining the initial migration rate based on the current business state; wherein the current business state includes a first business state, a second business state, and a third business state, and the initial migration rate corresponding to the first business state, the second business state, and the third business state increases sequentially.

5. The data migration method according to claim 4, characterized in that, The initial migration rate corresponding to the first service state is the product of a first coefficient and a basic migration rate; the initial migration rate corresponding to the second service state is the product of a second coefficient and the basic migration rate; and the initial migration rate corresponding to the third service state is the product of a third coefficient and the basic migration rate. Decreasing or increasing the initial migration rate includes: keeping the base migration rate constant while decreasing or increasing the first coefficient, the second coefficient, or the third coefficient.

6. The migration method according to claim 3, characterized in that, Determining the preset value includes: obtaining the current input / output load corresponding to the current image data; in response to the current input / output load being greater than the preset input / output load, the preset value being less than the base value; in response to the current input / output load being less than the preset input / output load, the preset value being greater than the base value.

7. The data migration method according to claim 1, characterized in that, Migrating the current image data from the first storage device to the second storage device includes: responding to the fact that the current remaining storage space of the first storage device is less than or equal to the reserved storage space in the first storage device, using the reserved bandwidth of the cloud storage system, and sequentially migrating the other image data in the first storage device and the current image data to the second storage device in descending order of their residence time in the first storage device; wherein, during the migration of the other image data in the first storage device and the current image data to the second storage device, the residence time corresponding to the image data written to the first storage device is simultaneously reduced to a preset proportion of the preset duration, and the residence time is gradually increased to less than or equal to the preset duration according to a preset period.

8. The data migration method according to any one of claims 2-7, characterized in that, The reserved storage space in the first storage device is the maximum value among the following: the product of the estimated storage space corresponding to the first storage device and the first preset ratio to which the current image data is written; and the product of the total storage space in the first storage device and the second preset ratio.

9. An electronic device, characterized in that, It includes a memory and a processor coupled to each other, the processor being used to execute program instructions stored in the memory to implement the data migration method according to any one of claims 1 to 8.

10. A non-volatile computer-readable storage medium storing program instructions thereon, characterized in that, When the program instructions are executed by the processor, they implement the data migration method according to any one of claims 1 to 8.

Images