Object storage data life cycle processing method and system
The method and system optimize object storage data lifecycle processing through a thread prediction model and deep reinforcement learning to address performance bottlenecks, enhancing processing speed and storage efficiency in large data environments.
Patent Information
- Application Number
- CN202411275215.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-12
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-09-12
AI Technical Summary
Existing object storage data lifecycle processing methods face performance bottlenecks due to sequential execution, leading to slow processing speeds and inefficient recycling of storage space, especially when dealing with large volumes of data, resulting in system instability and reduced availability.
A method and system that employs a multivariate linear regression algorithm to construct a thread prediction model, optimizing the number of concurrent threads based on historical parameter information, enabling full-link concurrent processing from reading to processing objects, and using a deep reinforcement learning algorithm to dynamically adjust task concurrency for efficient resource utilization.
Enhances the speed of lifecycle processing and object recycling, increases storage space reuse, and improves system stability and availability by optimizing thread concurrency, especially in large data scenarios.
Smart Images

Figure CN118796481B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of data storage, and in particular, relates to a method and system for processing object storage data lifecycle. Background Art
[0002] The existing lifecycle processing method is executed sequentially when acquiring objects, which has a performance bottleneck. The threads in the thread pool need to wait for the previous process to complete before concurrent processing. Therefore, even if the number of threads is increased, the processing speed cannot be further accelerated. When the number of objects in the bucket is large, this problem will be more prominent, resulting in the inability to recycle the storage space in time. When the storage space is insufficient, the system's IO performance will drop significantly, and even make the system unusable.
[0003] In addition, there is still much room for improvement in terms of how to increase the speed of lifecycle processing and object recycling, increase the reuse rate of storage space, and improve the stability and availability of the system when dealing with large amounts of data.
[0004] Therefore, it is necessary to provide a new object storage data lifecycle processing method and system to solve the above problems. Summary of the invention
[0005] The present invention aims to provide an object storage data lifecycle processing method and system to solve the technical problems in the prior art when there are a large number of objects in a storage bucket, such as slow processing speed and inability to recycle storage space in time, or even unusable related systems, due to untimely resource use and recovery, and how to improve the reuse rate of storage space when there is a large amount of data. The technical problems to be solved by the present invention are achieved through the following technical solutions.
[0006] The first aspect of the present invention proposes a method for processing the life cycle of object storage data, comprising: step S101: obtaining the following historical parameter information marked with a thread processing completion time label, and establishing a training data set: file size, resource utilization, total task amount, system configuration, and number of concurrent threads; step S102: using a multivariate linear regression algorithm to construct a thread prediction model, and using the training data set to train the thread prediction model; the training process of the thread prediction model also includes fine-tuning the thread prediction model using the number of concurrent threads in the historical parameter information as a learning parameter to obtain an optimal solution for the number of concurrent threads; step S103: inputting the current file size, resource utilization, total task amount, and system configuration parameters into the trained thread prediction model, and outputting matching thread prediction models. Step S104: Based on the output thread pool and the number of concurrent threads, execute each thread concurrently, determine whether there is an index shard of the bucket in the current thread, and when it is determined that there is an index shard of the bucket, obtain the next index shard; Step S105: Further determine whether there is a target object in the index shard of the bucket, and when there is a target object, obtain a specific number of target objects; if no target object is found, repeat step S103 until it is determined that there is an index shard of the bucket and a specific number of target objects are obtained; Step S106: Submit the obtained specific number of target objects to the pending queue, and repeatedly determine whether there are target objects that have not been determined to be expired in the pending queue until all target objects have been determined to be expired.
[0007] The second aspect of the present invention proposes an object storage data lifecycle processing system, which adopts the object storage data lifecycle processing method described in the first aspect of the present invention. The object storage data lifecycle processing system includes: a data acquisition module, which is used to obtain the following historical parameter information marked with a thread processing completion time label to establish a training data set: file size, resource utilization, total task volume, system configuration, and number of concurrent threads; a model construction module, which adopts a multivariate linear regression algorithm to construct a thread prediction model, and uses the training data set to train the thread prediction model; the training process of the thread prediction model also includes using the number of concurrent threads in the historical parameter information as a learning parameter to fine-tune the thread prediction model to obtain the optimal solution for the number of concurrent threads; a model determination module, which is used to convert the current file size, resource utilization, total task volume, and system configuration parameters into a model prediction model. The trained thread prediction model is inputted, and the matching thread pool and the number of concurrent threads are outputted; the first judgment processing module concurrently executes each thread based on the output thread pool and the number of concurrent threads, and judges whether there is an index shard of the storage bucket in the current thread. When it is determined that there is an index shard of the storage bucket, the next index shard is obtained; the second judgment processing module is used to further judge whether there is a target object in the index shard of the storage bucket, and when there is a target object, a specific number of target objects are obtained; if no target object is found, step S103 is repeated until it is determined that there is an index shard of the storage bucket and a specific number of target objects are obtained; the repeated determination module submits the obtained specific number of target objects to the pending queue, and repeatedly determines whether there are target objects in the pending queue that have not been judged to be expired, until all target objects have been judged to be expired.
[0008] The third aspect of the present invention provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in the first aspect of the present invention.
[0009] A fourth aspect of the present invention provides a computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method described in the first aspect of the present invention.
[0010] The embodiments of the present invention include the following advantages:
[0011] Compared with the prior art, the present invention adjusts the processing flow of the object storage system life cycle to achieve full-link concurrent processing from reading objects to processing objects, thereby solving the performance bottleneck of existing methods. When the amount of data is large, increasing the number of concurrent threads can improve the speed of life cycle processing and object recycling, thereby increasing the reuse rate of storage space and improving the stability and availability of the system when dealing with large amounts of data.
[0012] In addition, a deep reinforcement learning algorithm is used to determine the task concurrency of the current time period based on historical parameter information and the current total number of tasks, and to learn the best task, that is, the optimal solution concurrency, so that it can be completed in the shortest time and maximize the use of server resources. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] Figure 1 is a flowchart of an example of the object storage data lifecycle processing method of the present invention;
[0014] Figure 2 It is a schematic diagram of the network structure of the thread prediction model in the object storage data life cycle processing method of the present invention;
[0015] Figure 3 It is a partial flow chart of the object storage data life cycle processing method of the present invention;
[0016] Figure 4 It is a structural block diagram of the object storage data lifecycle processing system of the present invention;
[0017] Figure 5 is a schematic structural diagram of an electronic device embodiment according to the present invention;
[0018] Figure 6 It is a schematic diagram of the structure of a computer-readable medium embodiment according to the present invention. DETAILED DESCRIPTION
[0019] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. In the present invention, the upper surface of the object in the drawings is the upper surface, the lower surface of the object in the drawings is the lower surface, the vertical direction in the drawings is the vertical direction, and the direction perpendicular to the vertical direction is the horizontal direction. The present invention will be described in detail below with reference to the drawings and in combination with the embodiments.
[0020] The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.
[0021] In view of the above problems, the present invention proposes a method for processing the life cycle of object storage data. By adjusting the processing flow of the life cycle, full-link concurrent processing from reading objects to processing objects is realized, which solves the performance bottleneck of existing methods, improves the speed of life cycle processing and object recycling, improves the reuse rate of storage space, and improves stability and availability when large amounts of data are present. At the same time, a deep learning reinforcement algorithm is used to dynamically adjust the number of concurrency to optimize task completion efficiency and resource utilization.
[0022] It should be noted that in the present invention, objects in the computing environment are usually used by threads through the object life cycle, and are destroyed at the end of the object life cycle to reclaim the computing resources used by the object. The object storage data life cycle refers to the calculation of the expiration time of the target object according to the configuration rules (for example, the validity period is 3 days) when the target object is stored in the system, and the expiration time is written into the metadata of the index shard. In the regularly executed life cycle processing thread, it is checked whether the expiration time of the target object has been reached. If it has been reached, the target object is handled by strategies such as deletion, transfer or downgrade storage (the specific strategy needs to be adaptively adjusted according to the actual situation).
[0023] Example 1
[0024] Refer to the following Figures 1 to 3 , the contents of the present invention will be described in detail.
[0025] Figure 1 It is a flowchart of an example of the object storage data lifecycle processing method of the present invention. Figure 2 It is a schematic diagram of the network structure of the thread prediction model in the object storage data lifecycle processing method of the present invention.
[0026] like Figure 1 As shown, in step S101, the following historical parameter information marked with thread processing completion time tags is obtained to establish a training data set.
[0027] Specifically, historical parameter information of different tasks (for example, lifecycle monitoring tasks, which are tasks that specifically check whether the lifecycle of the managed target object has expired) and different historical time periods (for example, three months or six months forward from the current time, etc.) is obtained. The historical parameter information includes file size, resource utilization, total task volume, system configuration, and number of concurrent threads.
[0028] In a specific implementation, the historical parameter information marked with the time label of each thread processing completion is used to establish a training data set for subsequent model training. The historical parameter information includes file size, resource utilization, total task volume, system configuration (such as CPU information, memory information), and number of concurrent threads.
[0029] It should be noted that the above are only optional implementations and should not be construed as limitations of the present invention.
[0030] Specifically, for example, ten thousand pieces of parameter information are collected as samples, wherein the data set is divided at a ratio of 8:1, for example, 8000 pieces are used as training samples (ie, training samples of the training data set), and 2000 pieces are used as a validation set.
[0031] Determine the following historical parameter information: file size, resource utilization, total number of tasks, system configuration (cpu, memory), number of concurrency. Recorded as uppercase X (x1, x2, x3, x4, x5), lowercase x represents vector elements...., the final completion time is S, which is used as the training label. Thus, the training data set is obtained.
[0032] It should be noted that the above is only described as an optional example and should not be understood as a limitation to the present invention.
[0033] Next, in step S102, a multivariate linear regression algorithm is used to construct a thread prediction model, and the thread prediction model is trained using the training data set; the training process of the thread prediction model also includes fine-tuning the thread prediction model using the number of concurrent threads in the historical parameter information as a learning parameter to obtain an optimal solution for the number of concurrent threads.
[0034] Specifically, a multivariate linear regression algorithm is used to construct a thread prediction model.
[0035] like Figure 2 As shown, the network structure of the thread prediction model includes four layers, specifically a first network layer, a second network layer, a third network layer and an output layer.
[0036] For the first layer of the network, for example, a matrix of (5 rows * 128 columns), the output is 8000 * 128 (number of parameters), and the expression is: Yn = BN(relu(β0 + β1Xn + β2Xn + ... + β128*Xn)).
[0037] For the second layer network, for example, the matrix is (128 rows * 256 columns), the output is 8000 * 256 (number of parameters), and the expression is: Zn = BN(relu(θ0 + θ1Yn + θ2Yn + ... + θ256*Yn)).
[0038] For the third layer network, for example, the matrix is (256 rows * 16 columns), the output is 8000 * 16 (number of parameters), and the expression is: Kn = BN(relu(α0 + α1Zn + α2Zn + ... + α16*Zn)).
[0039] For the output layer, for example, a matrix of (16 rows * 1 column), the output is 8000 * 1 (number of parameters), and the expression is: On = BN(relu(t0 + t1Kn )).
[0040] The output expressions of the first network layer, the second network layer, the third network layer and the output layer are all mathematical formulas from input to output obtained through machine learning, where the parameters involved are the values to be calculated by machine learning, and the mathematical formulas from input to output are established through machine learning in each network layer. Then the current task situation and parameter information are brought in to calculate the number of concurrent threads of the current task, and the number of concurrent threads of the current task is used to start the corresponding number of threads in the thread pool, and concurrently perform life cycle management tasks, such as checking whether the target object is expired. If expired, take corresponding measures to deal with it, including deletion, transfer or downgrade storage.
[0041] It should be noted that in the present invention, the thread prediction model adopts a three-layer linear perception network structure, and a fully connected layer is used after the three-layer network to output, for example, a one-dimensional value y, which represents the actual time consumed by the tasks in this batch. The error between the model output y and the label S of the training sample is used as the cost.
[0042] In addition, in other implementations, for example, multi-dimensional [y1, y2, ..., yn] may be output to represent the time series actually consumed by multiple threads of the batch of tasks. The above is only described as an optional example and cannot be understood as a limitation of the present invention.
[0043] Next, the training process of the thread prediction model will be explained.
[0044] In a specific embodiment, in the model training process, two learning stages, a first learning stage and a second learning stage, are completed. In the first learning stage, the model parameters of the thread prediction model are trained using historical parameter information and historical thread consumption time. After completing the first learning stage, the second learning stage is started. In the second learning stage, the number of concurrent threads in the historical parameter information is used as a learning parameter to retrain the thread prediction model, and the number of concurrent threads for the optimal solution is obtained using model inverse learning.
[0045] For example, use MSE_LOSS to calculate the model loss error, that is, the above cost.
[0046] In the first learning stage, the thread prediction model is trained using the training data obtained in step S101. Specifically, the learning rate is initialized to, for example, 0.01. The optimizer uses the SGD gradient descent method to reduce the error.
[0047] Specifically, 8000 training samples are input into the thread prediction model, and in the first learning phase, the model is iteratively trained a first specified number of times (eg, 5000 times).
[0048] Further comprising the following steps:
[0049] Step S201: During iterative training, the full validation set is used for validation every 100 times; the model is saved every 1000 times.
[0050] Step S202: When the loss value of the model training is less than or equal to 0.08 (i.e., the training stop limit condition), the model training of the first learning stage is stopped, that is, the first learning stage training is qualified and serves as the final model parameters of the three-layer perception network (specifically including model parameters β, θ, α, t).
[0051] Step S203: When the thread prediction model does not meet the training stop limit condition (or does not achieve the optimal performance of the model, for example, the model accuracy is lower than the specified value), the above steps S201 and S202 are repeated.
[0052] In the second learning phase, the model is iteratively trained a second specified number of times. The first specified number of times is greater than the second specified number of times.
[0053] The second learning phase is the reasoning part, which uses the model to reversely learn the number of concurrent threads (also referred to as "concurrency" in this example). The model parameters β, θ, α, and t are frozen, and the inputs (x1, x2, x3, and x4) are frozen. The number of concurrent threads x5 is used as a learnable parameter, and the network is retrained for the second specified number of times (for example, 1,000 times) to obtain the optimal number of concurrent threads.
[0054] In the second learning stage, the model input data: X (x1, x2, x3, x4, x5), a total of 1000, recorded as 1000 * 5. Through the three-layer perception network Yn, Zn, Kn, plus a fully connected layer to get the final output On. Use MSE to find the minimum value.
[0055] Cost = 0.5 * (On) 2
[0056] Among them, Cost represents the error value between the actual consumption time and the predicted consumption time; On represents the output value of the thread prediction model.
[0057] Use the gradient descent method SGD to train x5 (the number of concurrent threads), and retrain the network for the second specified number of times (for example, 1000 times) to obtain the number of concurrent threads that results in the optimal solution.
[0058] Through the segmented training method of the above two learning stages, the activation function is used between each layer to delinearize, making the interpretation ability stronger after the spatial transformation. And BN normalization is followed by the activation function to make the distribution concentrated in the [0,1] interval. After the distribution is concentrated, the model convergence ability can be accelerated, and the gradient explosion caused by excessive values can be effectively avoided, thereby obtaining a thread prediction model with optimized model parameters and improved accuracy.
[0059] It should be noted that the above is only described as an optional example and should not be understood as a limitation to the present invention.
[0060] Next, in step S103, the current file size, resource utilization, total number of tasks, and system configuration parameters are input into the trained thread prediction model, and the matching thread pool and number of concurrent threads are output.
[0061] Specifically, the client's current file size (for example, 5M), resource utilization (for example, 20%), total number of tasks (for example, 900), and system configuration parameters (for example, kernel, memory, etc.) are obtained, and the obtained data is vectorized to obtain [x1, x2, x3, x4], which is input into the trained thread prediction model to output the matching thread a and the number of concurrent threads L.
[0062] Next, in step S104, based on the output thread pool and the number of concurrent threads, each thread is executed concurrently to determine whether there is an index fragment of the bucket in the current thread. When it is determined that there is an index fragment of the bucket, the next index fragment is obtained.
[0063] Specifically, according to the determined thread a and the number of concurrent threads L (for example, 6), the six threads are executed concurrently.
[0064] In a specific embodiment, Figure 3 As shown, the bucket index slice information (that is, the index slice information of the bucket) is obtained, and the six threads in the thread pool are started.
[0065] Next, it is determined whether there are bucket index shards in the current thread. If it is determined that there are bucket index shards, the next index shard is obtained. If it is determined that there are no bucket index shards, the entire process ends.
[0066] If it is determined that there is a bucket index slice, the next index slice is obtained, and then the process proceeds to the judgment step S105.
[0067] In another optional implementation, before concurrently executing each thread, the index shard information of the current bucket is obtained, and it is determined whether it is the first index shard in the current bucket according to the index shard information.
[0068] It is determined whether there is a first index slice of the current bucket in the current thread. When it is determined that there is a first index slice of the current bucket, the next index slice is obtained for subsequent determination.
[0069] In an optional implementation, when the number of threads in the output thread pool is greater than or equal to the number of concurrent threads output, threads of the number of concurrent threads are obtained from the thread pool.
[0070] In the case of taking some or all threads from the thread pool, when the number of threads in the output thread pool is less than the number of concurrent threads output, all threads are taken from the thread pool and the corresponding number of threads are newly created. For example, if there are five concurrent threads and there are three threads in the thread pool, two threads are newly created.
[0071] Optionally, when there are no available threads in the output thread pool, or the number of threads is less than the calculated six, a corresponding number of threads are obtained from the thread pool and re-established.
[0072] It should be noted that the above is only described as an optional example and should not be understood as a limitation to the present invention.
[0073] Next, in step S105, it is further determined whether there is a target object in the index shard of the bucket. If there is a target object, a specific number of target objects are obtained. If no target object is found, step S103 is repeated until it is determined that there is an index shard of the bucket and a specific number of target objects are obtained.
[0074] exist Figure 3 In the example, it is further determined whether the target object exists in the index shard of the current bucket.
[0075] When there are target objects, 900 target objects are obtained, wherein the target objects represent audio files, document files, and picture files that are managed by life cycle rules. Specifically, if there are (corresponding to Figure 3 If the target object does not exist (the corresponding Figure 3 If the query returns to the step of determining whether there is a bucket index slice in the current thread, then the query returns to the step of determining whether there is a bucket index slice in the current thread.
[0076] It should be noted that in this example, the target object refers to a file stored in the object storage system, including but not limited to audio and video, pictures, documents, etc., and the target object is managed by lifecycle rules, and the management information is recorded in the metadata of the index shard. However, it is not limited to this, and the above is only an optional example and cannot be understood as a limitation of the present invention.
[0077] In another optional implementation, it is further determined whether the target object exists in the first index fragment of the current storage bucket, and when the target object exists, a specific number of target objects are obtained.
[0078] Optionally, the specific number is in the range of 500 to 1200, for example, 900.
[0079] It should be noted that the above is only described as an optional example and should not be understood as a limitation to the present invention.
[0080] Step S106, submitting the acquired specific number of target objects to a queue to be processed, and repeatedly determining whether there are target objects in the queue to be processed that have not been determined to be expired, until all target objects have been determined to be expired.
[0081] exist Figure 3 In the example, the 900 target objects (for example, the object objects in the 900 bucket_index shards) are submitted to the processing queue (corresponding to Figure 3 900 target objects are submitted to the queue). Get the target object (i.e., object object) from the queue to be processed.
[0082] Determine whether each object is expired. If it is expired (corresponding to Figure 3 If the value is "yes" in the query, the expired object will be deleted. Figure 3 If the result is "No" in the processing result, the target object will continue to be obtained from the queue to be processed, and the obtained target object will be judged to be expired. That is, it is repeatedly determined whether there is a target object in the queue to be processed that has not been judged to be expired until all target objects have been judged to be expired.
[0083] Optionally, after deleting the expired target object, it is also necessary to determine whether there are any target objects in the queue to be processed (corresponding to Figure 3 "Is there any target object in the queue"), if yes (corresponding to Figure 3 If the target object is not available (the corresponding Figure 3 If "No" is shown in the figure), it returns to the step of determining whether there is a bucket index slice in the current thread.
[0084] It should be noted that the above is only described as an optional example and should not be understood as a limitation to the present invention.
[0085] In addition, the drawings are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the drawings do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.
[0086] Compared with the prior art, the present invention adjusts the processing flow of the object storage system life cycle to achieve full-link concurrent processing from reading objects to processing objects, thereby solving the performance bottleneck of existing methods. When the amount of data is large, increasing the number of concurrent threads can improve the speed of life cycle processing and object recycling, thereby increasing the reuse rate of storage space and improving the stability and availability of the system when dealing with large amounts of data.
[0087] In addition, a deep reinforcement learning algorithm is used to determine the task concurrency in the current time period based on historical parameter information and the current total number of tasks, and to learn the best task, that is, the optimal solution concurrency, so that it can be completed in the shortest time and maximize the use of server resources.
[0088] Example 2
[0089] The following is a system embodiment of the present invention, which can be used to implement the method embodiment of the present invention. For details not disclosed in the system embodiment of the present invention, please refer to the method embodiment of the present invention.
[0090] Figure 4 It is a structural diagram of an example of the object storage data lifecycle processing system according to the present invention.
[0091] Reference Figure 4 The second aspect of the present disclosure provides an object storage data lifecycle processing system 400, which adopts the object storage data lifecycle processing method described in the first aspect of the present invention.
[0092] The object storage data lifecycle processing system 400 includes: a data acquisition module 410 , a model determination module 420 , a first judgment processing module 430 , a second judgment processing module 440 and a duplication determination module 450 .
[0093] In a specific implementation, the data acquisition module 410 is used to obtain the following historical parameter information marked with thread processing completion time tags to establish a training data set: file size, resource utilization, total task volume, system configuration, and number of concurrent threads; the model construction module uses a multivariate linear regression algorithm to construct a thread prediction model, and uses the training data set to train the thread prediction model; the training process of the thread prediction model also includes fine-tuning the thread prediction model using the number of concurrent threads in the historical parameter information as a learning parameter to obtain the optimal solution for the number of concurrent threads. The model determination module 420 is used to input the current file size, resource utilization, total task volume, and system configuration parameters into the trained thread prediction model, and output a matching thread pool and number of concurrent threads. The first judgment processing module 430 concurrently executes each thread based on the output thread pool and the number of concurrent threads, and judges whether there is an index shard of the bucket in the current thread. When it is determined that there is an index shard of the bucket, the next index shard is obtained; the second judgment processing module 440 is used to further judge whether there is a target object in the index shard of the bucket, and when there is a target object, a specific number of target objects are obtained; if no target object is found, step S103 is repeated until it is determined that there is an index shard of the bucket and a specific number of target objects are obtained; the repeated determination module 450 submits the obtained specific number of target objects to the pending queue, and repeatedly determines whether there are target objects in the pending queue that have not been judged to be expired, until all target correspondences have been judged to be expired.
[0094] In the model training process, two learning stages, a first learning stage and a second learning stage, are completed. In the first learning stage, the model parameters of the thread prediction model are trained using historical parameter information and historical thread consumption time. After completing the first learning stage, the second learning stage begins. In the second learning stage, the number of concurrent threads in the historical parameter information is used as a learning parameter to retrain the thread prediction model, and the number of concurrent threads for the optimal solution is obtained using model inverse learning.
[0095] Specifically, in the first learning stage, the model is iteratively trained a first specified number of times; in the second learning stage, the model is iteratively trained a second specified number of times; and the first specified number of times is greater than the second specified number of times.
[0096] Before concurrently executing each thread, obtain the index shard information of the current bucket, and determine whether it is the first index shard in the current bucket according to the index shard information;
[0097] It is determined whether there is a first index slice of the current bucket in the current thread. When it is determined that there is a first index slice of the current bucket, the next index slice is obtained for subsequent determination.
[0098] According to an optional implementation manner, it is further determined whether the target object exists in the first index fragment of the current storage bucket, and when the target object exists, a specific number of target objects are obtained.
[0099] According to an optional implementation, when target objects exist, obtaining a specific number of target objects includes: when target objects exist, obtaining a specific number (for example, 900) of target objects, wherein the target objects represent audio files, document files, and picture files managed by lifecycle rules.
[0100] According to an optional implementation, when the number of threads in the output thread pool is greater than or equal to the number of concurrent threads output, threads of the number of concurrent threads are obtained from the thread pool.
[0101] When the number of threads in the output thread pool is less than the number of concurrent threads output, all threads are obtained from the thread pool and a corresponding number of threads are newly created.
[0102] Compared with the prior art, the present invention adjusts the processing flow of the object storage system life cycle to achieve full-link concurrent processing from reading objects to processing objects, thereby solving the performance bottleneck of existing methods. When the amount of data is large, increasing the number of concurrent threads can improve the speed of life cycle processing and object recycling, thereby increasing the reuse rate of storage space and improving the stability and availability of the system when dealing with large amounts of data.
[0103] In addition, a deep reinforcement learning algorithm is used to determine the task concurrency of the current time period based on historical parameter information and the current total number of tasks, and to learn the best task, that is, the optimal solution concurrency, so that it can be completed in the shortest time and maximize the use of server resources.
[0104] Figure 5 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
[0105] like Figure 5 As shown, the electronic device is presented in the form of a general computing device. The processor may be one or more and work in coordination. The present invention does not exclude distributed processing, that is, the processor may be dispersed in different physical devices. The electronic device of the present invention is not limited to a single entity, but may also be the sum of multiple physical devices.
[0106] The memory stores a computer executable program, which is usually a machine-readable code. The computer-readable program can be executed by the processor to enable the electronic device to perform the method of the present invention, or at least part of the steps in the method.
[0107] The memory includes a volatile memory, such as a random access memory unit (RAM) and / or a cache memory unit, and may also be a non-volatile memory, such as a read-only memory unit (ROM).
[0108] Optionally, in this embodiment, the electronic device further includes an I / O interface, which is used for the electronic device to exchange data with an external device. The I / O interface can represent one or more of several types of bus structures, including a storage unit bus or a storage unit controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of a variety of bus structures.
[0109] It should be understood that Figure 5 The electronic device shown is only an example of the present invention, and the electronic device of the present invention may also include elements or components not shown in the above examples. For example, some electronic devices also include display units such as display screens, and some electronic devices also include human-computer interaction elements such as buttons, keyboards, etc. As long as the electronic device can execute the computer-readable program in the memory to implement the method of the present invention or at least part of the steps of the method, it can be considered as an electronic device covered by the present invention.
[0110] Through the above description of the implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by combining software with necessary hardware. Figure 6 As shown, the technical solution according to the embodiment of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes a number of commands to enable a computing device (which can be a personal computer, a server, or a network device, etc.) to execute the above method according to the embodiment of the present invention.
[0111] The software product may be any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.
[0112] The computer readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with a command execution system, device, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.
[0113] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).
[0114] The computer-readable medium carries one or more programs (eg, computer-executable programs). When the one or more programs are executed by a device, the computer-readable medium implements the method of the present disclosure.
[0115] Those skilled in the art will appreciate that the above modules can be distributed in the device according to the description of the embodiment, or can be changed accordingly and only used in one or more devices different from the embodiment. The modules of the above embodiments can be combined into one module, or further divided into multiple sub-modules.
[0116] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the exemplary embodiments described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the embodiment of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several commands to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the embodiment of the present invention.
[0117] The exemplary embodiments of the present invention are specifically shown and described above. It should be understood that the present invention is not limited to the detailed structure, configuration or implementation method described herein; on the contrary, the present invention is intended to cover various modifications and equivalent configurations included in the spirit and scope of the appended claims.
Claims
1. A method for processing object storage data lifecycle, characterized in that: include: Step S101: Acquire the following historical parameter information marked with thread processing completion time tags to establish a training data set: file size, resource utilization, total task volume, system configuration, and number of concurrent threads; Step S102: constructing a thread prediction model using a multivariate linear regression algorithm, and using the training data set to train the thread prediction model; the training process of the thread prediction model also includes fine-tuning the thread prediction model using the number of concurrent threads in the historical parameter information as a learning parameter to obtain an optimal solution for the number of concurrent threads; The thread prediction model construction includes: in the model training process, completing a first learning stage and a second learning stage, wherein in the first learning stage, historical parameter information and historical thread consumption time are used to train the model parameters of the thread prediction model; after completing the first learning stage, the second learning stage is started, and in the second learning stage, the number of concurrent threads in the historical parameter information is used as a learning parameter to retrain the thread prediction model, and the number of concurrent threads of the optimal solution is obtained by using the model inverse learning; Step S103: input the current file size, resource utilization, total number of tasks, and system configuration parameters into the trained thread prediction model, and output the matching thread pool and number of concurrent threads; Step S104: based on the output thread pool and the number of concurrent threads, concurrently execute each thread, determine whether there is an index shard of the bucket in the current thread, and when it is determined that there is an index shard of the bucket, obtain the next index shard; before concurrently executing each thread, obtain the index shard information of the current bucket, and determine whether it is the first index shard in the current bucket according to the index shard information; determine whether there is the first index shard of the current bucket in the current thread, and when it is determined that there is the first index shard of the current bucket, obtain the next index shard for subsequent determination; Step S105: further determining whether the target object exists in the index shard of the bucket, and if the target object exists, obtaining a specific number of target objects; if the target object is not found, repeating step S103 until it is determined that the index shard of the bucket exists and the specific number of target objects is obtained; Step S106, submitting the acquired specific number of target objects to a queue to be processed, and repeatedly determining whether there are target objects in the queue to be processed that have not been determined to be expired, until all target objects have been determined to be expired.
2. The object storage data lifecycle processing method according to claim 1, characterized in that: Further including: In the first learning phase, the model is iteratively trained a first specified number of times; In the second learning phase, iteratively training the model a second specified number of times; as well as The first specified number of times is greater than the second specified number of times.
3. The object storage data lifecycle processing method according to claim 1, characterized in that: include: It is further determined whether there are target objects in the first index fragment of the current storage bucket, and when there are target objects, a specific number of target objects are obtained.
4. The object storage data lifecycle processing method according to claim 1 or 3, characterized in that: When the target object exists, obtaining a specific number of target objects includes: When there are target objects, a specific number of target objects are obtained, where the target objects represent audio files, document files, and picture files that are managed by lifecycle rules.
5. The object storage data lifecycle processing method according to claim 1, characterized in that: include: When the number of threads in the output thread pool is greater than or equal to the number of concurrent threads output, obtaining threads with the number of concurrent threads from the thread pool; When the number of threads in the output thread pool is less than the number of concurrent threads output, all threads are obtained from the thread pool and a corresponding number of threads are newly created.
6. An object storage data lifecycle processing system, characterized in that: The object storage data lifecycle processing method according to any one of claims 1 to 5 is adopted, and the object storage data lifecycle processing system comprises: The data acquisition module is used to obtain the following historical parameter information marked with thread processing completion time tags to establish a training data set: file size, resource utilization, total number of tasks, system configuration, and number of concurrent threads; A model building module, using a multivariate linear regression algorithm to build a thread prediction model, and using the training data set to train the thread prediction model; the training process of the thread prediction model also includes fine-tuning the thread prediction model using the number of concurrent threads in the historical parameter information as a learning parameter to obtain an optimal solution for the number of concurrent threads; the construction of the thread prediction model includes: in the model training process, completing a first learning stage and a second learning stage, wherein in the first learning stage, the model parameters of the thread prediction model are trained using the historical parameter information and the historical thread consumption time; after completing the first learning stage, the second learning stage is started, and in the second learning stage, the number of concurrent threads in the historical parameter information is used as a learning parameter to re-train the thread prediction model, and the number of concurrent threads of the optimal solution is obtained by using model inverse learning; The model determination module is used to input the current file size, resource utilization, total number of tasks, and system configuration parameters into the trained thread prediction model, and output the matching thread pool and number of concurrent threads; The first judgment processing module concurrently executes each thread based on the output thread pool and the number of concurrent threads, and judges whether there is an index shard of the bucket in the current thread. When it is determined that there is an index shard of the bucket, the next index shard is obtained; before concurrently executing each thread, the index shard information of the current bucket is obtained, and whether it is the first index shard in the current bucket according to the index shard information; judges whether there is the first index shard of the current bucket in the current thread, and when it is determined that there is the first index shard of the current bucket, the next index shard is obtained for subsequent judgment; A second judgment processing module is used to further judge whether there is a target object in the index shard of the storage bucket, and if there is a target object, obtain a specific number of target objects; if no target object is found, repeat step S103 until it is determined that there is an index shard of the storage bucket and the specific number of target objects are obtained; The repeated determination module submits a specific number of acquired target objects to a pending queue, and repeatedly determines whether there are target objects in the pending queue that have not been determined to be expired, until all target objects have been determined to be expired.
7. The object storage data lifecycle processing system according to claim 6, characterized in that: include: In the first learning phase, the model is iteratively trained a first specified number of times; In the second learning phase, iteratively training the model a second specified number of times; as well as The first specified number of times is greater than the second specified number of times.
