Data management method and device, equipment, medium and program product
By sorting data objects in the container and multi-threading, the problem of low query efficiency in the existing technology is solved, and the life cycle management efficiency and timeliness of data objects are improved.
Patent Information
- Application Number
- CN202310538752.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-12
- Publication Date
- 2024-11-12
AI Technical Summary
In the prior art, the efficiency of querying data objects in containers is low, resulting in poor timely management of data objects.
By obtaining the container's sequence list and sorting according to preset life cycle rules, target data objects that meet life cycle rules are filtered, query efficiency is improved, and life cycle management operations are performed through multi-threading, and management efficiency is improved.
It improves the query efficiency of data objects and the timeliness of life cycle management, reduces invalid scanning, and enhances the comprehensiveness and accuracy of the target data objects.
Smart Images

Figure CN118939645A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a data management method, apparatus, device, medium and program product. Background Art
[0002] With the rapid development of computer technology, the number of data objects included in containers is increasing, and it is necessary to timely perform life cycle management of data objects in containers, such as object deletion and object migration.
[0003] In the related art, when managing data objects in a container, it is usually necessary to query the data objects that need to be managed from the massive data in the container to perform corresponding management operations.
[0004] However, the efficiency of querying data objects in containers in related technologies is low, resulting in poor timeliness in the lifecycle management of data objects. Summary of the invention
[0005] Based on this, it is necessary to provide a data management method, device, equipment, medium and program product to address the above technical problems, so as to improve the query efficiency of objects in the container, thereby improving the life cycle management efficiency of data objects.
[0006] In a first aspect, the present application provides a data management method, the method comprising:
[0007] In response to the data management instruction of the container, obtaining a data object sequence list of the container; each data object in the data object sequence list is sorted based on the creation time;
[0008] According to the preset life cycle rules, the target data object satisfying the life cycle rules is obtained from the data object sequence table;
[0009] Manage the target data object according to the lifecycle management operations corresponding to the lifecycle rules.
[0010] In the technical solution provided by the embodiment of the present application, first respond to the data management instruction of the container, obtain the data object sequence list of the container, then obtain the target data object that meets the life cycle rule from the data object sequence list according to the preset life cycle rule, and finally manage the target data object according to the life cycle management operation corresponding to the life cycle rule. Wherein each data object in the data object sequence list is sorted based on the creation time. Since the embodiment of the present application obtains the target data object according to the life cycle rule on the basis of obtaining the data object sequence list, and the data object sequence represents the sorting based on the creation time. It is equivalent to the embodiment of the present application obtaining the target data object in sequence based on the sorting result of the creation time of the data object. Such an acquisition method can filter the data objects that do not meet the time management conditions corresponding to the life cycle rule, and improve the query efficiency of the target data object. Furthermore, according to the life cycle management operation corresponding to the life cycle rule, the target data object is managed to improve the timeliness of the life cycle management of the data object.
[0011] In one embodiment, the lifecycle rule includes a time threshold, and according to the preset lifecycle rule, obtaining a target data object satisfying the lifecycle rule from the data object sequence table includes:
[0012] For any lifecycle rule, determine the time range corresponding to the lifecycle rule based on the time threshold in the lifecycle rule;
[0013] Determine in the data object sequence table that the data object whose creation time is within the time range is a candidate data object;
[0014] Based on the candidate data objects, target data objects that meet the lifecycle rules are obtained.
[0015] In the technical solution provided by the embodiment of the present application, the time range corresponding to the lifecycle rule is determined according to the time threshold in the lifecycle rule, and the data objects in the data object sequence list whose creation time is within the time range are determined as candidate data objects, which is not affected by the scanning interruption. That is to say, when the candidate data object determination process is interrupted and the candidate data object determination process needs to be restarted, the embodiment of the present application obtains a new time range based on the restart time and the time threshold, and determines the candidate data objects again, and obtains the target data object. Obviously, the method of determining the target data object in the embodiment of the present application can obtain more comprehensive candidate data objects that meet the time threshold, thereby improving the comprehensiveness and accuracy of the target data object.
[0016] In one embodiment, the lifecycle rule further includes an object type identifier; based on the candidate data object, obtaining a target data object that satisfies the lifecycle rule includes:
[0017] According to the object type identifier in the life cycle rule, the type identifier of each data object in the candidate data object is verified;
[0018] The data objects with the same object type identifier among the candidate data objects are identified to determine the target data objects that meet the life cycle rules.
[0019] In the technical solution provided in the embodiment of the present application, the candidate data objects are further verified according to the object type identifier in the lifecycle rule, and the candidate data objects with the same object type identifier are determined as the target data objects, ensuring that the time threshold and object type identifier of the target data objects meet the lifecycle rules, thereby improving the validity of the target data objects and facilitating the execution of lifecycle management operations on the target data objects.
[0020] In one embodiment, managing the target data object according to the lifecycle management operation corresponding to the lifecycle rule includes:
[0021] storing the target data object in the management cache;
[0022] If the number of target data objects in the management cache is greater than a preset number threshold, a data object management task is generated, and the number of target data objects carried in the data object management task is the same as the preset number threshold;
[0023] Send a data object management task to the processing end, instructing the processing end to perform a lifecycle management operation corresponding to the lifecycle rule on the target data object in the data object management task.
[0024] In the technical solution provided by the embodiment of the present application, according to the lifecycle management operation, the target data object is stored in different management caches, and the object management tasks generated based on the target data object are sent to the processing end in batches to process the target data object. Since the processing end can perform lifecycle management operations on multiple target data objects in parallel through multi-threaded processing, the management operation time of the target data object is effectively shortened, and the data management efficiency is improved.
[0025] In one embodiment, the data management method further includes:
[0026] If a task completion instruction is received from the processing end, the current number of target data objects in the management cache is detected;
[0027] If the current number is greater than or equal to the preset number threshold, continue to send the data object management task to the processing end;
[0028] If the current quantity is less than the preset quantity threshold, the process returns to the step of obtaining the target data object that meets the lifecycle rule from the data object sequence table.
[0029] In the technical solution provided in the embodiment of the present application, by comparing the current number in the management cache with the preset number threshold, if the current number is greater than or equal to the preset number threshold, the data object management task continues to be sent to the processing end, otherwise it returns to the step of obtaining the target data object that meets the life cycle rules from the data object sequence table, and can perform life cycle management operations while obtaining the target data object. This parallel and continuous management operation method improves the management efficiency of data objects in the container.
[0030] In one embodiment, obtaining a data object sequence list of a container includes:
[0031] According to the creation time of each data object, the data objects in the container are sorted in ascending order to generate a data object sequence list of the container.
[0032] In the technical solution provided in the embodiment of the present application, the data objects in the container are sorted in ascending order based on the creation time of each data object. The data object sequence table generated in this way can intuitively and orderly display the creation time of each data object, while quickly acquiring the target data object that meets the life cycle rules, avoiding invalid scanning of the data objects in the container, thereby improving the query efficiency of the data objects.
[0033] In a second aspect, the present application further provides a data processing device, the device comprising:
[0034] A response module, configured to obtain a data object sequence list of the container in response to a data management instruction of the container; each data object in the data object sequence list is sorted based on a creation time;
[0035] An acquisition module is used to acquire a target data object that meets the life cycle rule from the data object sequence table according to the preset life cycle rule;
[0036] The management module is used to manage the target data object according to the lifecycle management operations corresponding to the lifecycle rules.
[0037] In a third aspect, the present application further provides a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method in any one of the embodiments of the first aspect when executing the computer program.
[0038] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method in any one of the embodiments of the first aspect.
[0039] In a fifth aspect, the present application further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the method in any one of the embodiments of the first aspect are implemented. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Figure 1 is an internal structure diagram of a computer device in one embodiment;
[0041] Figure 2 is a flow chart of a data management method in one embodiment;
[0042] Figure 3 A schematic diagram of a flow chart of a target data object determination step in one embodiment;
[0043] Figure 4 A schematic flow chart of a target data object determination step in another embodiment;
[0044] Figure 5 A flowchart of data management operation steps in one embodiment;
[0045] Figure 6 is a flow chart of a data management method in another embodiment;
[0046] Figure 7 is a flow chart of a data management method in another embodiment;
[0047] Figure 8 A schematic diagram of the structure of a data management method in one embodiment;
[0048] Fig. 9 FIG. 4 is a structural block diagram of a data management device in an embodiment. DETAILED DESCRIPTION
[0049] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0050] The data management method provided in the embodiment of the present application can be applied to a computer device. The computer device can be a server, and its internal structure diagram can be as follows: Figure 1As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store management data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data management method is implemented.
[0051] Those skilled in the art will understand that Figure 1 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0052] With the rapid development of computer technology, the number of data objects included in containers is increasing, and it is necessary to timely perform life cycle management on the data objects in the containers, such as data object deletion, data object migration, etc.
[0053] The lifecycle management of data objects in the container is applicable to the following scenarios:
[0054] (1) Log files are uploaded periodically and need to be saved within a preset time period. If the preset time period is exceeded, the log files will be deleted.
[0055] (2) Delete a large number of cache files in the container at one time. Manually deleting data objects is time-consuming and labor-intensive, and there is a limit on the number of files that can be deleted.
[0056] (3) Documents need to be frequently accessed within a preset time period, but are no longer accessed outside of the preset time period. In this case, these documents need to be converted to low-frequency access storage after the preset time period.
[0057] (4) Data for archiving purposes, such as media data, financial data, medical data, biological data, and data that must be retained to meet regulatory requirements.
[0058] In any of the above scenarios, the relevant technology performs lifecycle management on the data objects in the container, which is to query the data objects that need to be managed from the massive data in the container to perform corresponding management operations.
[0059] However, when the amount of object data in the container is large, the relevant technology needs to traverse all data objects in the container from beginning to end, which has the problem of low efficiency in querying data objects in the container, resulting in poor timeliness of life cycle management of data objects.
[0060] Based on this, the present application proposes a data management method, which can sort the data objects in the container, quickly query the data objects in the container, and timely manage the life cycle of the data objects in the container. The data management method proposed in the present application is described below through an embodiment.
[0061] In one embodiment, Figure 2 As shown, a data management method is provided, comprising the following steps:
[0062] S201, in response to a data management instruction of a container, obtaining a data object sequence list of the container; each data object in the data object sequence list is sorted based on creation time.
[0063] The data management instructions of the container are used to manage the data objects in the container, such as deleting data objects or migrating data objects. They can be triggered by the user or automatically triggered when the computer device where the container is located detects that the number of data objects in the container has reached a preset threshold.
[0064] There are tens of thousands of data objects stored in the container. Each data object has its own creation time when it is created in the container. The storage format of the creation time can be XXXX year XX month XX day, or XXXX year / XX month / XX day, etc. The embodiment of the present application does not limit the storage format of the creation time.
[0065] In response to the data management instruction of the container, the creation time of each data object in the container is obtained, and the data objects in the container are sorted according to the creation time to obtain a data object sequence list of the container.
[0066] Optionally, each data object generates a corresponding object time index table when it is created in the container, and the object time index table includes information such as the creation time, object name, storage size, and storage level of the data object. In response to the data management instruction of the container, the data objects in the container are sorted according to the creation time in the object time index table of each data object, and the data object sequence table of the container is obtained.
[0067] S202: According to a preset life cycle rule, a target data object satisfying the life cycle rule is obtained from a data object sequence table.
[0068] Lifecycle rules refer to the rules followed when deleting or converting the storage level of data objects in a container, including lifecycle management conditions of data objects, such as the creation time, object name, object type and other information of the data object, as well as lifecycle management operations of the data object.
[0069] Taking deleting a data object as an example, the format of the corresponding lifecycle rule is: data object type-preset time-deletion operation. Taking changing the storage level as an example, the format of the corresponding lifecycle rule is: data object type-preset time-current storage level-change storage level.
[0070] The lifecycle management conditions are obtained according to the preset lifecycle rules, and the data objects in the data object sequence table are screened according to the lifecycle management conditions corresponding to the lifecycle rules, and the data objects that meet the lifecycle management conditions are taken as the data objects that meet the lifecycle rules, that is, the target data objects.
[0071] It should be noted that for a container, there is no upper limit on the number of data objects included in the container. A container can include tens of thousands of data objects, but the number of lifecycle rules included in the container is limited. A container can include at most one thousand lifecycle rules. This means that a lifecycle rule can be applied to multiple target data objects.
[0072] S203: Manage the target data object according to the lifecycle management operation corresponding to the lifecycle rule.
[0073] Optionally, according to the lifecycle management operation corresponding to the lifecycle rule, a processing instruction is sent to the processing end to instruct to perform corresponding management on the target data object that meets the lifecycle rule.
[0074] Optionally, according to the lifecycle management operations corresponding to the lifecycle rules, the same number of threads as the number of target data objects are created, and corresponding management is performed on the target data objects that meet the lifecycle rules through the threads.
[0075] In an embodiment of the present application, first respond to the data management instruction of the container, obtain the data object sequence list of the container, then obtain the target data object that meets the life cycle rule from the data object sequence list according to the preset life cycle rule, and finally manage the target data object according to the life cycle management operation corresponding to the life cycle rule. Wherein each data object in the data object sequence list is sorted based on the creation time. Since the embodiment of the present application obtains the target data object according to the life cycle rule on the basis of obtaining the data object sequence list, and the data object sequence represents the sorting based on the creation time. It is equivalent to the embodiment of the present application obtaining the target data object in sequence based on the sorting result of the creation time of the data object. Such an acquisition method can filter the data objects that do not meet the time management conditions corresponding to the life cycle rule, and improve the query efficiency of the target data object. Furthermore, according to the life cycle management operation corresponding to the life cycle rule, the target data object is managed to improve the timeliness of the life cycle management of the data object.
[0076] When managing data objects in a container, the data objects in the container are scanned and managed based on the data object sequence table. The above embodiment describes the data object sequence table: each data object in the data object sequence table is sorted based on the creation time. Based on this, an embodiment is used below to describe a possible sorting method of the data object sequence table.
[0077] In one embodiment, obtaining a data object sequence list of a container includes:
[0078] According to the creation time of each data object, the data objects in the container are sorted in ascending order to generate a data object sequence list of the container.
[0079] The data object sequence list of a container refers to a sequence list including all data objects in the container.
[0080] The order of the data objects in the data object list is also the order of the creation time of the data objects from small to large. This means that the earlier the creation time of the data object, the higher the position of the data object in the data object sequence list.
[0081] In an embodiment of the present application, the data objects in the container are sorted in ascending order based on the creation time of each data object. The data object sequence table generated in this way can intuitively and orderly display the creation time of each data object, while quickly acquiring the target data object that meets the life cycle rules, avoiding invalid scanning of the data objects in the container, thereby improving the query efficiency of the data objects.
[0082] In the process of acquiring the target data object, the data objects in the data object sequence table are usually scanned according to multiple life cycle management conditions in the preset life cycle rules to improve the validity of the target data object. Based on this, the following is an example of an embodiment to illustrate the steps of acquiring the target data object when the life cycle rule is set with a time threshold management condition.
[0083] In one embodiment, Figure 3 As shown, the life cycle rule includes a time threshold. According to the preset life cycle rule, the target data object satisfying the life cycle rule is obtained from the data object sequence table, including:
[0084] S301: For any lifecycle rule, determine a time range corresponding to the lifecycle rule according to a time threshold in the lifecycle rule.
[0085] The time threshold refers to the length of time that a data object is allowed to be stored in the container in its current format. If the data object is stored in the container for longer than the time threshold, the data object is considered a candidate data object and is further verified according to other management conditions of the lifecycle rule.
[0086] Since the time for managing data objects according to lifecycle rules is uncertain, the time threshold set in the lifecycle rules is based on the management time. For any lifecycle rule, according to the time threshold and management time in the lifecycle rule, the time range before the management time and the time period from the management time greater than the time threshold is determined as the time range corresponding to the lifecycle rule.
[0087] For example, if the time threshold in the lifecycle rule is three days, and the management time for data objects according to the lifecycle rule is this Friday, then the time range corresponding to the lifecycle rule is the time before this Tuesday. This is because the data objects created before this Tuesday have been stored in the container in the current format for more than three days, which meets the time threshold in the lifecycle rule.
[0088] S302: Determine data objects in the data object sequence list whose creation time is within a time range as candidate data objects.
[0089] According to the time range corresponding to the life cycle rule, the creation time of each data object in the data object sequence table is compared, and the data object whose creation time is within the time range corresponding to the life cycle rule is determined as the candidate data object.
[0090] Still taking the case where the time threshold is three days and the management time is this Friday as an example, since the time range corresponding to the lifecycle rule is the time before this Tuesday, the data object created on Monday of this week and stored in the container in the current format for more than three days is determined to be a candidate data object, while the data object created on Thursday of this week and stored in the container in the current format for less than three days is not a candidate data object.
[0091] S303: Based on the candidate data objects, obtain a target data object that meets the lifecycle rule.
[0092] The candidate data object is a data object in the data object sequence table that meets the time threshold in the life cycle rule.
[0093] If the lifecycle management condition in the lifecycle rule only has one condition, which is the time threshold, then the candidate data object is the target data object that satisfies the lifecycle rule.
[0094] If the lifecycle management conditions in the lifecycle rules include other management conditions besides the time threshold, such as object type, object name, object version, etc., the candidate data objects can be further judged according to the other management conditions to obtain the target data objects that meet the lifecycle rules.
[0095] In the embodiment of the present application, the time range corresponding to the lifecycle rule is determined according to the time threshold in the lifecycle rule, and the data objects in the data object sequence table whose creation time is within the time range are determined as candidate data objects, which is not affected by the scanning interruption. That is to say, when the candidate data object determination process is interrupted and the candidate data object determination process needs to be restarted, the embodiment of the present application obtains a new time range based on the restart time and the time threshold, and determines the candidate data objects again, and obtains the target data object. Obviously, the method of determining the target data object in the embodiment of the present application can obtain a more comprehensive candidate data object that meets the time threshold, thereby improving the comprehensiveness and accuracy of the target data object.
[0096] The most basic and important lifecycle management condition of lifecycle rules is the time threshold. However, considering the diversity of data objects and the differences in management methods, lifecycle rules usually need to be combined with other lifecycle management rules other than time thresholds to screen data objects to ensure the validity of the target data objects. Based on this, the following is an example of how to obtain the target data object when the lifecycle rule also includes an object type identifier.
[0097] In one embodiment, Figure 4As shown, the lifecycle rule also includes an object type identifier; based on the candidate data object, obtaining a target data object that meets the lifecycle rule includes:
[0098] S401 , verifying the type identifier of each data object in the candidate data objects according to the object type identifier in the life cycle rule.
[0099] The object type identifier in the life cycle rule is verified with the type identifier of each data object in the candidate data object to determine whether the type identifier of each data object in the candidate data object is consistent with the object type identifier set in the life cycle rule.
[0100] S402: Determine a target data object that satisfies a life cycle rule from among the candidate data objects the data object with the same object type identifier.
[0101] Data objects in the candidate data objects that are different from the object type identifiers in the life cycle rules are regarded as non-target data objects and are removed from the candidate data objects; data objects in the candidate data objects that are the same as the object type identifiers in the life cycle rules are determined as target data objects. Obviously, the target data objects meet both the time threshold and the object type identifier in the life cycle rules.
[0102] In an embodiment of the present application, the candidate data objects are further verified based on the object type identifier in the lifecycle rule, and the candidate data objects with the same object type identifier are determined as the target data objects, ensuring that the time threshold and object type identifier of the target data objects meet the lifecycle rules, thereby improving the validity of the target data objects and facilitating the execution of lifecycle management operations on the target data objects.
[0103] In the process of managing the target data object, the target data object is usually managed in a targeted manner according to the life cycle rules satisfied by the target data object. Based on this, the following describes the steps of managing the target data object through an embodiment.
[0104] In one embodiment, Figure 5 As shown, the target data object is managed according to the lifecycle management operations corresponding to the lifecycle rules, including:
[0105] S501, storing the target data object in the management cache.
[0106] It should be noted that each lifecycle rule corresponds to a lifecycle management operation. Among them, the lifecycle management operation includes any one of data object deletion and data object storage level conversion. According to the lifecycle management operation corresponding to the lifecycle rule, the target data object is stored in the corresponding management cache. The data object deletion management operation corresponds to the deletion cache, and the data object storage level conversion operation corresponds to the conversion cache.
[0107] Exemplarily, if the lifecycle management operation is data object deletion, the target data object corresponding to the lifecycle rule is stored in the deletion cache.
[0108] Exemplarily, if the lifecycle management operation is a data object storage level conversion, the target data object corresponding to the lifecycle rule is stored in the conversion cache.
[0109] S502: If the number of target data objects in the management cache is greater than a preset number threshold, a data object management task is generated, and the number of target data objects carried in the data object management task is the same as the preset number threshold.
[0110] For the target data objects in the management cache, corresponding lifecycle management operations need to be performed. In addition, one lifecycle management operation is generated for each target data object. The preset number threshold refers to the number of lifecycle management operations that can be processed in parallel.
[0111] If the number of target data objects in the management cache is greater than a preset number threshold, target data objects identical to the preset number threshold are extracted from the management cache, and data object management tasks identical to the preset number threshold are generated to perform lifecycle management operations identical to the preset number threshold.
[0112] Taking the deletion cache including 4000 target data objects as an example, if the preset quantity threshold is 1000, 1000 target data objects are extracted from the 4000 target data objects, and 1000 data object management tasks are generated.
[0113] S503: Send the data object management task to the processing end, instructing the processing end to perform the lifecycle management operation corresponding to the lifecycle rule on the target data object in the data object management task.
[0114] The data object management tasks are uniformly sent to the processing end, and the processing end is instructed to establish multiple threads to execute life cycle management operations corresponding to the life cycle rules in parallel for the target data objects in the data object management tasks.
[0115] It should be noted that due to the influence of factors such as the space utilization rate of the data disk and metadata disk on the processing end, the load pressure of the processing end is limited and needs to be dynamically adjusted according to actual needs. In other words, the preset number threshold of the lifecycle management operations executed in parallel on the processing end needs to be dynamically adjusted according to the actual work efficiency of the processing end.
[0116] In the embodiment of the present application, according to the lifecycle management operation, the target data object is stored in different management caches, and the object management tasks generated based on the target data object are sent to the processing end in batches to process the target data object. Since the processing end can perform lifecycle management operations on multiple target data objects in parallel through multi-threaded processing, the management operation time of the target data object is effectively shortened, and the data management efficiency is improved.
[0117] Since the time it takes for the processing end to execute the object management task is much longer than the time it takes to store the target data object in the management cache, the number of target data objects in the management cache is usually greater than the number of parallel threads established by the processing end. In this case, it is necessary to send object management tasks from the management cache to the processing end in batches to perform lifecycle management operations on the target data object. Based on this, the following is an example of an embodiment to illustrate the corresponding data management method after the processing end executes a task completion instruction.
[0118] In one embodiment, Figure 6 As shown, the data management method also includes:
[0119] S601: If a task completion instruction sent by a processing end is received, the current number of target data objects in the management cache is detected.
[0120] If a task completion instruction is received from the processing end, the current number of target data objects in the management cache is detected, and further compared to see whether the current number is greater than or equal to a preset number threshold.
[0121] Taking the case where the target data object in the management cache is the original data object before the processing end executes the management operation as an example, the current number is explained as follows: if all threads on the processing end normally complete the object management task, the current number is the difference between the number of original target data objects in the management cache and the preset number threshold. If a thread on the processing end fails to execute the object management task, the current number is the difference between the number of original target data objects in the management cache and the preset number threshold minus the number of failed threads.
[0122] S602: If the current number is greater than or equal to the preset number threshold, continue to send the data object management task to the processing end.
[0123] If the current number is greater than or equal to the preset number threshold, the target data objects with the same number threshold as the preset number continue to be extracted from the management cache, a data object management task is generated, and the data object management task is sent to the processing end.
[0124] S603: If the current quantity is less than the preset quantity threshold, return to the step of obtaining the target data object that meets the lifecycle rule from the data object sequence table.
[0125] If the current number is less than the preset number threshold, it indicates that the number of target data objects in the management cache is too small and needs to be filled with new target data objects, then the process returns to the step of obtaining target data objects that meet the lifecycle rules from the data object sequence table.
[0126] In an embodiment of the present application, by comparing the current number in the management cache with the preset number threshold, if the current number is greater than or equal to the preset number threshold, the data object management task continues to be sent to the processing end, otherwise it returns to the step of obtaining the target data object that meets the life cycle rules from the data object sequence table, and can perform life cycle management operations while obtaining the target data object. This parallel and continuous management operation method improves the management efficiency of data objects in the container.
[0127] In one embodiment, a data management method parallel to the above embodiment is also provided, such as Figure 7 As shown, the data management method includes:
[0128] S701: If a task completion instruction sent by a processing end is received, the current number of target data objects in the management cache is detected.
[0129] If a task completion instruction is received from the processing end, the current number of target data objects in the management cache is detected, and further compared to see whether the current number is greater than or equal to a preset number threshold.
[0130] S702: If the current number is less than the cache threshold, return to execute the step of obtaining the target data object that meets the life cycle rule from the data object sequence table; the cache threshold is greater than the preset number threshold.
[0131] By detecting the current number and the cache threshold in the management cache, if the current number is less than the cache threshold, it indicates that the number of target data objects in the management cache is too small and new target data objects are needed to fill the cache. Then, the process returns to the step of obtaining the target data objects that meet the lifecycle rules from the data object sequence table.
[0132] In an embodiment of the present application, by comparing the current number and the cache threshold, the target data object in the management cache is filled in a timely manner. Since the cache threshold is greater than the preset number threshold, it can ensure that the processing end continuously performs lifecycle management operations while the management cache also continuously fills the target data objects to improve the efficiency of data object management.
[0133] In one embodiment, a data management method is provided, comprising the following steps:
[0134] (1) In response to the data management instruction of the container, the data objects in the container are sorted in ascending order according to the creation time of each data object, and a data object sequence list of the container is generated.
[0135] (2) For any lifecycle rule, determine the time range corresponding to the lifecycle rule based on the time threshold in the lifecycle rule.
[0136] (3) Determine the data objects in the data object sequence list whose creation time is within the time range as candidate data objects.
[0137] (4) Based on the object type identifier in the life cycle rule, the type identifier of each data object in the candidate data objects is verified.
[0138] (5) Data objects with the same object type identifier among the candidate data objects are identified as target data objects that meet the life cycle rules.
[0139] (6) According to the lifecycle management operation corresponding to the lifecycle rule, the target data object is stored in the management cache.
[0140] (7) If the number of target data objects in the management cache is greater than a preset number threshold, a data object management task is generated, and the number of target data objects carried in the data object management task is the same as the preset number threshold.
[0141] (8) Sending a data object management task to the processing end, instructing the processing end to perform a lifecycle management operation corresponding to the lifecycle rule on the target data object in the data object management task.
[0142] (9) If a task completion instruction is received from the processing end, the current number of target data objects in the management cache is detected.
[0143] (10) If the current number is greater than or equal to the preset number threshold, continue to send the data object management task to the processing end.
[0144] (11) If the current quantity is less than the preset quantity threshold, return to execute the step of obtaining the target data object that meets the life cycle rule from the data object sequence table.
[0145] In an embodiment of the present application, first respond to the data management instruction of the container, obtain the data object sequence list of the container, then obtain the target data object that meets the life cycle rule from the data object sequence list according to the preset life cycle rule, and finally manage the target data object according to the life cycle management operation corresponding to the life cycle rule. Wherein each data object in the data object sequence list is sorted based on the creation time. Since the embodiment of the present application obtains the target data object according to the life cycle rule on the basis of obtaining the data object sequence list, and the data object sequence represents the sorting based on the creation time. It is equivalent to the embodiment of the present application obtaining the target data object in sequence based on the sorting result of the creation time of the data object. Such an acquisition method can filter the data objects that do not meet the time management conditions corresponding to the life cycle rule, and improve the query efficiency of the target data object. Furthermore, according to the life cycle management operation corresponding to the life cycle rule, the target data object is managed to improve the timeliness of the life cycle management of the data object.
[0146] To further illustrate the data management method, in one embodiment, Figure 8 As shown, taking the management cache including 4000 original target data objects, the cache threshold of the management cache being 2000, and the preset number threshold being 1000 as an example, the process of the processing end interacting with the management cache to perform the object management task is described, including the following steps:
[0147] (1) Waiting for data object management signal. The management cache waits for the data object management signal.
[0148] (2) Submitting target data object management tasks. The management cache submits 1,000 target data object management tasks to the processing end according to the cache threshold.
[0149] (3) Execute the target data object management task. The processing end executes the life cycle management operation of the target data object.
[0150] (4) The lifecycle management operation fails, and the retry time is less than the preset time. The processing end fails to perform a lifecycle management operation, and the retry time after the failure is less than 5 minutes.
[0151] (5) Sending an instruction to reduce the number of target data object management tasks to be submitted. The processing end sends an instruction to reduce the number of target data object management tasks to be submitted to the management cache.
[0152] (6) Waiting for the data object management signal. The management cache waits for the data object management signal.
[0153] (7) Submitting the target data object management task. The management cache reduces the number of target data object management tasks and submits the target data object management task to the management cache.
[0154] (8) Manage the cache to detect the current number. If the current number is less than 2000, re-acquire the target data object.
[0155] In an embodiment of the present application, by managing the cooperation between the cache and the processing end, the rate at which the processing end performs lifecycle management operations on the target data object is dynamically adjusted, so that the processing end can perform multi-threaded lifecycle management operations and manage the cache to obtain parallel operations of the target data object, thereby improving data management efficiency.
[0156] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.
[0157] Based on the same inventive concept, the embodiment of the present application also provides a data management device for implementing the data management method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more data management device embodiments provided below can refer to the limitations on the data management method above, and will not be repeated here.
[0158] In one embodiment, Fig. 9 As shown, a data management device 900 is provided, including: a response module 901, an acquisition module 902, and a management module 903, wherein:
[0159] A response module 901 is used to obtain a data object sequence list of the container in response to a data management instruction of the container; each data object in the data object sequence list is sorted based on the creation time;
[0160] The acquisition module 902 is used to acquire the target data object satisfying the life cycle rule from the data object sequence table according to the preset life cycle rule;
[0161] The management module 903 is used to manage the target data object according to the lifecycle management operation corresponding to the lifecycle rule.
[0162] In one embodiment, the acquisition module 902 includes: a range determination unit, a candidate determination unit and an object determination unit, wherein:
[0163] A range determination unit, for determining, for any lifecycle rule, a time range corresponding to the lifecycle rule according to a time threshold in the lifecycle rule;
[0164] A candidate determination unit, used to determine data objects whose creation time in the data object sequence table is within a time range as candidate data objects;
[0165] The object determination unit is used to obtain a target data object that meets the life cycle rules based on the candidate data objects.
[0166] In one embodiment, the object determination unit includes: an identification verification subunit and an identification determination subunit, wherein:
[0167] The identification verification subunit is used to verify the type identification of each data object in the candidate data objects according to the object type identification in the life cycle rule;
[0168] The identification determination subunit is used to identify the target data objects that meet the life cycle rules from the candidate data objects that have the same object type identification.
[0169] In one embodiment, the management module 903 includes an object storage unit, a task generation unit and an operation instruction unit, wherein:
[0170] An object storage unit, used for storing the target data object in the management cache;
[0171] A task generating unit, configured to generate a data object management task if the number of target data objects in the management cache is greater than a preset number threshold, wherein the number of target data objects carried in the data object management task is the same as the preset number threshold;
[0172] The operation instruction unit is used to send a data object management task to the processing end, and instruct the processing end to perform a life cycle management operation corresponding to the life cycle rule on the target data object in the data object management task.
[0173] In one embodiment, the data management device 900 further includes a sorting module, which is used to sort the data objects in the container in ascending order according to the creation time of each data object, and generate a data object sequence list of the container.
[0174] Each module in the above data management device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.
[0175] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:
[0176] In response to the data management instruction of the container, obtaining a data object sequence list of the container; each data object in the data object sequence list is sorted based on the creation time;
[0177] According to the preset life cycle rules, the target data object satisfying the life cycle rules is obtained from the data object sequence table;
[0178] Manage the target data object according to the lifecycle management operations corresponding to the lifecycle rules.
[0179] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0180] For any lifecycle rule, determine the time range corresponding to the lifecycle rule based on the time threshold in the lifecycle rule;
[0181] Determine in the data object sequence table that the data object whose creation time is within the time range is a candidate data object;
[0182] Based on the candidate data objects, target data objects that meet the lifecycle rules are obtained.
[0183] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0184] According to the object type identifier in the life cycle rule, the type identifier of each data object in the candidate data object is verified;
[0185] The data objects with the same object type identifier among the candidate data objects are identified to determine the target data objects that meet the life cycle rules.
[0186] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0187] storing the target data object in the management cache;
[0188] If the number of target data objects in the management cache is greater than a preset number threshold, a data object management task is generated, and the number of target data objects carried in the data object management task is the same as the preset number threshold;
[0189] Send a data object management task to the processing end, instructing the processing end to perform a lifecycle management operation corresponding to the lifecycle rule on the target data object in the data object management task.
[0190] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0191] If a task completion instruction is received from the processing end, the current number of target data objects in the management cache is detected;
[0192] If the current number is greater than or equal to the preset number threshold, continue to send the data object management task to the processing end;
[0193] If the current quantity is less than the preset quantity threshold, the process returns to the step of obtaining the target data object that meets the lifecycle rule from the data object sequence table.
[0194] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0195] According to the creation time of each data object, the data objects in the container are sorted in ascending order to generate a data object sequence list of the container.
[0196] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:
[0197] In response to the data management instruction of the container, obtaining a data object sequence list of the container; each data object in the data object sequence list is sorted based on the creation time;
[0198] According to the preset life cycle rules, the target data object satisfying the life cycle rules is obtained from the data object sequence table;
[0199] Manage the target data object according to the lifecycle management operations corresponding to the lifecycle rules.
[0200] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0201] For any lifecycle rule, determine the time range corresponding to the lifecycle rule based on the time threshold in the lifecycle rule;
[0202] Determine in the data object sequence table that the data object whose creation time is within the time range is a candidate data object;
[0203] Based on the candidate data objects, target data objects that meet the lifecycle rules are obtained.
[0204] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0205] According to the object type identifier in the life cycle rule, the type identifier of each data object in the candidate data object is verified;
[0206] The data objects with the same object type identifier among the candidate data objects are identified to determine the target data objects that meet the life cycle rules.
[0207] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0208] storing the target data object in the management cache;
[0209] If the number of target data objects in the management cache is greater than a preset number threshold, a data object management task is generated, and the number of target data objects carried in the data object management task is the same as the preset number threshold;
[0210] Send a data object management task to the processing end, instructing the processing end to perform a lifecycle management operation corresponding to the lifecycle rule on the target data object in the data object management task.
[0211] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0212] If a task completion instruction is received from the processing end, the current number of target data objects in the management cache is detected;
[0213] If the current number is greater than or equal to the preset number threshold, continue to send the data object management task to the processing end;
[0214] If the current quantity is less than the preset quantity threshold, the process returns to the step of obtaining the target data object that meets the lifecycle rule from the data object sequence table.
[0215] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0216] According to the creation time of each data object, the data objects in the container are sorted in ascending order to generate a data object sequence list of the container.
[0217] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:
[0218] In response to the data management instruction of the container, obtaining a data object sequence list of the container; each data object in the data object sequence list is sorted based on the creation time;
[0219] According to the preset life cycle rules, the target data object satisfying the life cycle rules is obtained from the data object sequence table;
[0220] Manage the target data object according to the lifecycle management operations corresponding to the lifecycle rules.
[0221] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0222] For any lifecycle rule, determine the time range corresponding to the lifecycle rule based on the time threshold in the lifecycle rule;
[0223] Determine in the data object sequence table that the data object whose creation time is within the time range is a candidate data object;
[0224] Based on the candidate data objects, target data objects that meet the lifecycle rules are obtained.
[0225] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0226] According to the object type identifier in the life cycle rule, the type identifier of each data object in the candidate data objects is verified;
[0227] The data objects with the same object type identifier among the candidate data objects are identified to determine the target data objects that meet the life cycle rules.
[0228] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0229] storing the target data object in the management cache;
[0230] If the number of target data objects in the management cache is greater than a preset number threshold, a data object management task is generated, and the number of target data objects carried in the data object management task is the same as the preset number threshold;
[0231] Send a data object management task to the processing end, instructing the processing end to perform a lifecycle management operation corresponding to the lifecycle rule on the target data object in the data object management task.
[0232] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0233] If a task completion instruction is received from the processing end, the current number of target data objects in the management cache is detected;
[0234] If the current number is greater than or equal to the preset number threshold, continue to send the data object management task to the processing end;
[0235] If the current quantity is less than the preset quantity threshold, the process returns to the step of obtaining the target data object that meets the lifecycle rule from the data object sequence table.
[0236] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0237] According to the creation time of each data object, the data objects in the container are sorted in ascending order to generate a data object sequence list of the container.
[0238] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.
[0239] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0240] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. A data management method, characterized in that: The method comprises: In response to a data management instruction of a container, obtaining a data object sequence list of the container; each data object in the data object sequence list is sorted based on creation time; According to a preset life cycle rule, obtaining a target data object satisfying the life cycle rule from the data object sequence table; The target data object is managed according to the lifecycle management operation corresponding to the lifecycle rule.
2. The method according to claim 1, characterized in that The life cycle rule includes a time threshold, and acquiring a target data object satisfying the life cycle rule from the data object sequence table according to the preset life cycle rule includes: For any lifecycle rule, determine the time range corresponding to the lifecycle rule according to the time threshold in the lifecycle rule; Determine in the data object sequence table that the data object whose creation time is within the time range is a candidate data object; Based on the candidate data objects, a target data object satisfying the lifecycle rule is acquired.
3. The method according to claim 2, characterized in that The lifecycle rule also includes an object type identifier; and obtaining a target data object that satisfies the lifecycle rule based on the candidate data object includes: Verifying the type identifier of each data object in the candidate data objects according to the object type identifier in the life cycle rule; The data objects with the same object type identifier among the candidate data objects are determined as target data objects that meet the lifecycle rule.
4. The method according to any one of claims 1 to 3, characterized in that: The managing the target data object according to the lifecycle management operation corresponding to the lifecycle rule includes: Storing the target data object in a management cache; If the number of target data objects in the management cache is greater than a preset number threshold, a data object management task is generated, and the number of target data objects carried in the data object management task is the same as the preset number threshold; The data object management task is sent to a processing end, instructing the processing end to perform a lifecycle management operation corresponding to the lifecycle rule on a target data object in the data object management task.
5. The method according to claim 4, characterized in that The method further comprises: If a task completion instruction sent by the processing end is received, detecting the current number of target data objects in the management cache; If the current number is greater than or equal to the preset number threshold, continue to send the data object management task to the processing end; If the current quantity is less than the preset quantity threshold, the process returns to the step of acquiring the target data object satisfying the lifecycle rule from the data object sequence table.
6. The method according to claim 1, characterized in that The step of obtaining a data object sequence list of the container includes: According to the creation time of each data object, the data objects in the container are sorted in ascending order to generate a data object sequence list of the container.
7. A data processing device, characterized in that: The device comprises: A response module, configured to obtain a data object sequence list of the container in response to a data management instruction of the container; each data object in the data object sequence list is sorted based on a creation time; An acquisition module, used to acquire a target data object satisfying a life cycle rule from the data object sequence table according to a preset life cycle rule; A management module is used to manage the target data object according to the lifecycle management operation corresponding to the lifecycle rule.
8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.
Citation Information
Patent Citations
File management method, expired file deletion method and devices
CN108932236A
Cloud archived file filtering method, device thereof and equipment and readable storage medium
CN112221130A
Object storage life cycle management method, management device, equipment and medium
CN114489511A
File cleaning method and device for distributed file system, equipment and medium
CN115203130A
