Data management method and device

By presetting rules in the object storage unit and generating tasks when the object expires, the problems of high task storage costs and heavy execution burden in traditional storage architectures are solved, and more efficient data management and storage are achieved.

CN113392127BActive Publication Date: 2024-11-26TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110075597.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-01-20
Publication Date
2024-11-26
Estimated Expiration
2041-01-20

AI Technical Summary

Technical Problem

When traditional SAN and NAS storage architectures process unstructured data, task storage costs are high and the execution burden after the rules are modified is heavy, resulting in inefficiency in the system.

Method used

By pre-setting data storage rules and data processing rules in the object storage unit, it is stored only when the object is uploaded, and tasks are generated according to the rules when the object expires, avoiding the establishment of tasks during uploading, ensuring that the rules take effect immediately after modification.

Benefits of technology

Save the storage space of tasks, reduce the system execution burden, improve the response efficiency after the rule modification, and avoid the execution of repeated tasks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113392127B_ABST
    Figure CN113392127B_ABST
Patent Text Reader

Abstract

The embodiments of the present application relate to the field of computer technology, and in particular to a data management method and device for solving the problem of high task storage cost and heavy execution burden after task modification. The method is: according to the data storage rules corresponding to the target object storage unit and the storage time point of the object stored in the target object storage unit, at least one target object is selected from the target object storage unit, wherein the data storage rules at least include the maximum storage duration of the object; the data storage rules are used to compare with the attribute information of the object when the object is stored to determine the object storage unit of the object; according to the data processing rules corresponding to the target object storage unit and the to-be-executed information corresponding to the at least one target object, at least one task is generated; according to the at least one task, data processing is performed on the at least one target object.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular, to a data management method and device. Background Art

[0002] At present, people need more and more data, and most of them are unstructured data. In the future, unstructured data will show exponential growth. With the explosive growth of unstructured data, traditional SAN (storage area network) and NAS (network attached storage) storage architectures cannot cope with the problems caused by data explosion. Based on its flat and scalable structural characteristics, object storage has become the best solution for unstructured data storage. By replacing penetrating SAN and NAS storage with object storage, the efficiency of unstructured data access and storage has been greatly improved. Object storage has all the advantages of distributed storage, flexible scalability, and metadata management; through its powerful version management function, object storage effectively avoids logical errors caused by human operations.

[0003] In the related art, when an object is uploaded, a task is generally formed and stored according to the execution rules and execution time of the object. When the object expires, the system polls the expired tasks and executes them one by one. Since the expiration time of the object is fixed in the index when it is uploaded and stored, not only is the storage cost high, but if the execution rules change, there will be a large number of duplicate tasks waiting to be executed, which increases the execution burden of the system. Summary of the invention

[0004] The embodiments of the present application provide a data management method and device to solve the problems of high task storage cost and heavy execution burden after task modification.

[0005] The present application provides a data management method, comprising:

[0006] Selecting at least one target object from the target object storage unit according to a data storage rule corresponding to the target object storage unit and a storage time point when the object is stored in the target object storage unit, wherein the data storage rule at least includes a maximum storage duration of the object; the data storage rule is used to compare with the attribute information of the object when the object is stored to determine the object storage unit of the object;

[0007] At least one task is generated according to the data processing rule corresponding to the target object storage unit and the to-be-executed information corresponding to the at least one target object; the target object is stored in the target object storage unit after being uploaded, and the at least one task is generated when the storage time length in the target object storage unit is greater than the maximum storage time length;

[0008] According to the at least one task, data processing is performed on the at least one target object.

[0009] On the other hand, the present application provides a data management device, comprising:

[0010] A screening unit, configured to select at least one target object from the target object storage unit according to a data storage rule corresponding to the target object storage unit and a storage time point when the object is stored in the target object storage unit, wherein the data storage rule at least includes a maximum storage time of the object; the data storage rule is used to compare with the attribute information of the object when the object is stored to determine the object storage unit of the object;

[0011] A creation unit, configured to generate at least one task according to a data processing rule corresponding to the target object storage unit and to-be-executed information corresponding to the at least one target object; the target object is stored in the target object storage unit after being uploaded, and the at least one task is generated when the storage time length in the target object storage unit is greater than the maximum storage time length;

[0012] An execution unit is used to perform data processing on the at least one target object according to the at least one task.

[0013] Optionally, a matching unit is further included, for:

[0014] Receiving an uploaded object and determining attribute information of the object;

[0015] Matching the attribute information of the object with the data storage rule of the target object storage unit;

[0016] If it is determined that the attribute information of the object successfully matches the data storage rule of the target object storage unit, the object is saved in the target object storage unit.

[0017] Optionally, a matching unit is used to:

[0018] For each object storage unit, a corresponding data storage rule is set respectively, and each obtained data storage rule is converted into a corresponding unit feature vector respectively;

[0019] Converting the attribute information of the object into a corresponding object feature vector;

[0020] Matching the object feature vector with a unit feature vector corresponding to the target object storage unit;

[0021] It is determined that the object feature vector is the same as a unit feature vector corresponding to the target object storage unit.

[0022] Optionally, a storage unit is further included, for:

[0023] According to the attribute information of the at least one target object, determining a task storage partition corresponding to the corresponding at least one task in any preset storage unit, wherein the task storage unit includes a plurality of task storage partitions;

[0024] The at least one task is stored in a corresponding task storage partition.

[0025] Optionally, the task storage partition is provided with a status mark;

[0026] Execution units are also used to:

[0027] Retrieve the at least one task from at least one task storage partition marked as unfinished;

[0028] After determining that all tasks in the at least one task storage partition have been executed, the at least one task storage partition mark is changed to completed.

[0029] Optionally, the execution unit is also used to:

[0030] Based on at least one created execution process, and according to the generation time of each task, obtaining at least one task in a free state from a task storage partition marked as unfinished; wherein one execution process corresponds to one task;

[0031] Through the at least one execution process, the state of the corresponding at least one task is changed from a free state to a locked state.

[0032] Optionally, the execution unit is also used to:

[0033] adding lease time information to the state of the corresponding at least one task by the at least one execution process;

[0034] After determining that the lease time information corresponding to any one of the tasks has expired, the any one of the tasks is changed from a locked state to a free state;

[0035] The any one of the tasks is acquired through an execution process other than the at least one execution process, and the state of the any one of the tasks is changed to a locked state.

[0036] Optionally, the execution unit is also used to:

[0037] In the at least one task storage partition, determining the task concurrency of the object storage unit associated with each task;

[0038] The at least one task is selected from tasks associated with the target object storage unit whose task concurrency is lower than a set threshold.

[0039] On the other hand, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned data management method is implemented.

[0040] On the other hand, an embodiment of the present application further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the computer program is executed by the processor, the processor implements the above-mentioned data management method.

[0041] The object storage unit in the embodiment of the present application is pre-set with a data storage rule. When a user uploads an object, the object's attribute information is compared with the data storage rule to determine the object storage unit of the object, and the object is stored in the object storage unit, wherein the data storage rule at least includes the maximum storage duration of the object. The target object storage unit is any object storage unit. According to the data storage rule corresponding to the target object storage unit and the storage time point of the object stored in the target object storage unit, when at least one target object storage expires, at least one target object is selected from the target object storage unit. According to the data processing rule corresponding to the target object storage unit and the to-be-executed information corresponding to at least one target object, at least one task is generated, and data processing is performed on at least one target object according to the at least one task. The present application does not establish a task when the object is uploaded. When the target object storage expires and needs to be executed, a task is generated according to the to-be-executed information of the target object and the data processing rule of the target object storage unit, so that the storage space of the task can be saved, and the rule filtering and task generation are performed only when the task is executed, so that the rule modification can be guaranteed to take effect immediately, and there will be no problem of re-entry. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A process diagram of a data management method in the related art;

[0043] Figure 2 A process diagram of another data management method in the related art;

[0044] Figure 3 A process diagram of a data management method in an embodiment of the present application;

[0045] Figure 4 This is a schematic diagram of an application architecture of a data management method in an embodiment of the present application;

[0046] Figure 5 A flowchart of a data management method provided in an embodiment of the present application;

[0047] Figure 6A schematic diagram of task storage partitions provided in an embodiment of the present application;

[0048] Figure 7 A schematic diagram of multiple task processes obtaining tasks from an unfinished task storage partition provided in an embodiment of the present application;

[0049] Figure 8 A schematic diagram of a task provided in an embodiment of the present application where the remaining task processes continue to execute the task after the task execution fails;

[0050] Fig. 9 A curve chart of the total number of tasks for the day and the task concurrency threshold of the object storage unit provided in the embodiment of the present application;

[0051] Fig.10 A schematic diagram of the overall structure of the data management system provided in the embodiment of the present application;

[0052] Fig.11 A schematic diagram of a specific task execution process provided for a specific embodiment of the present application;

[0053] Fig.12 A schematic diagram of the structure of a data management device provided in an embodiment of the present application;

[0054] Fig.13 Schematic diagram of the physical architecture of a computer device in an embodiment of the present application. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0056] To facilitate understanding of the embodiments of the present application, several concepts are briefly introduced below:

[0057] Lifecycle system: Automatically manage data based on time, including expiration deletion and expiration cooling of data.

[0058] DB (DoggaByte, data storage unit): data storage unit in computer, 1DB = 1024 8 In the embodiment of the present application, DB is used to store the task data of the object.

[0059] CSV (Comma-Separated Values): Files that store tabular data (numbers and text) in plain text. Plain text means that the file is a sequence of characters and does not contain data that must be interpreted like binary numbers. CSV files consist of an arbitrary number of records separated by some kind of line break character; each record consists of fields, and the separator between fields is another character or string, most commonly a comma or a tab. Usually, all records have exactly the same sequence of fields. Usually they are plain text files.

[0060] Bucket is a figurative name for a storage space in MOS (Management Operating System), and is a container for storing objects. Object storage is a very flat storage method. Objects stored in a bucket are all at the same logical level, unlike a file system that has a multi-level file structure. In MOS, bucket names are globally unique. When each bucket is created, a default bucket ACL (Access Control List) is generated. Each item in the bucket ACL list contains what permissions are granted to the authorized user, such as read permission (READ), write permission (WRITE), full control permission (FULL_CONTROL), etc. Only when the user has the corresponding permissions on the bucket can he operate on the bucket, such as creating, deleting, displaying, and setting bucket ACLs.

[0061] Cloud computing: is a computing model that distributes computing tasks on a resource pool consisting of a large number of computers, so that various application systems can obtain computing power, storage space and information services as needed. The network that provides resources is called "cloud". The resources in the "cloud" are infinitely expandable in the eyes of users, and can be obtained at any time, used on demand, expanded at any time, and paid for by use.

[0062] As a provider of basic cloud computing capabilities, a cloud computing resource pool (referred to as a cloud platform, generally referred to as an Infrastructure as a Service (IaaS) platform) will be established, and various types of virtual resources will be deployed in the resource pool for external customers to choose to use. The cloud computing resource pool mainly includes: computing devices (virtualized machines, including operating systems), storage devices, and network devices.

[0063] According to the logical function division, the Platform as a Service (PaaS) layer can be deployed on the IaaS layer, and the Software as a Service (SaaS) layer can be deployed on the PaaS layer, or SaaS can be directly deployed on IaaS. PaaS is a platform for software operation, such as databases, web containers, etc. SaaS is a variety of business software, such as web portals, SMS mass senders, etc. Generally speaking, SaaS and PaaS are upper layers relative to IaaS.

[0064] With the development of society, human social activities generate a large amount of data, and these data can only fully realize their value after analysis and processing. At present, a large part of the data is unstructured data. With the explosive growth of unstructured data, object storage has become the best solution for unstructured data storage. Object storage, also known as object-based storage, is a general term used to describe the method of solving and processing discrete units, which are called objects. Like files, objects contain data, but unlike files, objects do not have a hierarchical structure in a layer structure. Each object is at the same level in a flat address space called a storage pool, and one object does not belong to the next level of another object.

[0065] In the related art, when an object is uploaded, the expiration time is written into the object index, and the TTL (Time To Live) function of the DB or the bypass system is used to scan and select the object for execution. The specific structure is as follows: Figure 1 As shown in the figure, when an object is uploaded, its expiration time is fixed in the index. If the user rules are changed, the TTL time of the existing data will become invalid and difficult to modify.

[0066] Another related technology is Figure 2 As shown in the figure, when an object is uploaded, the solution generates a task in a time sequence based on the execution time of the object and the operations to be performed, and writes it to an independent DB. The Worker (execution process) of the lifecycle system is used to poll the expired tasks and execute them one by one. The task volume of this solution is very large. At the same time, for the bucket of the lifecycle, the index storage cost consumed by the object increases; when the rules change, there will be a large number of repeated tasks waiting to be executed, which greatly increases the burden on the Worker.

[0067] In view of this, in order to solve the above problems, in the embodiments of the present application, Figure 3As shown, when a user uploads an object, only the object is stored without establishing a task. Specifically, the object is stored in a bucket according to the data storage rules, wherein each bucket is pre-set with data storage rules and data processing rules, and the object can select the corresponding bucket for storage through rule matching. The data storage rules include at least the maximum storage duration of the object. Then, according to the data storage rules corresponding to the bucket and the storage time of the object in the bucket, when the storage of the target object expires, the target object is selected from the bucket. According to the to-be-executed information corresponding to the target object and the data processing rules corresponding to the bucket, a task is established for the target object, and data processing is performed. In the embodiment of the present application, a task is not established when an object is uploaded, which can save storage space for the task, and rule filtering is performed to generate tasks during execution, which can ensure that the rule modification takes effect immediately, and there will be no problem of re-entering the warehouse.

[0068] The preferred implementation modes of the present application are further described in detail below in conjunction with the accompanying drawings.

[0069] See also Figure 4 , which is a schematic diagram of the application architecture of the data management method in an embodiment of the present application, including a server 100 and a terminal device 200.

[0070] The terminal device 200 can be a mobile or fixed electronic device, such as a mobile phone, a tablet computer, a laptop computer, a desktop computer, various wearable devices, a smart TV, a vehicle-mounted device, or other electronic devices capable of implementing the above functions. The terminal device 200 can upload objects to the server 100 and receive feedback from the server 100.

[0071] The terminal device 200 and the server 100 can be connected via the Internet to achieve mutual communication. Optionally, the above-mentioned Internet uses standard communication technology and / or protocol. The Internet is usually the Internet, but it can also be any network, including but not limited to any combination of a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a dedicated network or a virtual private network. In some embodiments, the data exchanged through the network is represented by technologies and / or formats including Hyper Text Mark-up Language (HTML), Extensible Markup Language (XML), etc. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec) can also be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can also be used to replace or supplement the above-mentioned data communication technologies.

[0072] The server 100 can provide various network services for the terminal device 200, and the server 100 can use cloud computing technology for information processing. Among them, the server 100 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, and this application is not limited here.

[0073] Specifically, the server 100 may include a processor 110 (Center Processing Unit, CPU), a memory 120, an input device 130 and an output device 140, etc. The input device 130 may include a keyboard, a mouse, a touch screen, etc., and the output device 140 may include a display device, such as a liquid crystal display (Liquid Crystal Display, LCD), a cathode ray tube (Cathode Ray Tube, CRT), etc.

[0074] The memory 120 may include a read-only memory (ROM) and a random access memory (RAM), and provides the processor 110 with program instructions and data stored in the memory 120. In the embodiment of the present invention, the memory 120 may be used to store the program of the data management method in the embodiment of the present invention.

[0075] The processor 110 calls the program instructions stored in the memory 120, and the processor 110 is used to execute the steps of any one of the data management methods in the embodiments of the present invention according to the obtained program instructions.

[0076] In addition, the application architecture diagram in the embodiment of the present invention is intended to more clearly illustrate the technical solution in the embodiment of the present invention, and does not constitute a limitation on the technical solution provided in the embodiment of the present invention. Of course, it is not limited to gastrointestinal disease diagnosis business applications. For other application architectures and business applications, the technical solution provided in the embodiment of the present invention is also applicable to similar problems.

[0077] Various embodiments of the present invention are applied to Figure 4 The application architecture diagram shown is used as an example for schematic description.

[0078] Figure 5 FIG. 1 is a flow chart showing a data management method provided by an embodiment of the present application. Figure 5 As shown, the method comprises the following steps:

[0079] Step 501: Select at least one target object from the target object storage unit according to the data storage rule corresponding to the target object storage unit and the storage time point when the object is stored in the target object storage unit.

[0080] The data storage rule includes at least a maximum storage duration of the object; the data storage rule is used to compare with the attribute information of the object when the object is stored to determine the object storage unit of the object.

[0081] Specifically, a plurality of object storage units, i.e., buckets, are provided in the server. Each object storage unit is pre-set with corresponding data storage rules and data processing rules, such as user ID, object prefix, operation mode, storage days, object tag, etc. The data storage rules and data processing rules of different object storage units are different, and here all the rules in the data storage rules and data processing rules may be different, or some of the rules in the data storage rules and data processing rules may be the same while the other rules may be different.

[0082] After receiving the object uploaded by the user, the server stores the object in the corresponding object storage unit according to the data storage rules and data execution rules. The data storage rules at least include the maximum storage duration of the object, which is used to identify the storage time of the object, that is, after the object storage expires, the object is filtered out and a task is created.

[0083] Generally speaking, for ease of operation, target objects can be obtained from the target object storage unit at a set frequency, for example, if object screening is performed at a specified time point every day, then the number of target objects obtained from the target object storage unit is multiple. Furthermore, since there are multiple object storage units stored in the server, the screening range is also all object storage units.

[0084] For example, there are 10 buckets set up in the server, and the maximum storage duration of objects is set to 5 days, 7 days, 10 days, etc. After the object is uploaded, it is stored in the corresponding bucket according to the maximum storage duration. At 0:00 every day, for each bucket, according to the maximum storage duration of the bucket and the storage time point of each object stored therein, all objects that expire on that day are filtered out from the bucket as target objects. In a possible scenario, 350 target objects are filtered out from the first bucket, 120 target objects are filtered out from the second bucket... and 200 target objects are filtered out from the tenth bucket.

[0085] Step 502: Generate at least one task according to the data processing rule corresponding to the target object storage unit and the to-be-executed information corresponding to at least one target object. The target object is stored in the target object storage unit after being uploaded, and at least one task is generated when the storage time in the target object storage unit is greater than the maximum storage time.

[0086] Specifically, the object selects the corresponding target object storage unit according to the data storage rules and data processing rules of the object storage unit, that is, the object stored in the target object storage unit not only matches the data storage rules of the target object storage unit, but also matches the data processing rules of the target object storage unit. In this way, when the storage of the target object expires, the target object is screened out from the target object storage unit, and the task of the target object is generated according to the data processing rules corresponding to the target object storage unit and the to-be-executed information corresponding to the target object.

[0087] In the specific implementation process, objects are generally screened for multiple object storage units, and the number of target objects screened from the same object storage unit is also multiple, that is, multiple target objects can be screened from the same batch. For each target object, a corresponding task is generated according to the to-be-executed information of the target object and the data processing rules of the corresponding target object storage unit.

[0088] Step 503: Perform data processing on at least one target object according to at least one task.

[0089] In the specific implementation process, one target object corresponds to one task, and the tasks of different target objects may be the same or different, for example, deleting the target object, or performing read and write operations on the target object, etc. In addition, in the embodiment of the present application, the processing of tasks can be serial processing, that is, the same process processes all tasks in sequence; it can also be parallel processing, that is, creating the same number of processes for a certain number of tasks for parallel processing; it can also be a combination of serial processing and parallel processing, that is, creating a second number of processes for a first number of tasks, each process processes multiple tasks in sequence, tasks of a single process are serially processed, and tasks of different processes are parallel processed.

[0090] The object storage unit in the embodiment of the present application is pre-set with data storage rules, and the objects uploaded by the user are stored in the object storage unit according to the data storage rules, wherein the data storage rules are at least used to set the maximum storage duration of the object. The target object storage unit is any object storage unit, and at least one target object is selected from the target object storage unit according to the data storage rules corresponding to the target object storage unit. At least one task is generated according to the data processing rules corresponding to the target object storage unit and the to-be-executed information corresponding to at least one target object, and data processing is performed on at least one target object according to the at least one task. The present application does not establish a task when the object is uploaded. When the target object storage expires and needs to be executed, a task is generated according to the to-be-executed information of the target object and the data processing rules of the target object storage unit. This can save the storage space of the task, and the rule filtering is performed to generate the task only when it is executed, which can ensure that the rule modification takes effect immediately, and there will be no problem of re-entry into the warehouse.

[0091] The following begins with object uploading and specifically introduces the object processing scheme in the embodiment of the present application.

[0092] In the embodiment of the present application, after the object is uploaded, the task is not immediately constructed, but the object is directly stored in the corresponding target object storage unit. That is, according to the data storage rule corresponding to the target object storage unit, before selecting at least one target object from the object storage unit, it also includes:

[0093] Receive the uploaded object and determine the attribute information of the object;

[0094] Matching the object's attribute information with the data storage rule of the target object storage unit;

[0095] If it is determined that the attribute information of the object successfully matches the data storage rule of the target object storage unit, the object is saved in the target object storage unit.

[0096] In the specific implementation process, multiple object storage units are set in the server, and each object storage unit is set with data storage rules and data processing rules. The data storage rules and data processing rules of the same object storage unit may overlap or may not overlap, that is, the data storage rules of the object storage unit may include data processing rules, or the data storage rules may not include data processing rules.

[0097] For example, the target object storage unit is set with 50 rules, of which 25 are data storage rules and the other 25 are data processing rules. When storing the target object, the attribute information of the target object is matched with the 25 data storage rules of the target object storage unit. If all matches are successful, the target object is stored in the target object storage unit. In addition, since the objects stored in the target object storage unit need to execute the data processing rules of the target object storage unit, when the target object in the target object storage unit reaches the storage period, data processing is performed according to the 25 data processing rules corresponding to the target object and the task to be executed information generation of the target object.

[0098] In another embodiment, the target object storage unit is provided with 50 rules, of which 30 are data storage rules, and 10 of the data storage rules are also data processing rules, so the number of data processing rules is 30. That is to say, when the target object is stored, the attribute information of the target object needs to be matched with the 30 data storage rules. The 10 data processing rules need to be matched not only when the target object is stored, but also when the task is established, the task of the target object needs to be established according to the 10 data processing rules.

[0099] In another embodiment, the target object storage unit is provided with 50 rules, all of which are data storage rules, of which 25 rules are also data processing rules, that is, the number of data processing rules is 25. Thus, when the target object is stored, the attribute information of the object needs to be matched with all 50 data storage rules. The 25 data processing rules need to be matched not only when the target object is stored, but also when a task is established, the task of the target object needs to be established according to the 25 data processing rules.

[0100] In addition, after receiving the target object uploaded by the user, the target object is matched with multiple object storage units set in the server in sequence. For example, if there are 10 object storage units set in the server, the target object is matched with the data storage rule of the first object storage unit. If the match is successful, the target object is stored in the first object storage unit; if the match is unsuccessful, the target object is matched with the data storage rule of the second object storage unit. If the match is successful, the target object is stored in the second object storage unit; if the match is unsuccessful, the target object is matched with the data storage rule of the third object storage unit, and so on. In particular, if the target object does not match the data storage rules of all object storage units, the target object can be considered as an erroneous object, and will not be stored, and feedback will be given to the user.

[0101] Furthermore, since each object storage unit has many rules set, directly performing rule matching places great pressure on the system processing. Therefore, the embodiment of the present application formats the rules set for the object storage unit, optimizes CPU usage by matching vectors with vector sets, and reduces matching time.

[0102] Before receiving the uploaded object, it also includes:

[0103] For each object storage unit, a corresponding data storage rule is set respectively, and each obtained data storage rule is converted into a corresponding unit feature vector respectively;

[0104] Match the object's attribute information with the data storage rules of the target object storage unit, including:

[0105] Convert the attribute information of the object into the corresponding object feature vector;

[0106] Matching the object feature vector with the cell feature vector corresponding to the target object storage cell;

[0107] Determining that the object's attribute information successfully matches the data storage rule of the object storage unit includes:

[0108] It is determined that the object feature vector is identical to the cell feature vector corresponding to the target object storage cell.

[0109] In the specific implementation process, a bucket can be configured with 1,000 rules, where data storage rules include multiple conditions and operations such as ID, prefix, action, days, tags, etc. On the other hand, the attribute information of each object includes object name (Objname), modification time (ModifyTime), tags (Tags), version (Version), etc. Matching such complex data storage rules with the attribute information of the object consumes a lot of CPU.

[0110] Therefore, in the embodiment of the present application, the attribute information of the object and the data storage rules of the object storage unit are formatted, the data storage rules of the object storage unit are formatted into unit feature vectors, and the attribute information of each object is formatted into an object feature vector, and the matching between vectors replaces the matching between attribute information and data storage rules, thereby achieving the effect of optimizing CPU usage and reducing matching time. In addition, since the data storage rules of the object storage unit can set multiple rules for the same attribute, for example, the data storage rules of an object storage unit stipulate the prefix and label of the object, including: the prefix is ​​null, and the label is any one of a, b, c, d, and e; the prefix is ​​tmp, and the label is e or f; the prefix is ​​tmp2, and the label is g. After the above data storage rules are converted into unit feature vectors respectively, a vector set can be formed. If the object feature vector of the object matches any unit feature vector in the vector set, it means that the object feature vector matches the vector set, that is, the object matches the object storage unit.

[0111] Specifically, the vectorization process of the data storage rule of the object storage unit is as follows:

[0112] Step 1: Initialize the unit feature vector mp1.

[0113] Step 2: Assume that the content of the data storage rule includes a prefix and a tag, and add anchor points to mp1_prefix and mp1_tag, such as 000000010010101.

[0114] Step 3: Add anchor points to mp2, where the format of mp2 is: prefix anchor point -> rule corresponding to the anchor point. Get the unit feature vector value of the data storage rule.

[0115] Step 4: Finally, the vector value of the data storage rule is obtained and represented by roleinfo.

[0116] The following example illustrates this.

[0117] The data storage rules of the object storage unit are:

[0118] Rule 1-> prefix: null; tags: a, b, c, d, e;

[0119] Rule 2-> prefix: tmp; tags: e, f;

[0120] Rule 3-> prefix: tmp; tags: e, f;

[0121] Rule 4->Prefix: tmp2; tags: g.

[0122] Generate according to the above data storage rules:

[0123] mp1_prefix; Null:1,Tmp:2,Tmp2:3;

[0124] mp1_tag;A:4,B:5,C:6,D:7,E:8,F:9,G:10.

[0125] Then generate the vector:

[0126] Rule 1: 1+4+5+6+7+8;

[0127] Rule 2: 2+8+9;

[0128] Rule 3: 2+8+9;

[0129] Rule 4: 3+10.

[0130] And generate vector prefixes:

[0131] Rule 1:1;

[0132] Rule 2:2;

[0133] Rule 3:2;

[0134] Rule 4:3.

[0135] Further generate the unit feature vector mp_rule:

[0136] 1 (prefix analogy) -> Rule 1

[0137] 2 (prefix analogy) -> Rule 2, Rule 3

[0138] 3 (prefix analogy) -> Rule 4

[0139] Finally, the above rules are set in the bucket's cache.

[0140] It should be noted that, in the unit feature vector of the generated data storage rule, if the spare bits are considered to be all 0, the position of the first 1 is called the vector prefix. When matching the rules, the vector prefix is ​​compared first. If the vector prefix matches, the corresponding vector is fully matched to obtain the corresponding rule. According to the above embodiment, if the vector prefix of the object is 2, rule 2 and rule 3 are matched, and the matching workload is reduced by 2 / 3.

[0141] Furthermore, in order to improve the execution effect of the task, the storage layout of the task is optimized and the task DB is partitioned. Then, after generating at least one task according to the data processing rules corresponding to the object storage unit and the to-be-executed information corresponding to at least one target object, before executing data processing on at least one target object according to the at least one task, the method further includes:

[0142] According to the attribute information of at least one target object, in any preset storage unit, determine a task storage partition corresponding to at least one task, wherein the task storage unit includes a plurality of task storage partitions;

[0143] At least one task is stored in a corresponding task storage partition.

[0144] In the specific implementation process, the task is stored in an arbitrary storage unit. Generally, the storage space of an arbitrary storage unit is a set value, and multiple tasks can be stored in one task storage unit. When the task process is used to execute an arbitrary storage unit, in order to improve the execution efficiency of the task, each arbitrary storage unit is partitioned to obtain multiple task storage partitions. During the task storage process, the corresponding task storage partition can be determined based on the attribute information of the target object corresponding to the task, such as performing a hash calculation on the attribute information, and then the task is stored in the corresponding task storage partition.

[0145] For example, Figure 6 Figure 1 shows a schematic diagram of task storage partitioning. Figure 6 As shown, three task storage units are set, namely task DB1, task DB2, and task DB3. Each task DB is divided into three task storage partitions, and each task storage partition can store multiple tasks. It should be noted that the number of task storage partitions divided into different task DBs can be the same or different.

[0146] The embodiment of the present application can create a task process (worker) to perform tasks. Further, the task storage partition is provided with a status mark to indicate whether the task in the task storage partition has been completed. It can be clearly seen that the task process is only selected in the task storage partition whose status is marked as unfinished to prevent repeated execution of tasks.

[0147] Specifically, before performing data processing on at least one target object according to at least one task, the method further includes:

[0148] Get at least one task from at least one task storage partition marked as uncompleted;

[0149] After performing data processing on at least one target object according to at least one task, the method further includes:

[0150] After determining that all tasks in at least one task storage partition have been executed, change the at least one task storage partition mark to completed.

[0151] In the specific implementation process, multiple task processes randomly select task storage partitions from all task storage partitions marked as unfinished to reduce the problem of multiple task processes preempting a task. At the same time, the partitions that have been executed are marked, and the task storage partitions randomly selected by the task process are only selected from the unfinished task storage partitions to prevent repeated execution of tasks. Figure 7 FIG. 1 shows a schematic diagram of multiple task processes obtaining tasks from an unfinished task storage partition. Figure 7 As shown, among the three task storage partitions of task DB3, only the bottom task storage partition is unfinished, and the four task processes (workers) in the figure all obtain tasks from this task storage partition.

[0152] Furthermore, in order to prevent repeated execution of tasks, after the task process in the embodiment of the present application preempts the task, the task is also locked to prevent repeated execution of other task processes. The above-mentioned obtaining at least one task from at least one task storage partition marked as unfinished includes:

[0153] Based on at least one created execution process, and according to the generation time of each task, obtaining at least one task in a free state from a task storage partition marked as unfinished; wherein one execution process corresponds to one task;

[0154] By at least one execution process, the state of the corresponding at least one task is changed from a free state to a locked state.

[0155] In the specific implementation process, Figure 7As shown, tasks are stored in corresponding task storage partitions according to the generation time. For example, in the first task storage partition of task DB1, task A is the first generated task, task B is the second generated task, and task C is the third generated task. Then task A, task B, and task C are stored in sequence. After task process worker1 randomly selects the task storage partition, it selects task A to execute according to the storage location of task A, task B, and task C, or according to the generation time of task A, task B, and task C. At this time, task process worker1 corresponds to task A, and task process worker1 changes the state of task A from free state to locked state. At this time, task process worker2 also randomly selects the task storage partition. Although task A is still the first generated task, since the state of task A is locked, it indicates that task A has been executed by other task processes. Then task process worker2 selects from the remaining tasks, that is, selects task B to preempt. Similarly, task process worker2 also changes the state of task B from free state to locked state.

[0156] Furthermore, when a task process is abnormal and the task execution is not completed, other task processes can re-preempt the unfinished task and continue to execute it. Specifically, according to the generation time of each task, after obtaining at least one task in a free state from the task storage partition marked as unfinished, and before performing data processing on at least one target object according to the at least one task, it also includes:

[0157] adding lease time information to a state of corresponding at least one task by at least one execution process;

[0158] According to at least one task, data processing is performed on at least one target object, further comprising:

[0159] After the lease time information corresponding to any task is determined to expire, any task changes from the locked state to the free state;

[0160] Obtain any task through at least one execution process other than the execution process, and change the state of any task to a locked state.

[0161] During the specific implementation process, the execution of the task by the task process may be successful, or it may fail. When the task process fails due to an abnormality, other task processes must continue to execute the task. In order to meet the above needs, the embodiment of the present application sets lease time information for each task. Generally speaking, the length of the lease time is greater than the processing time of the normal task process for the task. Therefore, the processing of the task by the task process needs to be completed within the lease time. If the task is not completed after the lease time expires, it is considered that the executed task process is abnormal, the task execution fails, and the locked state of the task automatically changes to a free state. After the task is converted to a free state, the remaining task processes can re-seize the task and continue to execute the task.

[0162] Figure 8 Figure 1 shows a schematic diagram of how the remaining task processes continue to execute the task after the task fails. Figure 8 As shown by the solid arrows in , during the first round of task execution, task process worker1 selected the task in task storage partition 1, task process worker2 selected the task in task storage partition 2, and task process worker3 selected the task in task storage partition 4. Among them, task process worker2 executes task D, changes the state of task D to the locked state, and sets the lease time for task D. Task process worker2 encounters an exception during the execution of task D, resulting in the failure of task D. In this way, after the lease time of task D expires, the state of task D changes from the locked state to the free state. After task process worker1 executes the tasks in task storage partition 1, as shown in Figure 8 As shown by the dotted arrow in the middle, task storage partition 2 is selected. Since the state of task D is free at this time, task process worker 1 preempts task D and changes the state of task D from free to locked.

[0163] In this way, during the execution of the task, the status will be constantly modified, including the lease time and the task object offset. When the task process is abnormal and the task is not completed, other task processes can re-preempt the task after the lease time expires and continue to execute according to the offset, thus ensuring the smooth execution of the task.

[0164] Furthermore, since the number of objects processed by the system is very large, in order to prevent overload, the embodiment of the present application manages it through multi-level frequency control. Specifically, obtaining at least one task from at least one task storage partition marked as unfinished includes:

[0165] In at least one task storage partition, determining the task concurrency of the object storage unit associated with each task;

[0166] At least one task is selected from the tasks associated with the target object storage unit whose task concurrency is lower than the set threshold.

[0167] During the specific implementation, a task concurrency threshold is set for the object storage unit. The calculation of the threshold is as follows:

[0168] autoThreshold=2×(sqrt(x)+log2(x))……Formula 1

[0169] Among them, autoThreshold is the task concurrency threshold of the object storage unit, and x is the total number of tasks in the current time period.

[0170] Fig. 9 A curve chart showing the total number of tasks for the day and the task concurrency threshold of the object storage unit is shown in FIG. Fig. 9 As shown, the task concurrency threshold of each object storage unit is dynamically determined by the amount of tasks generated in the current time period. Generally speaking, the current time period here is set to one day, that is, the task concurrency threshold of the object storage unit is determined by the amount of tasks generated on that day.

[0171] In this way, when the task process selects a task, it will determine the number of tasks in the object storage unit corresponding to the task, that is, the task concurrency of the object storage unit. When the number of tasks corresponding to the object storage unit is lower than the set threshold, the task process will select a task from the tasks corresponding to the object storage unit for execution.

[0172] In this way, multiple object storage units are scheduled fairly to prevent object storage units with large task volumes from occupying the execution load of all task processes.

[0173] In addition, the embodiment of the present application is also provided with an overload fuse mechanism. If the object storage unit returns an overload-related error code during task execution, the fuse is directly blown and the object storage unit will no longer schedule and execute the task within 5 minutes.

[0174] The following is a specific example to illustrate the implementation process of the data management method provided in the embodiment of the present application. The overall architecture of the data management system in the specific implementation is as follows: Fig.10 shown.

[0175] Every day, a list subtask is generated for the user-configured Bucket and executed by the list system. After the list is executed, the filtered objects are packaged as files and stored in the object storage system, and an execution statistics file named manifest is generated. The lifecycle system is notified; the lifecycle system completes the execution of the filtered object task.

[0176] The specific task execution process is as follows: Fig.11Each task is a CSV file stored in the object storage system. Each line of the file represents the information of an object to be executed, such as Fig.11 As shown in the figure, the life cycle parses the pending execution information of each row of objects according to the Schema (a collection of database objects, generally one user corresponds to one Schema) information, queries the current object status, and executes it in combination with the rules. However, it is more complicated for multi-version objects. Since the timestamps of multi-version objects are dependent on the previous and subsequent historical versions, the execution of multi-version objects needs to be partially reversed.

[0177] When executing tasks in the lifecycle, an execution report function is required for each Bucket. Therefore, after receiving the list summary task, a Finish task will be created to collect the execution completion information.

[0178] Corresponding to the above method embodiment, the embodiment of the present application also provides a data management device. Fig.12 Schematic diagram of the structure of the data management device provided in the embodiment of the present application; Fig.12 As shown, the data management device includes:

[0179] A screening unit 121 is used to select at least one target object from the target object storage unit according to a data storage rule corresponding to the target object storage unit and a storage time point when the object is stored in the target object storage unit, wherein the data storage rule at least includes a maximum storage time of the object; the data storage rule is used to compare with the attribute information of the object when the object is stored to determine the object storage unit of the object;

[0180] The creation unit 122 is used to generate at least one task according to the data processing rule corresponding to the target object storage unit and the to-be-executed information corresponding to the at least one target object; the target object is stored in the target object storage unit after being uploaded, and the at least one task is generated when the storage time length in the target object storage unit is greater than the maximum storage time length;

[0181] The execution unit 123 is used to perform data processing on the at least one target object according to the at least one task.

[0182] Optionally, a matching unit 124 is further included, configured to:

[0183] Receiving an uploaded object and determining attribute information of the object;

[0184] Matching the attribute information of the object with the data storage rule of the target object storage unit;

[0185] If it is determined that the attribute information of the object successfully matches the data storage rule of the target object storage unit, the object is saved in the target object storage unit.

[0186] Optionally, the matching unit 124 is specifically configured to:

[0187] For each object storage unit, a corresponding data storage rule is set respectively, and each obtained data storage rule is converted into a corresponding unit feature vector respectively;

[0188] Converting the attribute information of the object into a corresponding object feature vector;

[0189] Matching the object feature vector with a unit feature vector corresponding to the target object storage unit;

[0190] It is determined that the object feature vector is the same as a unit feature vector corresponding to the target object storage unit.

[0191] Optionally, a storage unit 125 is further included, for:

[0192] According to the attribute information of the at least one target object, determining a task storage partition corresponding to the corresponding at least one task in any preset storage unit, wherein the task storage unit includes a plurality of task storage partitions;

[0193] The at least one task is stored in a corresponding task storage partition.

[0194] Optionally, the task storage partition is provided with a status mark;

[0195] The execution unit 123 is further configured to:

[0196] Retrieve the at least one task from at least one task storage partition marked as unfinished;

[0197] After determining that all tasks in the at least one task storage partition have been executed, the at least one task storage partition mark is changed to completed.

[0198] Optionally, the execution unit 123 is further configured to:

[0199] Based on at least one created execution process, and according to the generation time of each task, obtaining at least one task in a free state from a task storage partition marked as unfinished; wherein one execution process corresponds to one task;

[0200] Through the at least one execution process, the state of the corresponding at least one task is changed from a free state to a locked state.

[0201] Optionally, the execution unit 123 is further configured to:

[0202] adding lease time information to the state of the corresponding at least one task by the at least one execution process;

[0203] After determining that the lease time information corresponding to any one of the tasks has expired, the any one of the tasks is changed from a locked state to a free state;

[0204] The any one of the tasks is acquired through an execution process other than the at least one execution process, and the state of the any one of the tasks is changed to a locked state.

[0205] Optionally, the execution unit 123 is further configured to:

[0206] In the at least one task storage partition, determining the task concurrency of the object storage unit associated with each task;

[0207] The at least one task is selected from tasks associated with the target object storage unit whose task concurrency is lower than a set threshold.

[0208] Based on the same inventive concept, see Fig.13 As shown, the embodiment of the present application also provides a computer device 1300, which can be an electronic device such as a smart phone, a tablet computer, a laptop or a PC. Fig.13 As shown, the computer device 1300 includes a display unit 1340, a processor 1380 and a memory 1320, wherein the display unit 1340 includes a display panel 1341, which is used to display information input by a user or information provided to a user and various object selection pages of the computer device 1300, etc. In the embodiment of the present application, it is mainly used to display pages, shortcut windows, etc. of applications installed in the computer device 1300. Optionally, the display panel 1341 can be configured in the form of a liquid crystal display (LCD) or an organic light-emitting diode (OLED). The above-mentioned computer device 1300 can execute any method executed by the terminal or the processing server in the above-mentioned embodiments.

[0209] The processor 1380 is used to read the computer program and then execute the method defined by the computer program. For example, the processor 1380 reads the social application program, thereby running the application on the computer device 1300 and displaying the application page on the display unit 1340. The processor 1380 may include one or more general-purpose processors and may also include one or more digital signal processors (DSP) to perform related operations to implement the technical solutions provided in the embodiments of the present application.

[0210] The memory 1320 generally includes internal memory and external memory, and the internal memory may be a random access memory (RAM), a read-only memory (ROM), and a cache (CACHE), etc. The external memory may be a hard disk, an optical disk, a USB disk, a floppy disk, or a tape drive, etc. The memory 1320 is used to store computer programs and other data, and the computer program includes an application corresponding to the application, etc., and other data may include data generated after the operating system or the application is run, and the data includes system data (such as configuration parameters of the operating system) and user data. In the embodiment of the present application, program instructions are stored in the memory 1320, and the processor 1380 executes the program instructions stored in the memory 1320 to implement any method of execution of the terminal or processing server discussed above.

[0211] In addition, the computer device 1300 may also include a display unit 1340 for receiving input digital information, character information or contact touch operation / contactless gesture, and generating signal input related to user settings and function control of the computer device 1300. Specifically, in the embodiment of the present application, the display unit 1340 may include a display panel 1341. The display panel 1341, such as a touch screen, can collect the user's touch operation on or near it (such as the player using any suitable object or accessory such as a finger, stylus, etc. on the display panel 1341 or on the display panel 1341) and drive the corresponding connection device according to a pre-set program. Optionally, the display panel 1341 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the touch position of the user, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact point coordinates, and then sends it to the processor 1380, and can receive and execute commands sent by the processor 1380.

[0212] The display panel 1341 may be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the display unit 1340, the computer device 1300 may further include an input unit 1330, which may include but is not limited to one or more of a physical keyboard, a function key (such as a volume control key, a switch key, etc.), a trackball, a mouse, and a joystick. Fig.13 In the figure, the input unit 1330 includes an image input device 1331 and other input devices 1332 as an example.

[0213] In addition to the above, the computer device 1300 may also include a power supply 1390 for supplying power to other modules, an audio circuit 1360, a near field communication module 1370, and an RF circuit 1310. The computer device 1300 may also include one or more sensors 1350, such as an accelerometer, a light sensor, a pressure sensor, etc. The audio circuit 1360 specifically includes a speaker 1361 and a microphone 1362, etc. For example, the user can use voice control, the computer device 1300 can collect the user's voice through the microphone 1362, can be controlled by the user's voice, and when the user needs to be prompted, the corresponding prompt sound is played through the speaker 1361.

[0214] Based on the same inventive concept, an embodiment of the present application provides a computer-readable medium. When the instructions in the above-mentioned computer-readable storage medium are executed by a processor, the above-mentioned processor can execute any one of the methods executed by the terminal or processing server in the above-mentioned embodiments.

[0215] Alternatively, the computer-readable medium may be a non-transitory computer-readable storage medium, for example, the non-transitory computer-readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0216] Based on the same inventive concept, the embodiment of the present application provides a computer program product or a computer program, which includes a computer instruction, and the computer instruction is stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes any one of the methods executed by the terminal and the processing server in the above-mentioned embodiments.

[0217] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable signal medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0218] The program product of the embodiment of the present application can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a computing device. However, the program product of the present application is not limited to this. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with a command execution system, device or device.

[0219] The readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein the readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with a command execution system, apparatus, or device.

[0220] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0221] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0222] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A data management method, characterized in that: include: According to the data storage rule corresponding to the target object storage unit and the storage time point of the object stored in the target object storage unit, at least one target object with storage expiration is selected from the target object storage unit, wherein the data storage rule at least includes the maximum storage duration of the object; the data storage rule is used to compare with the attribute information of the object when the object is stored to determine the object storage unit of the object; At least one task is generated according to the data processing rule corresponding to the target object storage unit and the to-be-executed information corresponding to the at least one target object that has expired; the target object is stored in the target object storage unit after being uploaded, and the at least one task is generated when the storage time length in the target object storage unit is greater than the maximum storage time length; According to the attribute information of the at least one target object, determining a task storage partition corresponding to the corresponding at least one task in a preset task storage unit, wherein the task storage unit includes a plurality of task storage partitions; and storing the at least one task in the corresponding task storage partition; According to the at least one task, data processing is performed on the at least one target object.

2. The method according to claim 1, characterized in that Before selecting at least one target object from the object storage unit according to the data storage rule corresponding to the target object storage unit and the storage time point of the object stored in the target object storage unit, the method further includes: Receiving an uploaded object and determining attribute information of the object; Matching the attribute information of the object with the data storage rule of the target object storage unit; If it is determined that the attribute information of the object successfully matches the data storage rule of the target object storage unit, the object is saved in the target object storage unit.

3. The method according to claim 2, characterized in that Before receiving the uploaded object, the method further includes: For each object storage unit, a corresponding data storage rule is set respectively, and each obtained data storage rule is converted into a corresponding unit feature vector respectively; The matching of the attribute information of the object with the data storage rule of the target object storage unit includes: Converting the attribute information of the object into a corresponding object feature vector; Matching the object feature vector with a unit feature vector corresponding to the target object storage unit; The determining that the attribute information of the object successfully matches the data storage rule of the object storage unit includes: It is determined that the object feature vector is the same as a unit feature vector corresponding to the target object storage unit.

4. The method according to claim 1, characterized in that The task storage partition is provided with a status mark; Before performing data processing on the at least one target object according to the at least one task, the method further includes: Retrieve the at least one task from at least one task storage partition marked as unfinished; After performing data processing on the at least one target object according to the at least one task, the method further includes: After determining that all tasks in the at least one task storage partition have been executed, the at least one task storage partition mark is changed to completed.

5. The method according to claim 4, characterized in that The obtaining the at least one task from at least one task storage partition marked as unfinished comprises: Based on at least one created execution process, and according to the generation time of each task, obtaining at least one task in a free state from a task storage partition marked as unfinished; wherein one execution process corresponds to one task; Through the at least one execution process, the state of the corresponding at least one task is changed from a free state to a locked state.

6. The method according to claim 5, characterized in that After acquiring at least one task in a free state from the task storage partition marked as unfinished according to the generation time of each task, and before performing data processing on the at least one target object according to the at least one task, the method further includes: adding lease time information to the state of the corresponding at least one task by the at least one execution process; The performing data processing on the at least one target object according to the at least one task further includes: After determining that the lease time information corresponding to any one of the tasks has expired, the any one of the tasks is changed from a locked state to a free state; The any one of the tasks is acquired through an execution process other than the at least one execution process, and the state of the any one of the tasks is changed to a locked state.

7. The method according to claim 4, characterized in that The obtaining the at least one task from at least one task storage partition marked as unfinished comprises: In the at least one task storage partition, determining the task concurrency of the object storage unit associated with each task; The at least one task is selected from tasks associated with the target object storage unit whose task concurrency is lower than a set threshold.

8. A data management device, characterized in that: include: A screening unit, configured to select at least one target object with storage expiration from the target object storage unit according to a data storage rule corresponding to the target object storage unit and a storage time point when the object is stored in the target object storage unit, wherein the data storage rule at least includes a maximum storage duration of the object; the data storage rule is used to compare with the attribute information of the object when the object is stored to determine the object storage unit of the object; A creation unit, configured to generate at least one task according to a data processing rule corresponding to the target object storage unit and to-be-executed information corresponding to the at least one target object that has expired; the target object is stored in the target object storage unit after being uploaded, and the at least one task is generated when the storage time length in the target object storage unit is greater than the maximum storage time length; An execution unit is used to determine, based on the attribute information of the at least one target object, a task storage partition corresponding to at least one task in a preset task storage unit, wherein the task storage unit includes a plurality of task storage partitions; store the at least one task in the corresponding task storage partition; and perform data processing on the at least one target object based on the at least one task.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: It stores a computer program executable by a computer device. When the program is run on the computer device, the computer device executes the steps of any method described in claims 1 to 7.