Data processing method and device, electronic equipment, storage medium and program product

Abstract

The invention provides a data processing method and device, electronic equipment, a storage medium and a program product. The method comprises the steps that data records of each data file in a database are traversed, the data records comprise data write-in time, and each traversed data record is processed as follows: the sum between the data write-in time and effective duration is obtained, and in response to the fact that the sum is earlier than the current moment, a target data record corresponding to the sum is determined, the target data record is a data record to which the corresponding data writing time belongs, querying an entry node of the data file based on the target data record, and deleting the target data record and the data corresponding to the target data record based on the entry node. Through the method, the elimination efficiency of the expired data can be improved.

Description

Technical Field

[0001] The present application relates to data processing technology, and in particular to a data processing method, device, electronic device, storage medium and program product. Background Art

[0002] In distributed storage databases, data files often expire. To conserve memory usage and ensure that limited memory space is available for more valuable data, databases support setting expiration times to retain written data for a limited period of time. Typically, databases create indexes and delete expired data by querying these indexes. Building additional indexes increases disk storage space consumption, while not creating indexes requires scanning the entire table, consuming significant read time.

[0003] In the related technologies, there is currently no better way to improve the efficiency of eliminating expired data. Summary of the Invention

[0004] The embodiments of the present application provide a data processing method, device, electronic device, storage medium, and program product, which can improve the efficiency of eliminating expired data.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] This embodiment of the present application provides a data processing method, the method comprising:

[0007] Traversing the data records of each data file in the database, wherein the data records include: data writing time;

[0008] The following processing is performed on each of the data records traversed:

[0009] Obtaining the sum of the data writing time and the effective duration;

[0010] In response to the sum being earlier than the current time, determining a target data record corresponding to the sum, wherein the target data record is a data record to which a data writing time corresponding to the sum belongs;

[0011] querying an entry node of the data file based on the target data record;

[0012] The target data record and data corresponding to the target data record are deleted based on the entry node.

[0013] An embodiment of the present application provides a data processing device, including:

[0014] The data record acquisition module is used to traverse the data records of each data file in the database, wherein the data records include: data writing time;

[0015] a record determination module, configured to perform the following processing on each of the traversed data records: obtaining a sum of the data write time and the effective duration; in response to the sum being earlier than the current time, determining a target data record corresponding to the sum, wherein the target data record is the data record to which the data write time corresponding to the sum belongs;

[0016] The data deletion module is configured to query the entry node of the data file based on the target data record; and delete the target data record and the data corresponding to the target data record based on the entry node.

[0017] An embodiment of the present application provides an electronic device, comprising:

[0018] a memory for storing computer-executable instructions;

[0019] The processor is used to implement the data processing method provided in the embodiment of the present application when executing the computer-executable instructions or computer programs stored in the memory.

[0020] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions for implementing the data processing method provided in the embodiment of the present application when executed by a processor.

[0021] An embodiment of the present application provides a computer program product, including a computer program or computer-executable instructions. When the computer program or computer-executable instructions are executed by a processor, the data processing method provided in the embodiment of the present application is implemented.

[0022] The embodiments of the present application have the following beneficial effects:

[0023] The data record corresponding to the data includes at least the write time. By judging the relationship between the write time, the sum of the effective time and the current time, it is determined whether the data corresponding to the data record is expired. By querying the entry node of the data file through the target data record, the data file is called, and the expired data file can be determined in a targeted manner. Compared with the solution of additionally building an index and scanning the entire table data in the related technology, it can save the computing resources required to determine and delete expired data. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is a schematic diagram of an application mode of the data processing method provided in an embodiment of the present application;

[0025] Figure 2 is a structural diagram of an electronic device provided in an embodiment of the present application;

[0026] Figure 3AThis is a first flow chart of the data processing method provided in an embodiment of the present application;

[0027] Figure 3B This is a second flow chart of the data processing method provided in an embodiment of the present application;

[0028] Figure 3C 3 is a schematic diagram of a third flow chart of the data processing method provided in an embodiment of the present application;

[0029] Figure 4 4 is a schematic diagram of a fourth flow chart of a data processing method provided in an embodiment of the present application;

[0030] Figure 5 This is a hash index architecture diagram provided by an embodiment of the present application;

[0031] Figure 6 This is a first schematic diagram of the organization of entries and data files provided in an embodiment of the present application;

[0032] Figure 7 This is a second schematic diagram of the organization of entries and data files provided in an embodiment of the present application;

[0033] Figure 8 This is a time statistics diagram of the data file provided in the embodiment of the present application;

[0034] Figure 9A 5 is a schematic diagram of a fifth flow chart of a data processing method provided in an embodiment of the present application;

[0035] Figure 9B This is the sixth flow chart of the data processing method provided in the embodiment of the present application. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0037] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0038] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0039] It should be pointed out that the collection and processing of relevant data in this application should be strictly in accordance with the requirements of relevant national laws and regulations when applied in practice, and the informed consent or separate consent of the personal information subject should be obtained. Subsequent data use and processing should be carried out within the scope of authorization of laws and regulations and the personal information subject.

[0040] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0041] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0042] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0043] 1) MongoDB: An open-source database system based on distributed file storage, written in C++. MongoDB provides document-oriented storage, simple operation, supports "schemaless" data modeling, and can store complex data types.

[0044] 2) Log structure: A data storage system whose log structure is designed based on log files and is used to record data modification and update operations in a log in sequence.

[0045] 3) Storage Engine: The component or module in a database system that is responsible for data storage and retrieval, defining how data is organized, stored, and accessed on disk or in memory.

[0046] 4) Index: A data structure in a database used to improve data retrieval efficiency. It can quickly locate the location of stored data, thereby speeding up queries.

[0047] 5) Key: Used to uniquely identify a record in the database. Associated with the key is the value, which is used to store the data associated with the key.

[0048] 6) Key-value (KV) database: It is a database that uses key-value storage, and its data is organized, indexed, and stored in the form of key-value pairs.

[0049] 7) Hash index: Used to map key values ​​to storage buckets based on hash functions, thereby achieving fast data access. Different key values ​​may produce the same hash value, resulting in hash conflicts. Common solutions include the zipper method.

[0050] 8) Zipper method: Maintain a linked list in each bucket of the hash table and store key-value pairs with the same hash value in the same bucket.

[0051] 9) Time to Live (TTL): Indicates the survival time or validity period of the data. If it expires, the data needs to be eliminated.

[0052] 10) Unix timestamp: The number of seconds that have passed since January 1, 1970, can be converted into the corresponding year, month, day, hour, minute and second.

[0053] 11) Data cleansing: The process of eliminating data holes in the database by rebuilding data files or other data reorganization strategies. The holes are caused by the storage engine deleting or updating data.

[0054] Embodiments of the present application provide a data processing method, a data processing device, an electronic device, and a computer-readable storage medium, which can improve the efficiency of eliminating expired data.

[0055] The following describes exemplary applications of electronic devices provided in embodiments of the present application. The electronic devices provided in embodiments of the present application can be implemented as terminal devices, such as laptop computers, tablet computers, desktop computers, set-top boxes, smart TVs, vehicle-mounted terminals, virtual reality (VR) devices, augmented reality (AR) devices, and other types of user terminals, and can also be implemented as servers. The following describes exemplary applications of electronic devices implemented as terminal devices or servers.

[0056] See also Figure 1 , Figure 1 This is a schematic diagram of an application mode of the data processing method provided in an embodiment of the present application; for example, Figure 1The server 200, the network 300 and the terminal device 400 are involved. The terminal device 400 is connected to the server 200 via the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0057] In some embodiments, the user is a technical personnel in this field who sets the expiration time (i.e., the effective duration of the data), the server 200 is a server that executes the deletion of data records, for example: the server of a distributed data processing platform, the terminal device 400 is a terminal operated by the user, the terminal device 400 is installed with an application for the user to set the elimination time, and the database 500 stores the data records stored by the distributed data processing platform.

[0058] For example, the user-operable terminal device 400 is used to receive the expiration and elimination time of the table set by the user. The terminal device 400 sends the acquired expiration and elimination time to the server 200 through the network 300. The server 200 performs corresponding operations on the data records read from the database 500 according to the expiration and elimination time sent by the network 300, deletes the expired corresponding data records and data files, and realizes the elimination of expired data.

[0059] The embodiments of the present application can be implemented using database technology. A database, in short, can be considered an electronic filing cabinet that stores electronic files, allowing users to add, query, update, and delete data in these files. A "database" is a collection of data that is stored together in a specific manner, can be shared by multiple users, has minimal redundancy, and is independent of applications.

[0060] A database management system (DBMS) is a computer software system designed for managing databases, typically providing basic functions such as storage, retrieval, security, and backup. DBMSs can be categorized by the database model they support, such as relational or XML (Extensible Markup Language); by the type of computer they support, such as server clusters or mobile phones; by the query language they use, such as SQL or XQuery; by performance priorities, such as maximum scale or maximum speed; or by other classification methods. Regardless of the classification method used, some DBMSs are cross-category, for example, supporting multiple query languages ​​simultaneously.

[0061] The embodiments of the present application can also be implemented through cloud technology. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model. It can form a resource pool that can be used on demand and is flexible and convenient. Cloud computing technology will become an important support. The background services of the technical network system require a large amount of computing and storage resources, such as video websites, picture websites and more portal websites. With the rapid development and application of the Internet industry, as well as the promotion of search services, social networks, mobile commerce and open collaboration, each item may have its own hash code identification mark in the future, and all of them need to be transmitted to the background system for logical processing. Data of different levels will be processed separately. All kinds of industry data require strong system backing support, which can only be achieved through cloud computing.

[0062] In some embodiments, the server 200 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The electronic device can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited to this. The terminal device and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0063] See also Figure 2 , Figure 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may be Figure 1 Server 200 in Figure 2 The server 200 shown includes: at least one processor 410, a memory 450 and at least one network interface 420. The various components in the server 200 are coupled together via a bus system 440. It is understood that the bus system 440 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 440 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 440 is not described in detail. Figure 2 Various buses are labeled as bus system 440 .

[0064] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0065] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.

[0066] The memory 450 includes volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0067] In some embodiments, the memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplified below.

[0068] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, which are used to implement various basic services and process hardware-based tasks;

[0069] A network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420. Exemplary network interfaces 420 include Bluetooth, Wi-Fi, and Universal Serial Bus (USB).

[0070] In some embodiments, the apparatus provided in the embodiments of the present application may be implemented in software. Figure 2 The data processing device 455 stored in the memory 450 is shown. The data processing device 455 can be a software in the form of a program or plug-in, and includes the following software modules: a data record acquisition module 4551, a record determination module 4552, and a data deletion module 4553. These modules are logical and can be arbitrarily combined or further divided according to the functions implemented. Figure 2 For the sake of convenience, all the above modules are shown at once, but it should not be considered that the data processing device 455 excludes the implementation of only including the data record acquisition module 4551. The functions of each module will be explained below.

[0071] In some embodiments, the terminal or server can implement the data processing method provided in the embodiments of the present application by running a computer program. For example, the computer program can be a native program or software module in the operating system; it can be a native application (APP, Application), that is, a program that needs to be installed in the operating system to run, such as a live broadcast APP or an instant messaging APP; it can also be a small program, that is, a program that can be run only by downloading it into a browser environment; it can also be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be any form of application, module or plug-in.

[0072] The data processing method provided in the embodiment of the present application will be explained in combination with the exemplary application and implementation of the server provided in the embodiment of the present application.

[0073] The following describes the data processing method provided by the embodiment of the present application. As mentioned above, the electronic device that implements the data processing method of the embodiment of the present application can be a terminal or a server, or a combination of the two. Therefore, the execution entity of each step will not be repeated below.

[0074] See also Figure 3A , Figure 3A This is a first flow chart of the data processing method provided in the embodiment of the present application, which will be combined with Figure 3A The steps shown are explained.

[0075] In step 301, the data records of each data file in the database are traversed.

[0076] Here, the data record includes: data writing time.

[0077] For example, the traversal of the data records to be processed is a sequential traversal according to the data write time. The database in the embodiment of the present application is a distributed log-structured database, which is used to set the expired elimination function for the specified data table and implement the data elimination operation on the table in the background. The data file is a file used to write data in the database. The data file and the hash index constitute the storage engine of the database. Independent data records are stored in the data file. The data file consists of three parts: header information (head), key (key) and value (value). The data record corresponds to a complete set of relevant information for a row of information in the data source. The header information (head) includes the data write time.

[0078] See also Figure 7 , Figure 7This is the second schematic diagram of the entry and data file organization provided in the embodiment of the present application. The structure of the data record includes: the magic number (Magic Number) of the data file, the format version number, the header cyclic redundancy check (CRC), the format version number, the key length, the compression type, the retention information, the hash value length, the cyclic redundancy check of the key and the cyclic redundancy check of the hash value.

[0079] For example, the data writing time includes the writing start time and the writing end time. Figure 3B , Figure 3B This is a second flow chart of the data processing method provided in an embodiment of the present application. Before executing step 301, steps 3011 to 3014 are executed, which are described in detail below.

[0080] In step 3011, data to be stored is obtained.

[0081] For example, the data to be stored is obtained, the data to be stored is written into a data file, and the field information of the entry node in the hash index table of the database is updated based on the hash code of the data file.

[0082] Here, the types of field information include: the data file number and hash code. The hash code is the key of the data file, which uniquely identifies a record in the database. Associated with the key is the key value, which is used to store the data associated with the key.

[0083] For example, the field information further includes: the number of the next entry node, and the number of the next entry node is used to form an entry chain for each entry node with adjacent numbers.

[0084] See also Figure 5 , Figure 5 This is a hash index architecture diagram provided by an embodiment of the present application; Figure 5 Entry page 0 (502) and entry page 1 (503) show the field information of the entry node. In the field information of each entry in the entry page, the next entry number 0 to the next entry number 7 represent the next entry node number of the entry node. For example, the next node number of entry node 4 is 3, which means that the next node of entry node 4 is entry node 3. At this time, a hash conflict will occur, so there is a hash conflict chain (504) between entry node 3 and entry node 4. Hash conflict means that different keys of entry nodes may produce the same hash value. Hash conflict is resolved by using the zipper method. The zipper method maintains a linked list in each bucket of the hash table and stores key-value pairs with the same hash value in the same bucket.

[0085] Hash indexes are implemented using hash tables. For hash tables and their subclasses, a hash algorithm is used to determine the storage location of elements in a combination. Initializing a hash table creates an array of entries. The locations in the data used to store elements are called "buckets." A hash table bucket stores the address numbers of multiple entry nodes. An entry object contains a reference variable that points to the next entry, which is the number of the next entry node, forming an entry chain. An entry page stores multiple entries, and each entry page stores a fixed number of entry nodes. When the next entry node is numbered 0, it indicates that there is no entry node after it.

[0086] In step 3012, the data to be stored is written into the data file.

[0087] Here, the first data file is an empty file before the data to be stored is written.

[0088] For example, the process of filling an empty file with data to be stored is a data writing process. Step 3012 can be implemented as follows: performing the following process for each data to be stored: obtaining a first data file; writing the data to be stored into the first data file; and in response to the storage space occupied by the first data file reaching a preset storage capacity and there being remaining unwritten data to be stored, converting the first data file into a read-only file and creating a second data file.

[0089] For example, the second data file is used to continue writing the remaining part of the data to be stored. If the newly created data file still cannot write all the contents of the data to be stored, you can continue to create new files until all the data to be stored are written. That is, according to the needs of the actual application scenario, the number of second data files can be multiple, for example: set the size of the written data file, write the data to be stored into the data file in chronological order, and when the size of the written data reaches the preset data file size, convert the data file filled with data into a read-only file, and create a new file to continue writing the subsequent data to be stored.

[0090] See also Figure 6 , Figure 6 This is a first schematic diagram of the organization of entries and data files provided in the embodiment of the present application; Figure 6 As shown in the file being written (606), the data records to be stored are written into the data file, and the file is converted to a read-only file (605) after the data file size is met. For example, when the size of the written data file is set to 256MB, when the written data records reach 256MB, the file being written (606) is converted to a read-only file (605), and data is no longer written. A new file being written (606) is created to continue writing subsequent data.

[0091] In step 3013 , in response to the writing process of writing the data to be stored into the data file, the writing start time and the writing end time of the writing process are recorded.

[0092] In some embodiments, step 3031 can be implemented as follows: in response to the start of a write process for writing the data to be stored into the data file, a first timestamp is obtained, and the time corresponding to the first timestamp is used as the write start time. In response to the end of the write process for writing the data to be stored into the data file, a second timestamp is obtained, and the time corresponding to the second timestamp is used as the write end time, wherein the time corresponding to the second timestamp is later than the time corresponding to the first timestamp.

[0093] For example, the data write time uses the Unix timestamp of the system time, which can be converted to the corresponding year, month, day, hour, minute, and second. When writing data to each data file, the write time distribution of the data record is statistically analyzed. The time statistics are composed of the data write start time and the write end time. During the write process, the write start time is also called the earliest write time (oldest_tm), and the write end time is also called the latest write time (latest_tm).

[0094] When creating a write file, the write start time and write end time are both initialized to 0. The Unix timestamp of the data record write time is referred to as the current time. When new data is written, the write start time and write end time are updated. The update principle is: when the write end time is later than the current time (now_time), the write end time is updated to the current time. If the latest write time is earlier than the current time, no change is required. When the write start time is 0 or later than the current time, the write start time is updated to the current time. If the write start time is earlier than the current time, no change is required.

[0095] For example, the above-mentioned update process of the write start time and the write end time is performed for each file string, and the data in the data file is written in chronological order. Therefore, the time range of each data file basically does not overlap, and the data writing time of each data file basically increases in sequence.

[0096] See also Figure 8 , Figure 8: This is a time statistics diagram of the data file provided by the embodiment of the present application; the start time and end time of writing are the writing time range shown in the time statistics (802), and n in the data file n (n is a positive integer) represents the file number. The time distribution of the data file is converted into a specific year, month, day, hour, minute, and second time. For example, the data writing time range of file 1 (i.e., the writing start time to the writing end time) is: 1660001000~1660201000, which is converted into a specific year, month, day, hour, minute, and second time: 2022-08-09 07:23:20~2022-08-11 14:56:40; the data writing time range of file 2 is: 1660201000~1660301000, which is converted into a specific year, month, day, hour, minute, and second time: 2022-08-1114:56:40~2022-08-12 18:43:20; the data write time range of file n is: 1660801000~1660901000, which is converted into the specific year, month, day, hour, minute, and second time: 2022-08-18 13:36:40~2022-08-19 17:23:20.

[0097] In step 3014, the writing start time and the writing end time are stored as the data writing time in the data record of the data file.

[0098] For example, when writing data to each data file, the write time distribution of the data records is statistically analyzed, and the statistical data includes the earliest start time and the latest start time of the written data.

[0099] In some embodiments, step 301 can be implemented by: obtaining a preconfigured write time; determining the data files to be traversed in the database, wherein the data files to be traversed are data files whose write start time is before the preconfigured write time; and traversing the data records of each data file to be traversed.

[0100] For example, in actual applications, the pre-configured write time can be set according to actual needs. For example, the pre-configured write time is a time N days before the current time, where N is a positive integer.

[0101] Continue to see Figure 8 , set to retain only the data of the last N days, for example, set to retain only data within 10 days (864000 seconds), assuming the current time is 2022-08-19 17:23:20, then the data records written before 2022-08-09 17:23:20 need to be deleted. Figure 8In the time statistics of the data files, the write start time of file 1 is 2022-08-09 07:23:20. Only file 1 contains data records with a write time before 2022-08-09 17:23:20. The data records of files 2 and file n are all written after 2022-08-09 17:23:20. There is no need to read these data files with input write times after the set retention period. Only the data records in file 1 need to be read for data elimination.

[0102] In an embodiment of the present application, the target data file to be read can be accurately selected according to the data writing time, and the range of traversed data can be specified without traversing all data files, thereby improving the efficiency of eliminating expired data and saving the time of performing deletion operations.

[0103] Continue to see Figure 3A In step 302, the following processing is performed on each data record traversed: the sum of the data writing time and the effective duration is obtained.

[0104] For example, the sum of the data write time and the validity period can be used to determine whether the data record traversed in the data file is the target data record to be processed. Data in the database is time-sensitive, and the validity period is the data's lifespan or validity period. Expired data that exceeds the validity period needs to be eliminated.

[0105] For example, the validity period in the preceding example is the last 10 days. Assuming the current time is 2022-08-19 17:23:20, the sum of the earliest write time of the data in file 1 (2022-08-09 07:23:20) and the validity period (10 days) is 2022-08-19 07:23:20, indicating that there are expired data records in file 1. It is necessary to traverse the data records in file 1 and perform data elimination.

[0106] In step 303 , in response to the sum being earlier than the current time, a target data record corresponding to the sum is determined.

[0107] Here, the target data record is the data record to which the data writing time corresponding to the sum belongs.

[0108] For example, when the sum of the data write time and the effective duration is earlier than the current time, the target data record can be determined to be an expired data record that needs to be eliminated. Continuing with the above example, the current time is 2022-08-1917:23:20, and the sum of the earliest data write time (2022-08-09 07:23:20) and the effective duration (10 days) in file 1 is 2022-08-19 07:23:20, which is not earlier than the current time. At this time, there is expired data in file 1; the sum of the earliest data write time (2022-08-11 14:56:40) and the effective duration (10 days) in file 2 is 2022-08-21 14:56:40, which is later than the current time, indicating that the data record in file 2 has not expired and there is no need to traverse the data records in file 2 to perform data elimination.

[0109] In step 304, an entry node of the data file is searched based on the target data record.

[0110] For example, a hash table is a data structure that is directly accessed based on a key value. Hash indexing can be achieved through a hash table. The hash table consists of buckets and entries, which are stored in memory. The linked list head number of the entry node is stored in the same way. After calculating the hash value of the key, the bucket size is modulo to find the location of the corresponding bucket. The entry node is stored in an entry page and the size of the entry page is a fixed value, that is, each entry special stores a fixed number of entry nodes, see Figure 5 , Figure 5 This is a hash index architecture diagram provided by an embodiment of the present application; Figure 5 Entry page 0 (502) and entry page 1 (503) each store four entry nodes, and the fixed value of their entry page is 4. By obtaining the entry node number, we can find the corresponding entry page and the offset address within the entry page based on the number, and thus obtain the information in the corresponding entry node. The entry node stores the file information and file offset address of the corresponding data record. Based on the information in the entry node, the corresponding data record can be found in the data file.

[0111] See also Figure 3C , Figure 3C This is a third flow chart of the data processing method provided in an embodiment of the present application. Step 304 is implemented through steps 3041 to 3044, which are described in detail below.

[0112] In step 3041 , the hash code and the address number of the entry node included in the target data record are determined.

[0113] For clarification, see Figure 7 , Figure 7This is a second schematic diagram of the entry and data file organization provided in an embodiment of the present application; the structure of the target data record includes: header information (701), a key (702) and a hash value of the key (703), and the key of the target data record is a hash code.

[0114] In step 3042, the hash index table of the database is obtained.

[0115] Here, the hash table includes multiple buckets and entry nodes stored in each bucket.

[0116] For the sake of explanation, see Figure 5 , Figure 5 This is a diagram of a hash index architecture provided by an embodiment of the present application: the entry node stores information corresponding to the data file where the data record is located and the offset address of the data file, for example: Figure 5 Entry 4 stores the page number 4, file number 4, offset address 4, hash value 4, and the next entry number (3). The offset address of a data file refers to the offset relative to the beginning of the data file when it is stored on the disk, starting from the first byte of the file and starting at 0.

[0117] In step 3043, a hash value corresponding to the hash code is determined, and based on the hash value, a bucket position associated with the target data record in the hash index table is determined.

[0118] Continue to see Figure 5 , determining the hash value corresponding to the hash code (for example: hash value 4) can locate the entry node described by the hash value (entry node 4). The entry node also contains page number information (page number 4). According to the page number information, the position of the bucket associated with the target data record can be found in the bucket (bucket [n]).

[0119] In step 3044 , the entry node corresponding to the target data record is determined based on the bucket position and the address number of the entry node.

[0120] Continue to see Figure 5 When the bucket position is obtained, the corresponding entry node (entry node 0) can be found based on the bucket position (for example: bucket [0]), and the information in the corresponding entry node can be obtained according to the address number of the entry node (offset address 0), and the entry node (entry node 0) corresponding to the target data record can be determined.

[0121] In an embodiment of the present application, the target data record is determined based on the hash index of the hash table, and accurate positioning can be performed according to the correspondence between the node information and the data record. The search can be performed in the existing hash table without the need to additionally construct a hash index, thereby reducing the hard disk storage space occupied.

[0122] Continue to see Figure 3A In step 305, the target data record and the data corresponding to the target data record are deleted based on the entry node.

[0123] For example, the data deleted based on the entry node is the portion of the data file corresponding to expired data. After deletion, the non-expired portion of the data file can be retained; or, the data file becomes an empty file after the deletion process.

[0124] For example, step 305 can be implemented by determining the file number and file offset address included in the entry node, calling the data file corresponding to the target data record based on the file number and file offset address, and deleting the written data corresponding to the data write time of the target data record in the called data file, and deleting the entry node.

[0125] For example, after determining the target data record, when deleting the target data record, the corresponding entry node in the hash index is directly deleted, thereby achieving deletion of the target data record.

[0126] In the embodiment of the present application, there is no need to construct an additional hash index based on the survival time, which saves the hard disk space of the database that supports persistence and avoids the situation where the database read and write performance is reduced due to the need to synchronize the index.

[0127] In some embodiments, after step 305, in response to the completion of traversal of each data record in the database, the first moment is recorded, and in response to the time interval between the first moment and the current moment reaching a preconfigured duration, the execution of step 301 is entered.

[0128] For example, after setting the effective duration, the data records of each data file in the database will be traversed sequentially immediately. When all data records are traversed, it is considered the end of an execution cycle. At this time, it waits until the start time of the next cycle and executes the traversal operation on the data records in the data file again. The length of the preconfigured duration can be fixed, or adaptively adjusted according to the interval between the current moment and the last traversal operation. For example: assuming that the data written in the last T seconds is retained, T seconds is considered to be the start cycle of a data elimination task. The rules for periodically starting the elimination task are as follows: When T is greater than 86400*2 seconds (2 days), the data elimination task is started every 86400 seconds. When T is less than 86400*2 seconds (2 days), the cycle of executing the data elimination task is adaptively shortened, for example, starting the data elimination task every T / 3 seconds.

[0129] In the embodiment of the present application, data elimination processing is performed periodically, which improves data elimination efficiency, maintains the stability of data in the database, and saves database storage space.

[0130] In some embodiments, after step 305, in response to the data file after the deletion processing including residual data, the residual data is stored in a new data file, and the original data file corresponding to the residual data is deleted. In response to the data record corresponding to the data file after the deletion processing being an invalid record, the data file is deleted.

[0131] For example, the target data record that has been deleted is considered an invalid data record, and the data file storing the invalid data record is deleted, completing the elimination of the expired data file. For some data files that store some invalid data records and some valid data records, the valid data records are rewritten into a new data file, keeping the data write time of the partially valid data records in the new data file unchanged. When all valid records are written into the new data file, the original data file is deleted. At this time, the partially valid data records are all organized into the new file, and the invalid data records are eliminated.

[0132] In some embodiments, the data processing method provided in the embodiments of the present application can be applied in the field of gaming, for example, deleting expired data of resource data in a virtual game scene, as described in detail below.

[0133] For example, the resource data of online games is stored in the corresponding database. In the game application scenario, the user can be a game developer. In order to save the storage space occupied by the resource data of online games in the server database and the player's terminal device, developers usually set the data to be retained within a recent period and delete the older expired data. For example: when the developer sets the retention of system emails or game match records and other data within the last 30 days (2592000 seconds), after setting the effective period, the server immediately starts to traverse the relevant system emails or game match records stored in the database, and regards the data files before the last 30 days as expired and need to be eliminated, and performs the deletion operation. At this time, it is possible to retain only the data records within the effective period set by the developer, saving storage space.

[0134] For example, based on the different needs of developers, different expiration times can be set for system emails or game records, such as retaining the last 7 days of system emails, the last 15 days of game records, and the last 30 days of interaction records. Data records are eliminated based on their last modification time, and the data elimination task is restarted in response to the arrival of the next execution cycle. In scenarios where the expiration time is relatively precise or different records have different elimination times, expired data in the database can still be eliminated in a timely and accurate manner.

[0135] In an embodiment of the present application, the data record corresponding to the data includes at least a write time. By judging the relationship between the write time and the sum of the effective time and the current time, it is determined whether the data corresponding to the data record is expired. By querying the entry node of the data file through the target data record, the data file is called, and the expired data file can be determined in a targeted manner. Compared with the solution of additionally building an index and scanning the entire table data in the related art, the computing resources required for determining and deleting expired data can be saved.

[0136] Below, an exemplary application of the data processing method according to an embodiment of the present application in a practical application scenario will be described.

[0137] In the related art, the database supports setting the validity period for the table to save only the data written in the most recent period. When the survival time of the data in the database exceeds the validity period, the data is expired. Usually the database will create an index based on time and perform deletion operations on expired data by querying the index. The creation of a validity period index for the survival time in the MongoDB distributed file storage database will take up additional hard disk space. Operations such as writing, updating, and deleting data require the synchronous update of the validity period index for the survival time, affecting the read and write performance. In scenarios with large amounts of data, creating a validity period index for the survival time consumes a lot of time, computing resources, and hard disk storage space. Related technologies make it difficult to improve the efficiency of eliminating expired data.

[0138] How to provide a high-performance data expiration and elimination method without creating an additional index corresponding to the survival time has become a problem to be solved in the related art. The data processing method provided by the embodiment of the present application does not require the establishment of an additional index. Regardless of the current data volume, setting and canceling the expiration and elimination function can take effect immediately. In addition, executing data elimination on the server only requires scanning part of the data, and does not require reading all data records. This saves computing resources and network resources, has little impact on the server's online business, and improves the efficiency of expired data elimination.

[0139] The embodiment of the present application traverses the data records in a data file to obtain the write time and effective duration of the data, and determines the target data record corresponding to the expired data by adding them together. This eliminates the need for additional index construction, thus avoiding waste of storage space and synchronizing indexes for other data operations, without affecting read and write functions. Based on the target data record, the entry node of the data file is queried, and the records and files to be deleted are determined based on the entry node. When performing data elimination, only part of the data is scanned, which greatly reduces data reading time and can achieve efficient data elimination.

[0140] The following is a description with reference to the accompanying drawings. Figure 4 , Figure 4This is a fourth flow chart of the data processing method provided in the embodiment of the present application. The execution subject may be a terminal device, a server, or a combination of the two. The embodiment of the present application takes the server as the execution subject as an example. Figure 4 The steps shown are explained in detail.

[0141] In step 401, parameters corresponding to the elimination task are set for the designated table, and the elimination task is started.

[0142] For example, a designated table refers to a database table. Database data is stored in a table. A table is a matrix of data, a grid-like virtual table that temporarily stores data. Setting parameters for a designated table elimination task can be accomplished by setting the time period for retaining recently written data. After setting the elimination task parameters, the elimination task starts immediately.

[0143] After the task is started, the expired elimination task is executed once immediately, which is the first execution. After the first execution is completed, the expired elimination task is scheduled according to the length of time the data is retained. Assuming that the data written in the last T seconds is retained, T is a positive integer. In the parameters corresponding to the elimination task, the rules for periodically starting the elimination task are as follows: when T is greater than 86400*2 seconds (2 days), the data elimination task is started every 86400 seconds (1 day); when T is less than 86400*2 seconds (2 days), the data elimination task is started every T / 3 seconds.

[0144] In step 402, a data file to be scanned is determined, and data records in the data file are read sequentially.

[0145] For example, the data files to be scanned are stored in a log-structured database.

[0146] The storage engine of a log-structured database consists of two parts: hash index and data files. Figure 5 , Figure 5 This is a diagram of the hash index architecture provided by the embodiment of the present application. The hash index is implemented through a hash table, which consists of buckets ( Figure 5 501) and entries, all of which are stored in memory, and hash conflicts (represented as hash conflict chains 504) are resolved using a zipper method.

[0147] The bucket (501) of the hash table stores the number of the linked list head of the entry node. After calculating the hash value of the key value, the size of the bucket is modulo, the corresponding bucket position is found, and the address number of the entry node stored in the corresponding bucket is obtained. The entry node is stored in the entry page (such as Figure 50 and 1) shown in 502 and 503 respectively, and the size is fixed. Each entry page stores a fixed number of entry nodes (such as entries 0 to 3 in entry page 0 shown in 502, or entries 4 to 7 in entry page 1 shown in 503). The number of the entry node is obtained (such as Figure 5 After the file number 0 to file number 7 shown in , find the corresponding entry page and the offset address in the entry page according to the file number (such as Figure 5 The offset address 0 to the offset address 7 shown in can be used to obtain the information in the corresponding entry node.

[0148] The entry node stores the file information and file offset address of the corresponding data record. The types of field information in the entry node include: flag, fid, offset, hash, and next.

[0149] Among them, flag represents the number of storage pages occupied by special tags and data (such as Figure 5 0 to 7 in the figure); fid represents the number of the data file (such as Figure 5 0 to 7 in the data file); offset represents the offset address in the data file (such as Figure 5 As shown in offset address 0 to offset address 7); hash represents the hash value of the key (such as Figure 5 Hash value 0 to hash value 7 shown in ); next represents the number of the next entry node (such as Figure 5 , whereby an entry linked list is formed, and 0 indicates that there is no subsequent entry node.

[0150] For example, the corresponding data record can be obtained from the data file according to the information in the entry node, see Figure 6 , Figure 6 This is a first schematic diagram of the organization of entries and data files provided by the embodiment of the present application. Figure 6 1 to n in the table). See Figure 7 , Figure 7This is a second schematic diagram of the entry and data file organization provided by an embodiment of the present application. Each data record mainly consists of three parts: header information (701), key (702), and key hash value (703). The data record header information includes the write time, which is the current system time when the current data record is written to the data file. It also includes: the data file's magic number (Magic Number), format version number, header cyclic redundancy check (CRC), format version number, key length, compression type, retention information, hash value length, key cyclic redundancy check, and hash value cyclic redundancy check. The magic number is referred to as "magic number" and is the first one or several bytes of a certain type of file. A computer can determine the file type based on its value, even if the file suffix has been changed incorrectly.

[0151] The information of the entry node can find the corresponding data record in the data file, such as Figure 6 As shown, entry page 0 (607), entry page 1 (608), and entry page 2 (609) record entry node information. Based on the entry node information, the data file sequence number (601) and the offset address (602) recorded in the file can be obtained. Based on the obtained file sequence number and offset address, the target data record in the read-only file (605) can be determined. Newly written data (603) and additionally written data (604) are stored in the memory buffer (610). The memory buffer and entry pages are both stored in the memory (612). When writing data, the data is first appended to the file (604). Based on the file number of the written file, the offset address in the file, and the hash value of the key, the information in the hash index is updated.

[0152] Unlike the memory buffer and entry pages, the read-only file (605) and the file being written (606) are stored on the hard disk (613). The file being written includes the unwritten portion of the memory buffer (611). When deleting data, the corresponding entry node in the hash index is directly deleted. When the written file reaches a specified size, such as the current file size limit of 256MB, it is converted to a read-only file (605) and a new file is created as the new write file.

[0153] In step 403, time statistics of the current data file are obtained.

[0154] For example, the time statistics of a data file consist of two data: the earliest write time of the data and the latest write time of the data, which are the earliest write time (oldest_tm) and the latest write time (latest_tm).

[0155] The data write time uses the Unix timestamp of the system time, which can be converted to the corresponding year, month, day, hour, minute, and second. When writing data to each file, the write time distribution of the data records is statistically analyzed. When creating a write file, the earliest write time and the latest write time are both initialized to 0. The Unix timestamp of the data record write time is abbreviated as the current moment (now_time). The earliest write time and the latest write time are updated as follows when writing data, where "greater than" means later in time, and "less than" means earlier in time:

[0156] When the latest write time is less than the current time, the latest write time is updated to the current time. When the latest write time is greater than the current time, it will not be changed.

[0157] When the earliest write time is 0, the earliest write time is directly updated to the current time. When the earliest write time is greater than the current time, the earliest write time is updated to the current time. When the earliest write time is less than the current time, it is not changed.

[0158] The written data records are written in the data file in chronological order. The time ranges of each file do not overlap, and the writing time of each data file increases in sequence. Figure 8 , Figure 8 This is a time statistics diagram of the data file provided in the embodiment of the present application. Figure 8 It includes data files (801) and time statistics (802) corresponding to the data files, where n is the number of the data file, and n is a positive integer. The Unix timestamp of the system time can convert the time distribution of the file into specific year, month, day, hour, minute, and second.

[0159] For example, if the write time range of file 1 is 1660001000 to 1660201000, the specific conversion time is from 2022-08-09 07:23:20 to 2022-08-11 14:56:40; the write time range of file 2 is 1660201000 to 1660301000, the specific conversion time is from 2022-08-11 14:56:40 to 2022-08-12 18:43:20; and the write time range of file n is 1660801000 to 1660901000, the specific conversion time is from 2022-08-18 13:36:40 to 2022-08-19 17:23:20.

[0160] In step 404, based on time statistics, the current file is scanned and outdated data is deleted.

[0161] For example, get the time statistics of the current data record, add the write time and the set expiration time, and determine the expired data to be deleted. Figure 9A, Figure 9A This is the fifth flow chart of the data processing method provided in the embodiment of the present application.

[0162] In step 901A, data records in the data file are read sequentially.

[0163] For example, when data is recorded in a data file, it is written in chronological order, the time ranges do not overlap, and the writing time of the data file increases.

[0164] In step 902A, the write time of the current data record is read.

[0165] Here, the header information of the data record includes the writing time, which is the system time when the current data record is written into the data file.

[0166] In step 903A, when the sum of the write time and the valid time length is less than the current time, the current data record expires.

[0167] For example, the set validity period is added to the writing time of the data record. If the sum is less than the current system time, the data record is expired, and a deletion operation is performed on the expired data record to be deleted.

[0168] If you want to keep only the data of the last N days, please refer to Figure 8 For example, if only data within the last 10 days (864000 seconds) is retained and the current time is 2022-08-19 17:23:20, data records written before 2022-08-09 17:23:20 need to be deleted. According to the time statistics of the data files, only file 1 contains data records written before 2022-08-09 17:23:20. The data records of files 2 and n are all written after 2022-08-09 17:23:20. Therefore, there is no need to read these data files. Only the data records in file 1 are read for data elimination.

[0169] In step 904A, based on the offset address of the current data record, the hash value of the key, and the file number, the corresponding entry node in the hash table is obtained.

[0170] For example, see Figure 5 When it is determined that a data record is expired and needs to be deleted, the offset address, key hash value and file number of the data record can be obtained. The offset address, key hash value and file number of the data record are used to update the hash index information, that is, after determining the hash index information, the corresponding entry node in the hash table can be queried.

[0171] After step 904A, execute step 905A or step 906A.

[0172] In step 905A, when the entry node of the data record in the hash table exists, the entry node is deleted.

[0173] For example, the existence of an entry node in the hash table indicates that the data record is valid and can be deleted. When deleting data, the corresponding entry node in the hash index is directly deleted. When the written file reaches the specified size, such as the current file size limit of 256MB, it is converted to a read-only file and a new file is created as the new write file.

[0174] In step 906A, when the entry node does not exist, the data is recorded as invalid data.

[0175] For example, for data files that have implemented data elimination, a large number of data record entry nodes have been deleted from the hash index and become invalid records. Some data files even have no valid records. For files without valid records, they are directly deleted.

[0176] In step 907A, when all data records in the data file have not been read, the next data record is obtained, and the above steps 902A to 905A or 906A are repeated until all data records in the current data file are read and all expired data are deleted.

[0177] In step 908A, after the data file scan is completed, data cleanup is performed.

[0178] For example, after all data records in a data file have been scanned, the data file is sorted. Data files with no valid records are directly deleted; data files with some valid records are rewritten to independent new files. This process does not change the write time of the data records. The time statistics of the new file are still within the time statistics of the old file. After all valid records in the old file are sorted into the new file, they will be deleted.

[0179] The data processing method provided in the embodiment of the present application is characterized as a judgment process as follows. Figure 9B , Figure 9B This is the sixth flow chart of the data processing method provided in the embodiment of the present application.

[0180] In step 901B, the parameters of the data elimination task of the specified table are obtained.

[0181] In step 902B, the elimination task is started.

[0182] For example, after setting data elimination for a specified table, a data elimination task is started immediately.

[0183] In step 903B, the data file to be scanned is obtained.

[0184] For example, after the elimination task is started, a data file to be scanned is obtained from a designated table. The data file to be scanned stores data records for determining whether to be eliminated due to expiration in subsequent steps.

[0185] In step 904B, time statistics of the current data file are obtained.

[0186] For example, statistics are collected on the distribution of the earliest data writing time and the latest data writing time of the data records in the current data file.

[0187] In step 905B, it is determined whether the sum of the earliest write time and the effective time is greater than the current time.

[0188] After step 905B, execute step 906B or step 907B.

[0189] For example, if the sum of the earliest write time and the valid time is less than the current time, step 906B is executed to complete the scanning of the current file and delete the expired data.

[0190] For example, if the sum of the earliest write time and the effective time is greater than the current time, step 907B is executed to determine whether all data files in the table have been scanned.

[0191] After step 907B, execute step 908B or step 909B.

[0192] For example, if all the data in the table has not been scanned, it means that there are still data files in the table that have not yet been judged to eliminate expired data. Step 908B is executed to obtain the next file to be scanned. After obtaining the next data file to be scanned, the process returns to step 904B.

[0193] For example, if all data files in the table are scanned, it means that all data files in the table have been subjected to the elimination judgment of expired data, and step 909B is executed to determine that this round of scanning is completed.

[0194] After step 909B, step 910B is executed to calculate the start time of the next round of scanning and wait for the next round of scanning to be executed.

[0195] For example, after the current round of scanning is completed, the start time of the next round of scanning is calculated, and the waiting state is maintained before the next round of scanning starts.

[0196] In step 911B, it is determined whether the next scanning time has arrived.

[0197] After executing step 911B, execute step 910B or 903B.

[0198] For example, if the start time of the next round of scanning has not been reached, the waiting operation in step 910B is executed. If the start time of the next round of scanning has been reached, the process returns to step 903B to start the next round of data file scanning.

[0199] Figure 9B The principle of the processing process can be seen in the above Figure 4 The principle is not described here.

[0200] In some embodiments, the data processing method provided in the embodiments of the present application can be applied in a KV database (such as a distributed NoSQL database). After the user sets the expiration elimination function for the specified table, data elimination will be implemented for the table in the background.

[0201] The data processing method provided in the embodiment of the present application has the following beneficial effects:

[0202] Since data is written sequentially over time, the statistical time in the data files can effectively exclude files that do not require data elimination. This eliminates the need to establish an additional expiration date index and accurately excludes data files with no expired data records. Only files with expired data records are read and eliminated, avoiding repeated reading of invalid data and improving data elimination performance. Sorting invalid data records can avoid reading a large number of invalid data records in old files when performing data elimination again, which would affect data elimination performance.

[0203] The following continues to describe the exemplary structure of the data processing device 455 provided in the embodiment of the present application implemented as a software module. In some embodiments, such as Figure 2 As shown, the software modules stored in the data processing device 455 of the memory 450 may include: a data record acquisition module 4551, used to traverse the data records of each data file in the database, wherein the data records include: data write time; a record determination module 4552, used to perform the following processing on each of the traversed data records: obtain the sum of the data write time and the effective duration; in response to the sum being earlier than the current moment, determine the target data record corresponding to the sum, wherein the target data record is the data record to which the data write time corresponding to the sum belongs; a data deletion module 4553, used to query the entry node of the data file based on the target data record; and delete the target data record and the data corresponding to the target data record based on the entry node.

[0204] In some embodiments, the data write time includes: a write start time and a write end time. The data record acquisition module 4551 is further configured to, before traversing the data records of each data file in the database, acquire the data to be stored; write the data to be stored into the data file; in response to the write process of writing the data to be stored into the data file, record the write start time and the write end time of the write process; and store the write start time and the write end time as the data write time in the data record of the data file.

[0205] In some embodiments, the data record acquisition module 4551 is also used to obtain a first timestamp in response to the start of the write processing of writing the data to be stored into the data file, and use the moment corresponding to the first timestamp as the write start time; in response to the end of the write processing of writing the data to be stored into the data file, obtain a second timestamp, and use the moment corresponding to the second timestamp as the write end time, and the moment corresponding to the second timestamp is later than the moment corresponding to the first timestamp.

[0206] In some embodiments, the data record acquisition module 4551 is also used to obtain a preconfigured write time; determine the data files to be traversed in the database, wherein the data files to be traversed are data files whose write start time is before the preconfigured write time; and traverse the data records of each of the data files to be traversed.

[0207] In some embodiments, the data record acquisition module 4551 is also used to perform the following processing for each of the data to be stored: obtaining a first data file, wherein the first data file is an empty file before the data to be stored is written; writing the data to be stored into the first data file; in response to the storage space occupied by the first data file reaching a preset storage amount and there being a remaining portion of the data to be stored that has not been written, converting the first data file into a read-only file and creating a second data file, wherein the second data file is used to continue writing the remaining portion of the data to be stored.

[0208] In some embodiments, the data record acquisition module 4551 is also used to obtain the data to be stored and write the data to be stored into a data file; based on the hash code of the data file, the field information of the entry node in the hash index table of the database is updated, wherein the type of the field information includes: the number of the data file and the hash code.

[0209] In some embodiments, the field information further includes: a number of the next entry node, wherein the number of the next entry node is used to form an entry chain with each of the entry nodes having adjacent numbers.

[0210] In some embodiments, the record determination module 4552 is also used to determine the hash code and address number of the entry node included in the target data record; obtain the hash index table of the database, wherein the hash index table includes: multiple buckets and entry nodes stored in each bucket; determine the hash value corresponding to the hash code, and determine the bucket position associated with the target data record in the hash index table based on the hash value; determine the entry node corresponding to the target data record based on the bucket position and the address number of the entry node.

[0211] In some embodiments, the data deletion module 4553 is also used to determine the file number and file offset address included in the entry node; call the data file corresponding to the target data record based on the file number and file offset address, and delete the written data corresponding to the data write time of the target data record in the called data file, and delete the entry node.

[0212] In some embodiments, the data deletion module 4553 is also used to record the first moment in response to the completion of traversal of each data record in the database; in response to the time interval between the first moment and the current moment reaching a preconfigured duration, it switches to the processing of the data records of each data file in the traversed database.

[0213] In some embodiments, the data deletion module 4553 is also used to, in response to the data file after deletion processing including residual data, store the residual data in a new data file and delete the original data file corresponding to the residual data; in response to the data record corresponding to the data file after deletion processing being an invalid record, delete the data file.

[0214] An embodiment of the present application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer program or computer-executable instructions from the computer-readable storage medium and executes the computer program or computer-executable instructions, causing the electronic device to perform the data processing method described in the embodiment of the present application.

[0215] The embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions or computer programs are stored. When the computer-executable instructions or computer programs are executed by a processor, the processor will be caused to execute the data processing method provided by the embodiment of the present application, for example, Figure 3A The data processing method is shown.

[0216] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or various devices including one or any combination of the above memories.

[0217] In some embodiments, computer-executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0218] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as, for example, in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0219] As an example, executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed across multiple sites and interconnected by a communication network.

[0220] To sum up, the data records corresponding to the data in the embodiment of the present application include at least the write time. By judging the relationship between the write time, the sum of the effective time and the current time, it is determined whether the data corresponding to the data record is expired. By querying the entry node of the data file through the target data record, the data file is called, and the expired data file can be determined in a targeted manner. Compared with the solution of additionally building an index and scanning the entire table data in the related technology, it can save the computing resources required to determine and delete expired data.

[0221] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. A data processing method, characterized in that: The method comprises: Traversing the data records of each data file in the database, wherein the data records include: data writing time; The following processing is performed on each of the data records traversed: Obtaining the sum of the data writing time and the effective duration; In response to the sum being earlier than the current time, determining a target data record corresponding to the sum, wherein the target data record is a data record to which a data writing time corresponding to the sum belongs; querying an entry node of the data file based on the target data record; The target data record and data corresponding to the target data record are deleted based on the entry node.

2. The method according to claim 1, characterized in that The data writing time includes: the writing start time and the writing end time; Before traversing the data records of each data file in the database, the method further includes: Get the data to be stored; Writing the data to be stored into a data file; In response to a write process of writing the data to be stored into the data file, recording the write start time and the write end time of the write process; The writing start time and the writing end time are stored as data writing time in the data record of the data file.

3. The method according to claim 2, characterized in that The step of recording the writing start time and the writing end time of the writing process in response to writing the data to be stored into the data file comprises: In response to starting to execute a write process of writing the data to be stored into the data file, obtaining a first timestamp, and taking a time corresponding to the first timestamp as the write start time; In response to the completion of the writing process of writing the data to be stored into the data file, a second timestamp is obtained, and the time corresponding to the second timestamp is used as the writing end time, where the time corresponding to the second timestamp is later than the time corresponding to the first timestamp.

4. The method according to claim 3, characterized in that The traversal of the data records of each data file in the database includes: Get the preconfigured write time; Determining a data file to be traversed in the database, wherein the data file to be traversed is a data file whose writing start time is before the preconfigured writing time; Traverse the data records of each of the data files to be traversed.

5. The method according to claim 2, characterized in that Writing the data to be stored into a data file includes: The following processing is performed for each data to be stored: Acquire a first data file, wherein the first data file is an empty file before the data to be stored is written; Writing the data to be stored into the first data file; In response to the storage space occupied by the first data file reaching a preset storage amount and there being a remaining portion of the data to be stored that has not been written, the first data file is converted into a read-only file and a second data file is created, wherein the second data file is used to continue writing the remaining portion of the data to be stored.

6. The method according to claim 1, characterized in that The querying the entry node of the data file based on the target data record includes: Determine the hash code and the address number of the entry node included in the target data record; Obtaining a hash index table of the database, wherein the hash index table includes a plurality of buckets and entry nodes stored in each bucket; Determine a hash value corresponding to the hash code, and determine a bucket position associated with the target data record in the hash index table based on the hash value; An entry node corresponding to the target data record is determined based on the bucket position and the address number of the entry node.

7. The method according to claim 6, characterized in that Before traversing the data records of each data file in the database, the method further includes: Acquire data to be stored, and write the data to be stored into a data file; The field information of the entry node in the hash index table of the database is updated based on the hash code of the data file, wherein the type of the field information includes: the number of the data file and the hash code.

8. The method according to claim 7, characterized in that The field information further includes: a number of a next entry node, where the number of the next entry node is used to form an entry chain with each of the entry nodes having adjacent numbers.

9. The method according to any one of claims 1 to 8, characterized in that The deleting the target data record and the data corresponding to the target data record based on the entry node includes: Determine the file number and file offset address included in the entry node; Calling the data file corresponding to the target data record based on the file number and the file offset address, and deleting the written data corresponding to the data writing time of the target data record in the called data file, and Delete the entry node.

10. The method according to any one of claims 1 to 8, characterized in that After deleting the target data record and the data corresponding to the target data record based on the entry node, the method further includes: In response to completion of traversal of each of the data records in the database, recording a first moment; In response to the time interval between the first moment and the current moment reaching a preconfigured duration, the process of traversing the data records of each data file in the database begins.

11. The method according to any one of claims 1 to 8, characterized in that After deleting the target data record and the data corresponding to the target data record based on the entry node, the method further includes: In response to the data file after the deletion process including residual data, the residual data is stored in a new data file, and the original data file corresponding to the residual data is deleted; In response to the data record corresponding to the data file after the deletion process being an invalid record, the data file is deleted.

12. A data processing device, characterized in that: The device comprises: The data record acquisition module is used to traverse the data records of each data file in the database, wherein the data records include: data writing time; a record determination module, configured to perform the following processing on each of the traversed data records: obtaining a sum of the data write time and the effective duration; in response to the sum being earlier than the current time, determining a target data record corresponding to the sum, wherein the target data record is the data record to which the data write time corresponding to the sum belongs; The data deletion module is configured to query the entry node of the data file based on the target data record; and delete the target data record and the data corresponding to the target data record based on the entry node.

13. An electronic device, characterized in that: The electronic device comprises: a memory for storing computer-executable instructions; The processor is configured to implement the data processing method according to any one of claims 1 to 11 when executing the computer-executable instructions or computer programs stored in the memory.

14. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the data processing method according to any one of claims 1 to 11 is implemented.

15. A computer program product comprising computer executable instructions or a computer program, characterized in that When the computer executable instructions or computer program are executed by a processor, the data processing method according to any one of claims 1 to 11 is implemented.

Images