Data processing method, apparatus and computer readable storage medium
By merging disk and memory data files, the problem of data loss during abnormal database restarts was solved, achieving persistence and efficient recovery of fully cached data.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHEJIANG CHINT IOT TECH CO LTD
- Filing Date
- 2023-09-13
- Publication Date
- 2026-05-12
AI Technical Summary
Existing data processing methods cannot support the persistence of cached data when the database restarts abnormally, resulting in data loss and affecting data processing efficiency.
By reading the target table data file stored on disk and the memory change log file, the table data and change data are merged and processed to build the target full cache data, and operation information is recorded to ensure data persistence.
When the database restarts abnormally, it can recover all cached data in memory, ensuring data persistence and improving data processing efficiency.
Smart Images
Figure CN117112570B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, and more specifically to a data processing method, apparatus, and computer-readable storage medium. Background Technology
[0002] In software systems using relational database management systems (MySQL), there are many tables with relatively small amounts of data. These tables can be fully cached. For example, a memory storage engine (Memory Storage Engine) can be used to fully cache the table data.
[0003] During the research and practice of existing technologies, it was found that the existing data processing methods that cache table data in full cannot support the persistence of cached data. When the database is restarted due to an anomaly, data loss is very likely to occur, resulting in low data processing efficiency. Summary of the Invention
[0004] This application provides a data processing method, apparatus, and computer-readable storage medium that can restore the latest data fully cached in memory when the database is restarted due to an anomaly, ensuring the persistence of the data fully cached in memory and thus improving data processing efficiency.
[0005] This application provides a data processing method, including:
[0006] Obtain the startup command for the target database and start the target database;
[0007] Read the target table data file and memory change log file corresponding to the target database stored on the disk. The memory change log file is used to record operation information of the full cache data of the target table data file in memory. The operation information includes the changed data operated on in the full cache data.
[0008] The table data in the target table data file and the change data in the memory change log file are merged to obtain the target table data;
[0009] Based on the target table data, construct the target full cache data corresponding to the target table data file in memory.
[0010] Accordingly, embodiments of this application provide a data processing apparatus, including:
[0011] A database startup unit is used to obtain a startup instruction for a target database and start the target database;
[0012] The file reading unit is used to read the target table data file and memory change log file corresponding to the target database stored on the disk. The memory change log file is used to record the operation information of the full cache data of the target table data file in memory. The operation information includes the changed data operated on in the full cache data.
[0013] The data merging unit is used to merge the table data in the target table data file and the change data in the memory change log file to obtain the target table data.
[0014] A data caching unit is used to construct target full cache data corresponding to the target table data file in memory based on the target table data.
[0015] In one embodiment, each line of data in the target table data file and the memory change log file has a global sequence number, and the data merging unit is used to:
[0016] Obtain the global sequence number of the change data for each line of change data in the memory change log file;
[0017] Based on the global sequence number of the changed data, the table data and the changed data are compared, and the target changed data that has not been written to the target table data file is determined from the changed data according to the comparison result.
[0018] The target change data and the table data are merged to obtain the target table data.
[0019] In one embodiment, the data processing apparatus further includes:
[0020] The target global sequence number recording unit is used to record the largest target global sequence number among the global sequence numbers of the changed data;
[0021] The data processing device further includes:
[0022] The target global sequence number acquisition unit is used to acquire the maximum target global sequence number when a data addition operation for the target full cache data is received;
[0023] The sequence number incrementing unit is used to increment the target global sequence number to obtain the first target global sequence number;
[0024] A data writing unit is used to write the first target data corresponding to the data addition operation into the target full cache data;
[0025] The first data update unit is used to update the memory change log file based on the first target global sequence number and the first target data.
[0026] In one embodiment, the first data update unit is configured to:
[0027] Obtain the amount of stored data in the memory change log file;
[0028] When the amount of stored data is not greater than a preset data amount threshold, the first target global sequence number and the first target data are written to the memory change log file;
[0029] When the amount of stored data exceeds the preset data volume threshold, obtain the maximum file number of the memory change log file;
[0030] Create a new memory change log file, and assign the file number of the new memory change log file to the file number obtained by incrementing the maximum file number;
[0031] Write the first target global sequence number and the first target data into the new memory change log file.
[0032] In one embodiment, the data processing apparatus further includes:
[0033] The data location unit is modified to locate the second target data corresponding to the data modification operation in the target full cache data in response to the data modification operation for the target full cache data;
[0034] The data modification unit is used to lock the row containing the second target data and modify the second target data based on the data modification operation;
[0035] The second data update unit is used to update the memory change log file based on the global sequence number of the second target data and the modified second target data.
[0036] In one embodiment, the data processing apparatus further includes a data deletion unit, configured to:
[0037] In response to a data deletion operation on the target full cache data, the third target data corresponding to the data deletion operation is located in the target full cache data;
[0038] Lock the row containing the third target data and modify the deletion marker corresponding to the third target data;
[0039] The memory change log file is updated based on the modified deletion marker and the global sequence number corresponding to the third target data.
[0040] In one embodiment, the data processing apparatus further includes a data synchronization unit, configured to:
[0041] Determine the number of memory change log files;
[0042] When the number of files exceeds a preset file number threshold, the target memory change log file to be processed is determined based on the file number of the memory change log file. The file number of the target memory change log file is not the largest file number among the file numbers of the memory change log files.
[0043] Based on the global sequence number in the target memory change log file, determine whether the changed data in the target memory change log file has been synchronized to the target table data file on the disk;
[0044] If the table data in the target memory change log file is not synchronized to the target table data file on the disk, synchronize the table data in the target memory change log file to the target table data file on the disk.
[0045] If the target table data file is synchronized to the disk, delete the target memory change log file.
[0046] Furthermore, embodiments of this application also provide a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute steps in any of the data processing methods provided in embodiments of this application.
[0047] Furthermore, this application also provides a computer device, including a processor and a memory, wherein the memory stores an application program, and the processor is used to run the application program in the memory to implement the data processing method provided in this application.
[0048] This application also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps in the data processing method provided in this application.
[0049] This embodiment of the application starts the target database by obtaining a startup command for the target database; it reads the target table data file and memory change log file corresponding to the target database stored on disk. The memory change log file records operation information on the full cached data of the target table data file in memory, including the changed data operated on in the full cache data; it merges the table data in the target table data file and the changed data in the memory change log file to obtain the target table data; and it constructs the target full cache data corresponding to the target table data file in memory based on the target table data. Thus, by recording the operation information on the full cached data of the target table data file in memory in the memory change log file, when the database restarts due to an anomaly, the latest data of the target table data file fully cached in memory at the time of the anomaly can be determined based on the target table data file stored on disk and the memory change log file. This enables the recovery of the table data fully cached in memory when the database anomaly occurs, ensuring the persistence of the fully cached data in memory and improving data processing efficiency. Attached Figure Description
[0050] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0051] Figure 1 This is a schematic diagram illustrating an implementation scenario of a data processing method provided in an embodiment of this application;
[0052] Figure 2 This is a flowchart illustrating a data processing method provided in an embodiment of this application;
[0053] Figure 3 This is a schematic diagram of the database startup process of a data processing method provided in an embodiment of this application;
[0054] Figure 4 This is a schematic diagram of the data addition process of a data processing method provided in an embodiment of this application;
[0055] Figure 5 This is a schematic diagram of the data modification process of a data processing method provided in an embodiment of this application;
[0056] Figure 6 This is a schematic diagram of the data deletion process of a data processing method provided in an embodiment of this application;
[0057] Figure 7This is a schematic diagram of a data query process for a data processing method provided in an embodiment of this application;
[0058] Figure 8 This is a schematic diagram of the data synchronization process of a data processing method provided in an embodiment of this application;
[0059] Figure 9 This is a schematic diagram of the structure of the data processing apparatus provided in the embodiments of this application;
[0060] Figure 10 This is a schematic diagram of the structure of the computer device provided in the embodiments of this application. Detailed Implementation
[0061] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0062] This application provides a data processing method, apparatus, and computer-readable storage medium. The data processing apparatus can be integrated into a computer device, which may be a server or a terminal, etc.
[0063] The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery network (CDN) acceleration services, and big data and artificial intelligence platforms. The terminal can include, but is not limited to, mobile phones, computers, smart voice interaction devices, smart home appliances, vehicle terminals, and aircraft. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited herein.
[0064] Please see Figure 1 Taking the integration of data processing devices into computer equipment as an example, Figure 1This is a schematic diagram illustrating an implementation scenario of the data processing method provided in this application. The computer device can be a server or a terminal. The computer device can obtain a startup command for the target database and start the target database; read the target table data file and memory change log file corresponding to the target database stored on the disk. The memory change log file records operation information for the full cache data of the target table data file in memory, including the changed data operated on in the full cache data; merge the table data in the target table data file and the changed data in the memory change log file to obtain the target table data; and construct the target full cache data corresponding to the target table data file in memory based on the target table data.
[0065] It should be noted that, Figure 1 The illustrated implementation environment scenario of the data processing method is merely an example. The implementation environment scenario of the data processing method described in this application embodiment is for the purpose of more clearly illustrating the technical solution of this application embodiment and does not constitute a limitation on the technical solution provided in this application embodiment. Those skilled in the art will understand that with the evolution of data processing and the emergence of new business scenarios, the technical solution provided in this application is also applicable to similar technical problems.
[0066] The solutions provided in this application are specifically illustrated through the following embodiments. It should be noted that the order of description of the following embodiments is not intended to limit the preferred order of the embodiments.
[0067] This embodiment will be described from the perspective of a data processing device, which can be integrated into a computer device, which can be a server, and this application does not limit it.
[0068] Please see Figure 2 , Figure 2 This is a flowchart illustrating the data processing method provided in an embodiment of this application. The data processing method includes:
[0069] In step 101, a startup command for the target database is obtained, and the target database is started.
[0070] The target database can be a database using a relational database management system (MySQL), or optionally, a relational database using the memory storage engine in MySQL for data storage. The startup command can be an instruction to start the target database. This startup command can be triggered manually or automatically. For example, it can be actively generated when the target database experiences a failure, power outage, or other abnormal situation, or it can be triggered by the user performing a startup operation.
[0071] In step 102, the target table data file and memory change log file corresponding to the target database stored on the disk are read.
[0072] The disk can be a local disk, such as the disk of the server where the target database resides. The target table data file can be a table data file that uses full caching. This table data file can be data stored in a relational database management system, specifically a software system that stores data from tables that can be fully cached. The memory change log file records operation information on the fully cached data of the target table data file in memory. This fully cached data can be the table data of the target table data file fully cached in memory. This table data can be data stored in table format. The operation information can include the modified data operated on in the fully cached data. This modified data can be the data operated on in the operations on the fully cached data, and can also include the operation type of the modified data, such as add, delete, modify, query, etc. It can also include the global sequence number corresponding to the modified data. This global sequence number can be information that identifies each row of data in the fully cached table data file. Each row of data in the target table data file corresponds to a global sequence number, which uniquely identifies a row of data.
[0073] Optionally, a new global sequence number can be generated monotonically increasing based on the writing order of each row of data. For example, assuming the largest global sequence number in the target table data file is 0035467, when a new data row is written to the target table data file, the global sequence number corresponding to the newly written data row can be 0035467+1=0035468.
[0074] Optionally, the memory change log file can be a Write-Ahead Log (WAL) file. The WAL file can be a write-optimized format, written sequentially according to the operation order. After submitting Data Definition Language (DDL) and Data Manipulation Language (DML) operations on the fully cached data of the target table in memory, the changed data is written to the WAL file. The WAL file is segmented according to the size of the written data, with each segment corresponding to one WAL file. For example, it can be divided into file numbers such as WAL_1, WAL_2, ..., WAL_N, with each file number corresponding to one WAL file. All tables using the full-caching storage engine can share a single set of WAL files.
[0075] In one embodiment, the file format of the memory change log file can be as shown in Table 1 below. The memory change log file can record operation information for each operation on the full cached data of the target table data file in memory. This operation information may include the changed data being operated on, the global sequence number corresponding to the changed data, and the operation type, etc. The data portion of the memory change log file is used to store the changed data corresponding to the operation.
[0076]
[0077] Table 1
[0078] In one embodiment, the file format of the target table data file can be as shown in Table 2 below. The target table data file may include each row of data in the table, a global sequence number corresponding to each row, and a deletion flag. This data portion can be used to store each row of data in the table, and the deletion flag can be used to indicate whether the data in the current row has been deleted. The data portion can be stored using variable-length fields. Variable-length fields allow for the storage of data of different lengths; for example, a maximum variable-length field can be used for fixed-length storage.
[0079]
[0080] Table 2
[0081] In step 103, the table data in the target table data file and the change data in the memory change log file are merged to obtain the target table data.
[0082] The target table data can include table data from the target table data file and changed data from the memory change log file. In a relational database based on a relational database management system, the target table data file needs to be fully cached in memory using a memory storage engine when the database starts. During subsequent tasks, CRUD operations (Create, Read, Update, Delete) are performed on the fully cached data in memory. Then, during idle time or when a data merge command is received, the changed data in memory is synchronized to the target table data file on disk. This achieves strong consistency between the database table and the fully cached data in memory, thus ensuring normal business operations. However, when the target database experiences power outages, malfunctions, or other abnormal situations during operation, restarting the target database may result in the loss of all cached data in memory, leading to data inconsistency. To address this, the data processing method provided in this application records the operation information of the target table data file's fully cached data in memory in a memory change log file. This allows the system to determine the latest data of the target table data file fully cached in memory when the database restarts due to an anomaly, based on the target table data file stored on disk and the memory change log file. This enables the recovery of the fully cached table data in memory when the database fails, ensuring the persistence of the fully cached data in memory and thus improving data processing efficiency.
[0083] There are several ways to merge the table data in the target table data file and the change data in the memory change log file to obtain the target table data. For example, each line of data in the target table data file and the memory change log file can have a global sequence number. This allows us to obtain the global sequence number of each line of change data in the memory change log file. By comparing the table data and the change data based on the global sequence number, we can identify the target change data that has not been written to the target table data file based on the comparison results. Finally, we merge the target change data and the table data to obtain the target table data.
[0084] The global sequence number of the change data can be the global sequence number of each line of change data in the memory change log file, and the target change data can be the data in the target table data file that has not been written to disk from the change data in the memory change log file.
[0085] There are several ways to determine the target changed data that has not been written to the target table data file based on the comparison of the global sequence number of the changed data and the changed data. For example, one can search for data rows in the target table data file with the same global sequence number as the changed data based on the global sequence number of the changed data. Then, compare the table data stored in the data row with the changed data corresponding to the global sequence number of the changed data. If the comparison result is that the data is the same, it indicates that the changed data has been written to the target table data file. If the comparison result is that the data is different, it indicates that the changed data has not been written to the target table data file. Thus, the target changed data that has not been written to the target table data file can be determined based on the comparison result.
[0086] After identifying the target changed data that has not been written to the target table data file based on the comparison results, the target changed data and table data can be merged to obtain the target table data. There are several ways to merge the target changed data and table data. For example, the corresponding data row can be found in the target table data file based on the global sequence number of the target changed data. Then, operations can be performed on the table data stored in that data row according to the operation type corresponding to the target changed data. For instance, if the operation type is modification, the data stored in that data row can be modified based on the changed data; if the operation type is deletion, the data stored in that data row can be deleted based on the changed data, and so on.
[0087] In one embodiment, please refer to Figure 3 , Figure 3 This is a schematic diagram of the database startup process of a data processing method provided in this application embodiment. When the target database starts, the target table data file and memory change log file corresponding to the target database stored on the disk can be read. In this way, the global sequence number of each line of changed data in the memory change log file can be obtained. Then, the table data in the target table data file and the changed data in the memory change log file are merged according to the global sequence number of the changed data to obtain the target table data. The target global sequence number with the largest sequence number among the global sequence numbers of the changed data can also be recorded. The target global sequence number can be the global sequence number with the largest sequence number among the global sequence numbers of the changed data.
[0088] In step 104, target full cache data corresponding to the target table data file is constructed in memory based on the target table data.
[0089] The target full cache data can be full cache data that includes table data in the target table data file and change data in the memory change log file.
[0090] Optionally, the target full-cache data can be stored in memory blocks (BLOCKs). Fixed-size blocks can be allocated in memory, and a single block can store multiple rows of data. Blocks can be linked together using a linked list to facilitate full table scans, thereby improving data retrieval speed. In this way, using blocks to cache the target table data allows the fully cached data to be stored sequentially in memory as much as possible, reducing the need for reallocating space when inserting new data rows, and thus reducing space allocation costs.
[0091] Optionally, the row data format of the target full cache data can be as shown in Table 3 below. The row lock flag can be information indicating whether the data row is locked.
[0092]
[0093]
[0094] Table 3
[0095] Optionally, there are several ways to construct the target full cache data corresponding to the target table data file in memory based on the target table data. For example, please refer to [link / reference]. Figure 3 It can cache all the target table data in memory, thereby building the fully cached data in memory, and read the index information file corresponding to the target database stored on disk, thereby building index data in memory based on the index information file.
[0096] The index information file can be a file containing index information for the fully cached data. The file format of the index information file can be as shown in Table 4 below. The index information file can include the deletion flag for each index, the table name, the index name, the field sequence number, and the index sorting rules. The table name and index name can be stored using a fixed-length storage method.
[0097] Delete mark Table name Index name Field serial number Sorting rules … …
[0098] Table 4
[0099] The index data can be indexed. For tables with frequent range queries, a tree data structure (B+ tree) can be used to implement the index. For tables with single-point query types and frequent updates, a hash table can be used. The value (VALUE) of the index represents the memory address of the data row.
[0100] In one embodiment, before merging the table data in the target table data file and the change data in the memory change log file to obtain the target table data, the largest target global sequence number among the global sequence numbers of the change data can be recorded. After constructing the target full cache data corresponding to the target table data file in memory based on the target table data, when a data addition operation for the target full cache data is received, the recorded largest target global sequence number can be obtained, and the target global sequence number can be incremented to obtain the first target global sequence number. The first target data corresponding to the data addition operation is written into the target full cache data, and the memory change log file is updated based on the first target global sequence number and the first target data.
[0101] The data addition operation can be performed on data fully cached in memory. After the target database starts, it can receive and parse Structured Query Language (SQL) statements, and then perform operations such as adding, deleting, updating, and querying on the fully cached data in memory based on the parsed commands. Optionally, the data addition operation can be an insert statement in SQL, used to add new rows of data to the table. The first target global sequence number can be a global sequence number obtained by incrementing the target global sequence number. For example, please refer to... Figure 4 , Figure 4 This is a schematic diagram of a data addition process provided in an embodiment of this application. The target global sequence number can be incremented by one to obtain a first target global sequence number. Since a new data row needs to be added to the target full cache data stored in memory, the target global sequence number with the largest number can be incremented by one to obtain the global sequence number corresponding to the newly added data row, so that the newly added data row can be marked. This first target data can be the data operated on by the data addition operation, that is, the data that needs to be added to the target full cache data.
[0102] There are several ways to write the first target data corresponding to the data addition operation into the target full cache data. For example, a new data row can be added to the target full cache data according to the first target global sequence number, so that the first target data and the first target global sequence number can be written into the data row of the target full cache data.
[0103] The target full cache data can be stored in memory blocks, for example, please refer to [link / reference]. Figure 4Before adding a new data row to the target full cache data based on the first target global sequence number and writing the first target data and the first target global sequence number to that data row, the last memory block in memory can be retrieved to determine if there is any free storage space. If there is free storage space, a new data row can be added to the target full cache data based on the first target global sequence number, and the first target data and the first target global sequence number can be written to that data row. If there is no free storage space, a new memory block can be created in memory, a new data row can be added to the new memory block based on the first target global sequence number, and the first target data and the first target global sequence number can be written to that data row. Additionally, corresponding index data can be written to memory based on the first target data.
[0104] After writing the first target data corresponding to the data addition operation to the target full cache data, the memory change log file can be updated based on the first target global sequence number and the first target data. There are several ways to update the memory change log file based on the first target global sequence number and the first target data; for example, please refer to [link / reference]. Figure 4 It can obtain the amount of stored data in the memory change log file. When the amount of stored data is not greater than the preset data volume threshold, it writes the first target global sequence number and the first target data into the memory change log file. When the amount of stored data is greater than the preset data volume threshold, it obtains the maximum file number of the memory change log file, creates a new memory change log file, assigns the file number of the new memory change log file to the file number obtained by incrementing the maximum file number, and writes the first target global sequence number and the first target data into the new memory change log file.
[0105] The storage data volume refers to the amount of data stored in the memory change log file, which can be used to measure the file size or volume of the memory change log file. The preset data volume threshold can be a pre-defined critical value for the data volume. When the storage data volume of the memory change log file exceeds this critical value, it indicates that the storage space of the memory change log file is full. This preset data volume threshold can be set according to actual needs, for example, it can be the storage data volume of a file of 60 MB or 70 MB, etc., and is not limited here. The maximum file number can be the file number with the largest file number among the file numbers of the memory change log files. The new memory change log file can be a newly created memory change log file, and the maximum file number can be incremented by one to become the file number of the new memory change log file.
[0106] In one embodiment, in response to a data modification operation on the target full cache data, the system can locate the second target data corresponding to the data modification operation in the target full cache data, lock the row containing the second target data, modify the second target data based on the data modification operation, and update the memory change log file based on the global sequence number of the second target data and the modified second target data.
[0107] The data modification operation can be an operation that modifies the target full cache data, and the second target data can be the data to be modified in the target full cache data by the data modification operation.
[0108] Optional, please refer to Figure 5 , Figure 5 This is a schematic diagram of the data modification process of a data processing method provided in this application embodiment. In response to a data modification operation on the target full-cache data, it can determine whether an index needs to be used to query the second target data corresponding to the data modification operation. If an index query is required, it can be performed within the index data. If the index is not needed, the memory block and its data rows can be traversed to locate the second target data. A row lock can then be applied to the data row containing the second target data to prevent simultaneous modifications to the second target data. This enables the relational database's memory storage engine to support row locking, achieving parallel processing of the target full-cache data and further improving data processing efficiency. Thus, the second target data can be modified according to the data modification operation. After modification, it can be determined whether the index field value in the index data needs to be modified. If modification is required, the index data can be modified. Then, the second target data, its corresponding global sequence number, and the modification operation type can be written to the memory change log file.
[0109] In one embodiment, in response to a data deletion operation on the target full cache data, the third target data corresponding to the data deletion operation in the target full cache data can be located, the row containing the third target data can be locked, the deletion marker corresponding to the third target data can be modified, and the memory change log file can be updated based on the modified deletion marker and the global sequence number corresponding to the third target data.
[0110] The data deletion operation can be an operation to delete data from a data row in the target full cache data. The third target data can be the data to be deleted in the data deletion operation. The deletion flag can be information to mark whether the current data has been deleted. For example, it can include types such as deleted and not deleted. Alternatively, if the data to be deleted carries a deletion flag, it indicates that the current data has been deleted. If the data does not carry a deletion flag, it indicates that the current data has not been deleted. The specific settings can be set according to the actual situation.
[0111] Optional, please refer to Figure 6 , Figure 6 This is a schematic diagram of a data deletion process provided in an embodiment of this application. In response to a data deletion operation targeting full cached data, it determines whether an index is needed to query the third target data corresponding to the data deletion operation. If an index query is needed, the query can be performed within the index data. If an index is not needed, the memory block and its data rows can be traversed to locate the third target data. A row lock is added to the data row containing the third target data to prevent multiple operations from simultaneously accessing the third target data. The deletion flag corresponding to the third target data is modified to indicate that the third target data has been deleted. The third target data, its corresponding deletion flag, and the deletion operation type are then written to a memory change log file. Specifically, it can be determined whether the stored data volume of the memory change log file with the largest file number exceeds a preset data volume threshold. If it does, indicating the memory change log file is full, the file number of the memory change log file is incremented by one, and a new memory change log file is created based on the incremented file number for writing the changed data.
[0112] In one embodiment, when a transaction is received that is to be executed for the target full cache data, only the corresponding target full cache data in memory can be modified before the transaction commit operation. The operation information corresponding to the transaction execution is written to the memory change log file only after the transaction calls COMMIT. If the transaction calls ROLLBACK, the data modified by the transaction execution in memory can be restored to the data before the modification.
[0113] In one embodiment, please refer to Figure 7 , Figure 7This is a schematic diagram of a data query process for a data processing method provided in an embodiment of this application. In response to a data query operation targeting full cached data, it determines whether an index is needed to locate the fourth target data corresponding to the data query operation. If an index is required, the query can be performed within the index data. If an index is not needed, the memory block and its data rows can be traversed to locate the fourth target data. The fourth target data can be the data to be retrieved by the data query operation.
[0114] In one embodiment, the target full cache data cached in memory can be synchronized with the target table data file stored on disk to ensure data consistency. For example, the synchronization of the target full cache data cached in memory to the target table data file stored on disk can be achieved through the change data recorded in the memory change log file. For details, please refer to [link / reference]. Figure 8 , Figure 8 This is a schematic diagram of the data synchronization process of a data processing method provided in this application embodiment. It can determine the number of memory change log files. When the number of files is greater than a preset file number threshold, the target memory change log file to be processed is determined according to the file number of the memory change log file. According to the global sequence number in the target memory change log file, it is determined whether the changed data in the target memory change log file has been synchronized to the target table data file on the disk. If it has not been synchronized to the target table data file on the disk, the table data in the target memory change log file is synchronized to the target table data file on the disk. If it has been synchronized to the target table data file on the disk, the target memory change log file is deleted.
[0115] The number of files can be the number of memory change log files currently stored on the disk. The preset file number threshold can be a pre-defined critical value for the number of files. When the number of memory change log files exceeds this critical value, a synchronization operation can be performed between the target full cache data cached in memory and the target table data file stored on the disk. When the number of memory change log files is not greater than this critical value, the synchronization operation is not performed. The preset file number threshold can be 1. When the number of memory change log files is 1, it indicates that the current memory change log file may not be full, and recording can continue; therefore, data synchronization is not required. When the number of memory change log files is 0, it indicates that no memory change log file currently exists, and a new memory change log file can be created to continue recording operation information; therefore, data synchronization is also not required. Optionally, the preset file number threshold can also be other values, such as 2 or 3, which can be set according to the actual situation. The target memory change log file can be the memory change log file of the target table data file whose recorded change data needs to be synchronized to the disk. The file number of the target memory change log file is not the largest file number among the memory change log files, because the memory change log file with the largest file number may not be full and can continue to record operation information. Therefore, data synchronization processing is not required for the memory change log file with the largest file number.
[0116] There are several ways to determine whether changed data in the target memory change log file has been synchronized to the target table data file on disk, based on the global sequence number in the target memory change log file. For example, one can find the data row in the target table data file with the same global sequence number in the target memory change log file, and compare the changed data corresponding to that global sequence number in the target memory change log file with the table data in the target table data file for the same global sequence number. If the comparison results are the same, it indicates that the changed data has been synchronized to the target table data file on disk, and the target memory change log file can be deleted. If the comparison results are different, it indicates that the changed data has not been synchronized to the target table data file on disk, and the changed data can be synchronized to the target table data file. After successful synchronization, the target memory change log file can be deleted. Optionally, when the program is idle, the target full cache data and index data in memory can be reorganized according to data changes, and data and indexes marked for deletion can be cleaned up. In this way, data synchronization between the target full cache data cached in memory and the target table data file stored on disk is achieved, ensuring data consistency.
[0117] As described above, this embodiment of the application starts the target database by obtaining a startup command for the target database; reads the target table data file and memory change log file corresponding to the target database stored on the disk. The memory change log file records operation information on the full cached data of the target table data file in memory, including the changed data operated on in the full cached data; merges the table data in the target table data file and the changed data in the memory change log file to obtain the target table data; and constructs the target full cached data corresponding to the target table data file in memory based on the target table data. Thus, by recording the operation information on the full cached data of the target table data file in memory in the memory change log file, when the database restarts due to an anomaly, the latest data of the target table data file fully cached in memory at the time of the database anomaly can be determined based on the target table data file stored on the disk and the memory change log file. This enables the recovery of the table data fully cached in memory when the database anomaly occurs, ensuring the persistence of the fully cached data in memory and improving data processing efficiency.
[0118] To better implement the above methods, embodiments of the present invention also provide a data processing apparatus that can be integrated into a computer device, which can be a server.
[0119] For example, such as Figure 9The diagram shown is a structural schematic of a data processing device provided in an embodiment of this application. The data processing device may include a database startup unit 201, a file reading unit 202, a data merging unit 203, and a data caching unit 204, as follows:
[0120] The database startup unit 201 is used to obtain a startup instruction for the target database and start the target database;
[0121] The file reading unit 202 is used to read the target table data file and the memory change log file corresponding to the target database stored on the disk. The memory change log file is used to record the operation information of the full cache data of the target table data file in memory. The operation information includes the changed data operated on in the full cache data.
[0122] The data merging unit 203 is used to merge the table data in the target table data file and the change data in the memory change log file to obtain the target table data.
[0123] Data caching unit 204 is used to construct the target full cache data corresponding to the target table data file in memory based on the target table data.
[0124] In one embodiment, each line of data in the target table data file and the memory change log file has a global sequence number. The data merging unit 203 is used for:
[0125] Obtain the global sequence number of the change data for each line of change data in the memory change log file;
[0126] Based on the global sequence number of the changed data, compare the table data and the changed data, and determine the target changed data that has not been written to the target table data file based on the comparison results.
[0127] The target change data and table data are merged to obtain the target table data.
[0128] In one embodiment, the data processing apparatus further includes:
[0129] The target global sequence number recording unit is used to record the largest target global sequence number among the global sequence numbers of the changed data;
[0130] The data processing apparatus further includes:
[0131] The target global sequence number acquisition unit is used to acquire the maximum target global sequence number of the record when a data addition operation for the target full cache data is received.
[0132] The sequence number incrementing unit is used to increment the target global sequence number to obtain the first target global sequence number;
[0133] The data writing unit is used to write the first target data corresponding to the data addition operation into the target full cache data;
[0134] The first data update unit is used to update the memory change log file based on the first target global sequence number and the first target data.
[0135] In one embodiment, the first data update unit is configured to:
[0136] Get the amount of data stored in the memory change log file;
[0137] When the amount of stored data is no greater than the preset data amount threshold, the first target global sequence number and the first target data are written to the memory change log file;
[0138] When the amount of stored data exceeds a preset data volume threshold, obtain the maximum file number of the memory change log file;
[0139] Create a new memory change log file and assign the file number of the new memory change log file to the file number obtained by incrementing the maximum file number;
[0140] Write the first target's global sequence number and the first target's data to a new memory change log file.
[0141] In one embodiment, the data processing apparatus further includes:
[0142] The data location unit is modified to locate the second target data corresponding to the data modification operation in the target full cache data in response to a data modification operation on the target full cache data.
[0143] The data modification unit is used to lock the row containing the second target data and modify the second target data based on the data modification operation;
[0144] The second data update unit is used to update the memory change log file based on the global sequence number of the second target data and the modified second target data.
[0145] In one embodiment, the data processing apparatus further includes a data deletion unit for:
[0146] In response to a data deletion operation targeting the full cache data, locate the third target data corresponding to the data deletion operation within the target full cache data;
[0147] Lock the row containing the third target data and modify the deletion flag corresponding to the third target data;
[0148] Update the memory change log file based on the modified deletion marker and the global sequence number corresponding to the third target data.
[0149] In one embodiment, the data processing apparatus further includes a data synchronization unit, configured to:
[0150] Determine the number of memory change log files;
[0151] When the number of files exceeds the preset file number threshold, the target memory change log file to be processed is determined based on the file number of the memory change log file. The file number of the target memory change log file is not the largest file number among the file numbers of the memory change log files.
[0152] Based on the global sequence number in the target memory change log file, determine whether the changed data in the target memory change log file has been synchronized to the target table data file on the disk;
[0153] If the table data in the target memory change log file is not synchronized to the target table data file on the disk, synchronize the table data in the target memory change log file to the target table data file on the disk.
[0154] If the target table data file is synchronized to disk, delete the target memory change log file.
[0155] In practice, each of the above units can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units, please refer to the previous method embodiments, which will not be repeated here.
[0156] As described above, in this embodiment, the database startup unit 201 obtains the startup instruction for the target database and starts the target database; the file reading unit 202 reads the target table data file and memory change log file corresponding to the target database stored on the disk. The memory change log file records the operation information of the full cache data of the target table data file in memory, including the changed data operated on in the full cache data; the data merging unit 203 merges the table data in the target table data file and the changed data in the memory change log file to obtain the target table data; and the data caching unit 204 constructs the target full cache data corresponding to the target table data file in memory based on the target table data. Thus, by recording the operation information of the full cache data of the target table data file in memory in the memory change log file, when the database restarts due to an anomaly, the latest data of the target table data file fully cached in memory at the time of the database anomaly can be determined based on the target table data file stored on the disk and the memory change log file. This enables the recovery of the table data fully cached in memory when the database anomaly occurs, ensuring the persistence of the fully cached data in memory and improving data processing efficiency.
[0157] This application also provides a computer device, such as... Figure 10 As shown, it illustrates a structural diagram of a computer device involved in an embodiment of this application. This computer device may be a server, specifically:
[0158] The computer device may include components such as a processor 301 with one or more processing cores, a memory 302 with one or more computer-readable storage media, a power supply 303, and an input unit 304. Those skilled in the art will understand that... Figure 10 The computer device structure shown does not constitute a limitation on the computer device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:
[0159] The processor 301 is the control center of the computer device, connecting various parts of the computer device through various interfaces and lines. It performs various functions and processes data by running or executing software programs and / or modules stored in the memory 302, and by calling data stored in the memory 302. Optionally, the processor 301 may include one or more processing cores; preferably, the processor 301 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 301.
[0160] The memory 302 can be used to store software programs and modules. The processor 301 executes various functional applications and data processing by running the software programs and modules stored in the memory 302. The memory 302 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 302 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 302 may also include a memory controller to provide the processor 301 with access to the memory 302.
[0161] The computer device also includes a power supply 303 that supplies power to the various components. Preferably, the power supply 303 can be logically connected to the processor 301 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 303 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.
[0162] The computer device may also include an input unit 304, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.
[0163] Although not shown, the computer device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 301 in the computer device loads the executable files corresponding to the processes of one or more applications into the memory 302 according to the following instructions, and the processor 301 runs the applications stored in the memory 302 to realize various functions, as follows:
[0164] Obtain the startup command for the target database and start the target database; read the target table data file and memory change log file corresponding to the target database stored on disk. The memory change log file is used to record operation information on the full cache data of the target table data file in memory, including the changed data operated on in the full cache data; merge the table data in the target table data file and the changed data in the memory change log file to obtain the target table data; construct the target full cache data corresponding to the target table data file in memory based on the target table data.
[0165] The specific implementation of each of the above operations can be found in the preceding embodiments, and will not be repeated here. It should be noted that the computer device provided in this application embodiment and the data processing method in the above embodiments belong to the same concept, and its specific implementation process can be found in the above method embodiments, and will not be repeated here.
[0166] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.
[0167] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute steps in any of the data processing methods provided in embodiments of this application. For example, the instructions can execute the following steps:
[0168] Obtain the startup command for the target database and start the target database; read the target table data file and memory change log file corresponding to the target database stored on disk. The memory change log file is used to record operation information on the full cache data of the target table data file in memory, including the changed data operated on in the full cache data; merge the table data in the target table data file and the changed data in the memory change log file to obtain the target table data; construct the target full cache data corresponding to the target table data file in memory based on the target table data.
[0169] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.
[0170] Since the instructions stored in the computer-readable storage medium can execute the steps of any of the data processing methods provided in the embodiments of this application, the beneficial effects that any of the data processing methods provided in the embodiments of this application can achieve can be realized, as detailed in the preceding embodiments, and will not be repeated here.
[0171] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various optional implementations of the above embodiments.
[0172] The data processing method, apparatus, and computer-readable storage medium provided in the embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.
Claims
1. A data processing method, characterized in that, include: Obtain the startup command for the target database and start the target database; The target table data file and memory change log file corresponding to the target database stored on the disk are read. The memory change log file is used to record operation information of the full cache data of the target table data file in memory. The operation information includes the changed data operated on in the full cache data. The target table data file is a table data file that uses full caching. The memory change log file is a write-ahead log file. Each line of data in the target table data file and the memory change log file has a global sequence number. Obtain the global sequence number of the change data for each line of change data in the memory change log file; Based on the global sequence number of the changed data, the table data and the changed data are compared, and the target changed data that has not been written to the target table data file is determined from the changed data according to the comparison result. The target change data and the table data are merged to obtain the target table data; Based on the target table data, construct the target full cache data corresponding to the target table data file in memory; Determine the number of memory change log files; When the number of files exceeds a preset file number threshold, the target memory change log file to be processed is determined based on the file number of the memory change log file. The file number of the target memory change log file is not the largest file number among the file numbers of the memory change log files. Based on the global sequence number in the target memory change log file, determine whether the changed data in the target memory change log file has been synchronized to the target table data file on the disk; If the table data in the target memory change log file is not synchronized to the target table data file on the disk, synchronize the table data in the target memory change log file to the target table data file on the disk. If the target table data file is synchronized to the disk, delete the target memory change log file.
2. The data processing method as described in claim 1, characterized in that, Also includes: Record the largest target global sequence number among the global sequence numbers of the changed data; After constructing the target full cache data corresponding to the target table data file in memory based on the target table data, the method further includes: When a data addition operation is received for the target full cache data, the maximum target global sequence number recorded is obtained; The target global index is incremented to obtain the first target global index; Write the first target data corresponding to the data addition operation into the target full cache data; The memory change log file is updated based on the first target global sequence number and the first target data.
3. The data processing method as described in claim 2, characterized in that, The step of updating the memory change log file based on the first target global sequence number and the first target data includes: Obtain the amount of stored data in the memory change log file; When the amount of stored data is not greater than a preset data amount threshold, the first target global sequence number and the first target data are written to the memory change log file; When the amount of stored data exceeds the preset data volume threshold, obtain the maximum file number of the memory change log file; Create a new memory change log file, and assign the file number of the new memory change log file to the file number obtained by incrementing the maximum file number; Write the first target global sequence number and the first target data into the new memory change log file.
4. The data processing method as described in claim 1, characterized in that, After constructing the target full cache data corresponding to the target table data file in memory based on the target table data, the method further includes: In response to a data modification operation on the target full cache data, the second target data corresponding to the data modification operation in the target full cache data is located; Lock the row containing the second target data, and modify the second target data based on the data modification operation; The memory change log file is updated based on the global sequence number of the second target data and the modified second target data.
5. The data processing method as described in claim 1, characterized in that, After constructing the target full cache data corresponding to the target table data file in memory based on the target table data, the method further includes: In response to a data deletion operation on the target full cache data, the third target data corresponding to the data deletion operation is located in the target full cache data; Lock the row containing the third target data and modify the deletion marker corresponding to the third target data; The memory change log file is updated based on the modified deletion marker and the global sequence number corresponding to the third target data.
6. A data processing apparatus, characterized in that, include: A database startup unit is used to obtain a startup instruction for a target database and start the target database; The file reading unit is used to read the target table data file and memory change log file corresponding to the target database stored on the disk. The memory change log file is used to record operation information on the full cache data of the target table data file in memory. The operation information includes the changed data operated on in the full cache data. The target table data file is a table data file that uses full caching. The memory change log file is a write-ahead log file. Each line of data in the target table data file and the memory change log file has a global sequence number. The data merging unit is used to obtain the global sequence number of each line of changed data in the memory change log file, compare the table data and the changed data according to the global sequence number of the changed data, determine the target changed data that has not been written to the target table data file according to the comparison result, and perform data merging processing on the target changed data and the table data to obtain the target table data. A data caching unit is used to construct target full cache data corresponding to the target table data file in memory based on the target table data; The data synchronization unit is used to determine the number of memory change log files. When the number of files exceeds a preset file number threshold, it determines the target memory change log file to be processed based on the file number of the memory change log files. The file number of the target memory change log file is not the largest file number among the file numbers of the memory change log files. Based on the global sequence number in the target memory change log file, it determines whether the changed data in the target memory change log file has been synchronized to the target table data file on the disk. If it has not been synchronized to the target table data file on the disk, it synchronizes the table data in the target memory change log file to the target table data file on the disk. If it has been synchronized to the target table data file on the disk, it deletes the target memory change log file.
7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to perform the steps of the data processing method according to any one of claims 1 to 5.
8. A computer device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the data processing method according to any one of claims 1 to 5.