Data processing method and device

Abstract

Embodiments of the present specification provide a data processing method and device, wherein the data processing method includes: determining a memory allocation area corresponding to an area address, and a memory allocator associated with the memory allocation area; obtaining index data of the memory allocation area, and an allocation status of the memory allocator, and writing the index data and the allocation status to a disk file; when a memory database containing the memory allocation area is restarted, reading the index data and the allocation status from the disk file; mapping the index data to the memory allocation area according to the area address, and updating the status of the memory allocator according to the allocation status.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of database technology, and in particular to a data processing method and device. Background Art

[0002] An in-memory database refers to a database that stores data in memory and operates directly on it. It has a faster read and write speed than a disk, and by storing data in memory rather than accessing it from a disk, the performance of the application can be greatly improved. However, since in-memory databases are based on computer memory, their persistence problem has always existed. To solve this problem, the prior art backs up the index in memory at a set time node so that the application can continue to run through the backed-up index after the in-memory database is restarted. However, in order to restore the index when the in-memory database is restarted, it takes a certain amount of time to scan the data. When the amount of data is large, restoring the index takes a lot of time, which restricts the scalability of the in-memory database. Therefore, there is an urgent need for an effective solution to solve the above problems. Summary of the Invention

[0003] In view of this, embodiments of this specification provide a data processing method. One or more embodiments of this specification also relate to a data processing apparatus, a computing device, a computer-readable storage medium, and a computer program to address technical deficiencies in the prior art.

[0004] According to a first aspect of an embodiment of this specification, there is provided a data processing method, including:

[0005] Determine a memory allocation region corresponding to the region address and a memory allocator associated with the memory allocation region;

[0006] Obtaining index data of the memory allocation area and allocation status of the memory allocator, and writing the index data and allocation status to a disk file;

[0007] When the in-memory database including the memory allocation area is restarted, reading the index data and the allocation status from the disk file;

[0008] The index data is mapped to the memory allocation area according to the area address, and the state of the memory allocator is updated according to the allocation state.

[0009] According to a second aspect of the embodiments of this specification, there is provided a data processing device, including:

[0010] a determination module configured to determine a memory allocation region corresponding to a region address and a memory allocator associated with the memory allocation region;

[0011] an acquisition module configured to acquire index data of the memory allocation area and an allocation status of the memory allocator, and write the index data and the allocation status into a disk file;

[0012] a reading module configured to read the index data and the allocation status from the disk file when the memory database containing the memory allocation area is restarted;

[0013] An update module is configured to map the index data to the memory allocation area according to the area address, and update the state of the memory allocator according to the allocation state.

[0014] According to a third aspect of an embodiment of this specification, a computing device is provided, including:

[0015] memory and processor;

[0016] The memory is used to store computer-executable instructions, and the processor is used to implement any step of the above-mentioned data processing method when executing the computer-executable instructions.

[0017] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of the above-mentioned data processing method are implemented.

[0018] According to a fifth aspect of the embodiments of this specification, a computer program is provided, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned data processing method.

[0019] The data processing method provided in this specification can obtain the index data of the memory allocation area and the allocation status of the memory allocator after determining the memory allocation area corresponding to the area address and the memory allocator that manages the memory allocation area. At this time, the index data and the allocation status can be written to the disk file together to achieve persistence of the index and status through physical replication, avoiding the problem of being unable to restore the index due to the restart of the memory database. When the memory database containing the memory allocation area is restarted, the index data and allocation status can be read directly from the disk file, and the index data can be mapped to the memory allocation area according to the allocation address. At the same time, the status of the memory allocator is updated according to the allocation status, and the recovery of the index and status is completed in combination with the disk file. This can effectively reduce the restart time of the memory database so that the memory database can continue to run in the state before the restart. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a flow chart of a data processing method provided by one embodiment of this specification;

[0021] Figure 2 is a schematic diagram of a data processing method provided by an embodiment of this specification;

[0022] Figure 3 This is a flowchart of a data processing method provided by one embodiment of this specification;

[0023] Figure 4 This is a schematic diagram of the structure of a data processing device provided by one embodiment of this specification;

[0024] Figure 5 This is a structural block diagram of a computing device provided by one embodiment of this specification. DETAILED DESCRIPTION

[0025] The following description sets forth many specific details to facilitate a thorough understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the scope of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0026] The terms used in one or more embodiments of this specification are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of this specification. The singular forms "a," "the," and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0027] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0028] First, the terms involved in one or more embodiments of this specification are explained.

[0029] Physical copying: refers to the operation of directly copying the memory data of the running program to other memory or disk areas.

[0030] Database index: refers to the data structure maintained by the database for query during operation.

[0031] In-memory databases operate on data stored directly in memory. Compared to disk, memory read and write speeds are several orders of magnitude higher. Storing data in memory can significantly improve application performance compared to accessing it from disk.

[0032] In this specification, a data processing method is provided. This specification also relates to a data processing apparatus, a computing device, a computer-readable storage medium, and a computer program, which are described in detail one by one in the following embodiments.

[0033] In real-world applications, in-memory databases typically use technologies such as redo logs and checkpoints to restore indexes during restart. Redo logs continuously generate records of index operations during database operation. During recovery, these records are re-executed to update the database index after restart. Once the records are executed, index recovery is complete. However, recovery is slow because the records must be executed sequentially. Checkpoints, on the other hand, periodically scan the index during database operation and save it to memory. During recovery, the index can be restored by scanning and saving it in memory. However, this process requires establishing checkpoints, and index saving is slow, disrupting the normal operation of in-memory databases. Therefore, an effective solution is urgently needed to address these issues.

[0034] In view of this, the data processing method provided in this specification can obtain the index data of the memory allocation area and the allocation status of the memory allocator after determining the memory allocation area corresponding to the area address and the memory allocator that manages the memory allocation area. At this time, the index data and the allocation status can be written to the disk file together, so as to achieve persistence of the index and status through physical copying, thereby avoiding the problem of being unable to restore the index due to the restart of the memory database. When the memory database containing the memory allocation area is restarted, the index data and the allocation status can be read directly from the disk file, and the index data can be mapped to the memory allocation area according to the allocation address. At the same time, the status of the memory allocator is updated according to the allocation status, and the recovery of the index and status is completed in combination with the disk file. This can effectively reduce the restart time of the memory database so that the memory database can continue to run in the state before the restart.

[0035] Figure 1 A flow chart of a data processing method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0036] Step S102: determining a memory allocation region corresponding to the region address and a memory allocator associated with the memory allocation region.

[0037] Specifically, the region address refers to the fixed address space used by the memory allocation area in memory, which is used to ensure that the address of the memory allocation area does not change before and after the memory database is restarted. Correspondingly, the memory allocation area refers to the allocation area in the memory database used to store index data, wherein the index data is used to operate and maintain the data structure for query during the operation of the memory database. Only the index data needs to be saved to ensure that the memory database can resume its pre-restart operation state after restart. Correspondingly, the memory allocator refers to the program in the memory database that manages the memory allocation area containing memory blocks, which can determine the allocation status of each memory block, how to allocate memory blocks for queries, how to handle free memory blocks after memory blocks are allocated, and how to release memory blocks.

[0038] Based on this, and considering that the memory block status of the memory allocation area is managed by the memory allocator, when the memory database is restarted, if only the index data of the memory allocation area is restored without considering the status of each memory block in the memory allocation area, it will cause a huge waste of memory resources. Therefore, in order to avoid wasting memory resources, it is possible to determine the memory allocation area corresponding to the area address during the stage when index data needs to be saved for the memory database, and at the same time determine the memory allocation area. The memory allocator associated with the memory allocation area, that is, the memory allocator that manages the memory allocation area, can be used to save both the index data and the allocation status in the future, so as to improve the restart speed of the memory database while avoiding waste of memory resources.

[0039] Furthermore, in the process of determining the memory allocation area, considering that the memory database includes not only the allocation area for storing index data but also the allocation area for storing other data, if data is saved in all areas at this stage, more time and resources will be consumed. Therefore, in order to improve resource utilization and reduce time consumption, the memory allocation area can be determined according to the pre-configured rules of the memory database. That is, after a backup event is detected, the memory allocation area will be determined according to the pre-configured rules. In this embodiment, the specific implementation method is as follows:

[0040] When a backup event corresponding to the memory database is detected, at least two initial memory allocation areas are determined in the memory database, and the at least two initial memory allocation areas include an index allocation area and a temporary allocation area; the area address is determined according to the backup event, and the index allocation area is selected as the memory allocation area in the at least two initial memory allocation areas according to the area address.

[0041] Specifically, a backup event refers to an event requiring the saving of index data for the in-memory database at the current time node. This is used to prevent the in-memory database from being unable to continue running programs on the node before the restart after the restart. Accordingly, an initial memory allocation area refers to multiple memory allocation areas defined based on different rules within the in-memory database, including an index allocation area and a temporary allocation area. The index allocation area refers to the memory allocation area for storing index data, and the temporary allocation area refers to the memory allocation area for storing temporary data.

[0042] Based on this, when a backup event corresponding to the in-memory database is detected, it indicates that the index data in the in-memory database needs to be backed up at the current time point. During this process, considering that the in-memory database contains multiple different initial memory allocation areas, and the data to be saved only involves index data, it is sufficient to determine the initial memory allocation area for storing the index data. Specifically, the area address can be determined based on the backup event, and then, based on the area address, the index allocation area is selected from at least two initial memory allocation areas as the memory allocation area for subsequent index data storage processing.

[0043] In specific implementation, when dividing the memory allocation area in the memory database, the memory allocation area that stores index data can be used as the index allocation area, and the remaining other memory allocation areas can be used as temporary allocation areas, so as to achieve a more concise distinction of the memory allocation areas in the memory database, thereby improving the efficiency of index data copying.

[0044] In summary, by distinguishing the initial memory allocation area and selecting the index allocation area corresponding to the area address as the memory allocation area for subsequent index data storage processing, the index data storage efficiency can be effectively improved.

[0045] Step S104: obtaining index data of the memory allocation area and the allocation status of the memory allocator, and writing the index data and the allocation status into a disk file.

[0046] Specifically, based on the above determination of the memory allocation area and memory allocator, in order to achieve persistence of index data and allocation status, so that after the in-memory database is restarted, the program running state before the restart can be restored based on the index data and allocation status to avoid resource waste, the index data stored in the memory allocation area and the allocation status of the memory allocator can be first obtained, and then both the index data and allocation status can be written to the disk file to achieve the purpose of data persistence.

[0047] The allocation state specifically refers to the state in which the memory allocator records the allocation status of each memory block in the memory allocation area; correspondingly, the disk file specifically refers to a computer file that is not affected by power outages and can achieve data persistence.

[0048] In actual applications, when writing index data to a disk file, you can use mmap mapping file to write the index data to the disk file, or directly write the disk file for physical copying, so that the index data can be saved using the disk file, so that the index data can be persisted to avoid loss and unusability.

[0049] Furthermore, in order to reduce the consumption of memory resources and enable the memory allocator to take over the restored memory allocation area in a shorter time after the in-memory database is restarted, it is also necessary to determine the allocation state of the memory allocator and persist it so that the stored allocation state can be reused after the in-memory database is restarted, thereby reducing the startup time of the in-memory database. In this embodiment, the specific implementation method is as follows:

[0050] A target memory allocation area associated with the memory allocation area is created, and an allocation operation of the memory allocation area is switched to the target memory allocation area; and the allocation state of the memory allocator is read according to the switching result.

[0051] Specifically, the target memory allocation area refers to a memory allocation area that is not affected by other operations when storing index data in the memory allocation area, thereby switching the allocation operation of the memory allocation area to the target memory allocation area. Accordingly, the allocation operation switching refers to the allocation operation that occurs during the index data storage phase and needs to be switched to the target memory allocation area for completion.

[0052] Based on this, in order to avoid conflicts in system operation during the saving process, the atomic scheme can be used to save the memory allocator state. In other words, the allocation state saving and allocation operation do not occur on the memory allocation area. That is, the memory allocator can first create a new memory allocation area, namely the target memory allocation area, and then switch the allocation operation of the memory allocation area to the target memory allocation area. In the index data saving phase, the operation on the memory allocation area can be transferred to the target memory allocation area to complete the corresponding allocation operation on the target memory allocation area. In this way, the allocation state of the memory allocator can be determined according to the result of the allocation operation switch.

[0053] It should be noted that when saving the index data in the memory allocation area, if the corresponding allocation operation is still performed on the old memory allocation area, the allocation status may not be saved accurately. On this basis, if the memory database is restarted, there will be a problem of mismatch between the index data and the allocation status. Therefore, in order to avoid the system instability caused by this problem, a new memory allocation area is created to complete the allocation operation on the new memory allocation area.

[0054] In summary, by creating a new memory allocation area and switching the allocation operation of the old memory allocation area to the new memory allocation area, the index data storage and allocation operations can be independent of each other, so that the index data can be saved without conflicts in the system operation, thereby achieving the purpose of index data persistence.

[0055] Furthermore, when reading the allocation status according to the allocation operation switching result of the memory allocation area, considering that the content allocation area contains multiple memory blocks, and each memory block corresponds to a different allocation status, it is necessary to integrate the allocation status of all memory blocks to determine the allocation status of the memory allocator. In this embodiment, the specific implementation is as follows:

[0056] The memory allocation area is scanned according to the switching result to determine the memory blocks contained in the memory allocation area; allocation information of the memory blocks is determined according to the memory allocator; and the allocation state of the memory allocator is generated by integrating the allocation information of the memory blocks.

[0057] Specifically, a memory block refers to the smallest memory unit in a memory allocation area, and multiple memory blocks constitute a memory allocation area; correspondingly, allocation information refers to information about whether a memory block is allocated. By integrating the allocation information of each memory block, the allocation status of the memory allocator can be determined.

[0058] Based on this, after the allocation operation switch of the memory allocation area is completed, the memory allocation area can be scanned to determine the memory blocks contained in the memory allocation area. Then, the allocation information of each memory block is determined according to the memory allocator, that is, the allocation status of each memory block. Finally, by integrating the allocation information of the memory blocks, the allocation status of the memory allocation area in the memory allocator can be obtained for subsequent persistence.

[0059] In summary, by scanning the allocation information of each memory block contained in the memory allocation area, the allocation status of the memory allocator can be accurately determined. Persistence is performed on this basis, so that the index data written in the disk file can match the allocation status, so that the in-memory database can be quickly started during the recovery phase.

[0060] In addition, considering that the memory allocation area may receive a release instruction during the process of saving index data and allocation status, if the release operation is performed directly in the content allocation area, the saved allocation status may not correspond to the current allocation status. If the memory database is restarted on this basis, the release operation will not be executed. Therefore, in order to avoid the above problem, the release process can be completed for the memory block status corresponding to the release operation. In this embodiment, the specific implementation method is as follows:

[0061] Upon receiving a memory block release instruction submitted for the memory allocation area, determine the target memory block; detect whether the target allocation status of the target memory block is saved to the disk file; if not, delay processing of the memory release instruction until the target allocation status is written to the disk file, and release the memory block mapped by the target memory block in the target memory allocation area according to the memory block release instruction; if so, release the memory block mapped by the target memory block in the target memory allocation area according to the memory block release instruction.

[0062] Specifically, a memory block release instruction refers to an operation for releasing a target memory block contained in a memory allocation area, thereby deleting data stored in the target memory block. Accordingly, a target memory block refers to a memory block in the memory allocation area that is currently being released. Accordingly, a target allocation state refers to the allocation state of the target memory block.

[0063] Based on this, when the allocation state of the memory allocator is persisted, if a memory block release instruction submitted to the memory allocation area is received, the target memory block that needs to be released by the memory block release instruction can be determined. At this time, it is possible to detect whether the target allocation state of the target memory block has been saved to the disk file, that is, to detect whether the target allocation state is persisted. If not, it means that the target allocation state of the target memory block has not been written to the disk file. If the target memory block is continuously released, it will affect the system operation and cause system conflicts, that is, affect the storage of index data and allocation state; therefore, in order to complete the memory release without affecting data storage, the memory release instruction can be delayed, that is, the memory release operation can be delayed until the target allocation state is written to the disk file, and then the memory block mapped by the target memory block is released in the target memory allocation area according to the delayed memory release instruction.

[0064] If so, it means that the target allocation status of the target memory block has been written to the disk file. At this time, the release process will not affect the allocation status writing to the disk file. Therefore, the memory block release instruction can be executed directly to release the memory block mapped by the target memory block in the target memory allocation area.

[0065] It should be noted that the target memory allocation area is a newly created memory allocation area for the memory allocation area. Therefore, when executing a memory block release instruction, the memory block mapped by the target memory block must be released in the target memory allocation area to achieve the purpose of freeing memory resources. In actual applications, when delaying the processing of memory block release instructions, tcache can be used to cache the release operation to achieve delayed processing.

[0066] In addition, after the allocation state is saved, in order to avoid waste of resources and considering that the old memory allocation area may still contain memory blocks that are not managed by the target allocation area, the memory allocation area and the target memory allocation area can be merged; that is, the ownership of the old memory allocation area is transferred to the new memory allocation area, so that the memory allocator can manage all memory blocks.

[0067] In summary, when a release operation occurs during storage, the release operation can be processed according to different situations, so that the release operation can be processed without affecting the storage of the allocation state, thereby improving the concurrent processing capability of the system.

[0068] Furthermore, after obtaining the index data and allocation status, considering that the index data may contain running index data, which only exists during the program running phase, if it is saved, it will cause a waste of storage resources. Therefore, the index data can be compressed after being saved to achieve the purpose of reducing the storage space occupied. In this embodiment, the specific implementation method is as follows:

[0069] Determine the running index data included in the index data; delete the running index data in the index data, and write the index data containing the deleted running index data into the disk file.

[0070] Specifically, the running index data refers to the index data that exists in the memory database during the program running process. After the program stops or is closed, this part of the index data will be released; that is, the running index data will only exist during the program running process, and in the index data saving stage, the program is in a running state. At this time, the index data will contain the running index data. If all the index data are saved at this time, more storage space will be consumed. In order to improve space utilization, the running index data in the index data can be determined first, and then the running index data can be deleted and the index data of the running index data can be written to the disk file to achieve the purpose of saving space resources.

[0071] In practical applications, when saving index data that does not include running index data, the storage space occupied by the running index data can be reduced by compressing the saved result after the index data is written to the disk file. That is, after the index data is written to the disk file, the running index data is determined from the index data written to the disk file and then deleted from the disk file to save storage space resources. The running index data includes, but is not limited to, linked list structures and linked list pointers.

[0072] During specific implementation, the compression method for index data can be selected according to the actual application scenario, and this embodiment does not impose any limitation on this.

[0073] In summary, by deleting running index data and reducing the amount of index data written to disk files, we can effectively improve the efficiency of index data writing and avoid wasting storage resources. In the subsequent recovery phase, we can also quickly restart the in-memory database based on a small amount of index data.

[0074] Furthermore, when writing the allocation state of the memory allocator to the disk file, it is considered that the allocation state is the state of the memory blocks contained in the memory allocation area, and different memory blocks may have different allocation states at the current moment, that is, allocated or unallocated. How to save the state corresponding to the unallocated memory block? In the recovery phase, it is also in an unused state and will not affect the program recovery operation. Therefore, if its state is saved, the startup speed of the memory database will be reduced. Therefore, when storing the allocation state, the unallocated memory block can be released, and then the allocation state can be updated to reduce the space occupancy of the allocation state. Finally, it can be written to the disk file. In this embodiment, the specific implementation method is as follows:

[0075] Determine unallocated memory blocks to be released in the memory allocation area according to the allocation status, and perform release processing on the memory blocks to be released; update the allocation status according to the release processing result, and write the updated allocation status into the disk file.

[0076] Specifically, the memory blocks to be released specifically refer to the unused memory blocks in the memory allocation area, that is, the unallocated memory blocks. Based on this, when storing the allocation status, the allocation status can be traversed first to determine the allocation status of each memory block in the memory allocation area, and then the unallocated memory blocks are selected as the memory blocks to be released. By releasing the memory blocks to be released, the memory blocks to be released are returned to the memory database. At this time, the allocation status can be updated according to the release result to remove the allocation status of the released memory blocks to be released. Finally, the updated allocation status can be written to the disk file to achieve the compression of the allocation status space occupancy rate.

[0077] For example, the memory allocation area is divided into n blocks for use, and a bitmap data structure is used to save the allocation status of each block. If all n memory blocks are used, continuing to use the bitmap will occupy more memory space. Therefore, 1 bit can be used to indicate that n memory blocks are allocated for use. At this time, the bitmap can be deleted and there is no need to write it to the disk file, so as to save space resources.

[0078] In summary, by adopting the method of releasing the memory blocks to be released, the file size of the allocation status can be reduced, so that the allocation status written to the disk file will not contain redundant information, thereby reducing the amount of allocation status data that needs to be restored and improving the startup speed of the in-memory database.

[0079] Step S106 : when the memory database including the memory allocation area is restarted, reading the index data and the allocation status from the disk file.

[0080] Specifically, after the above-mentioned writing of the index data and allocation status to the disk file is completed, the program can continue to be provided with an operating environment through the memory database. If the memory database is restarted due to external factors after the index data and allocation status are written to the disk file and before the new index data and allocation status are rewritten next time, such as a system crash restart, a power outage restart, etc., the index data and allocation status in the disk file can be used to restart the memory database, so that the memory database can be restored to the state before the restart and continue to provide an operating environment for the program. Therefore, after the memory database containing the memory allocation area is restarted, the index data and allocation status can be read from the disk file, and then the index data can be restored to the memory, and the memory allocator can be updated according to the allocation status, so as to achieve the purpose of restarting the memory database to restore the program's operating environment.

[0081] Furthermore, considering that during the restart phase of the memory database, continuing to provide memory for the program to run has the highest priority, the restoration of index data and allocation status can be completed according to the restart configuration information of the memory database. In this embodiment, the specific implementation is as follows:

[0082] Obtain the restart configuration information preset for the memory database, and restart the memory database according to the restart configuration information; based on the restart processing result of the memory database, detect whether the snapshot information of the memory allocator is available; if so, execute the step of reading the index data and the allocation status in the disk file; if not, restart the program and re-establish the index.

[0083] Specifically, the restart configuration information refers to the information that records the startup priority of each function when the in-memory database is restarted, which is used to control the restart order of each function during the restart phase of the in-memory database; correspondingly, the snapshot information refers to the information stored in the allocation status of the memory allocator. By judging whether the snapshot information is available, it can be determined whether the index data and allocation status can be restored.

[0084] Based on this, during the restart phase of the memory database, the restart configuration information of the memory database can be obtained first, and the memory database can be restarted according to the restart configuration information to restart some functions in the memory database in sequence, so as to achieve a linear restart method to avoid calling too many computing resources at the same time. After the restart, in order to restore the memory database to the state before the restart, it is necessary to detect the snapshot information of the memory allocator. If the snapshot information is available, it means that the index data and allocation status written in the disk file are not damaged. Then, step S106 can be executed to read the index data and allocation status in the disk file for subsequent use. If the snapshot information is not available, it means that the index data and allocation status written in the disk file are damaged. Then, it is necessary to restart the startup program and re-establish the index.

[0085] In summary, by restarting the in-memory database using restart configuration information, we can implement a linear restart of some functions to reduce the occurrence of high-concurrency memory usage problems, thereby increasing the restart speed of the in-memory database and restoring program operation in a shorter time, thus avoiding prolonged program interruption and affecting downstream services.

[0086] Step S108 : mapping the index data to the memory allocation area according to the area address, and updating the state of the memory allocator according to the allocation state.

[0087] Specifically, after reading the index data and allocation status from the disk file, in order to restore the memory database to the state before the restart, the index data can be mapped to the memory allocation area according to the area address, and the state of the memory allocator after the restart can be updated according to the allocation status, so as to realize that the memory database after the restart can be restored to the state before the restart so as to continue running the program.

[0088] Furthermore, when reading index data from a disk file and mapping it to a memory allocation area, it is actually after the memory database is restarted. The area address can be determined based on the information recorded in the disk file, thereby determining the memory allocation area in the memory database. Then, the index data can be read from the disk file and re-mapped back to the memory to complete the index reconstruction.

[0089] Furthermore, after the index data is restored, the allocation state needs to be restored to avoid wasting memory resources. Therefore, the memory allocation area needs to be re-registered to the memory allocator before the state is restored. In this embodiment, the specific implementation is as follows:

[0090] The memory allocation area is registered in the memory allocator, and the allocation status is scanned according to the registration result; and allocation information of the memory blocks contained in the memory allocation area is updated according to the scanning result as a status update of the memory allocator.

[0091] See also Figure 2 As shown in the schematic diagram, after starting the server and loading the server configuration from the initial allocation time, some functions can be selectively started to improve the startup speed of the memory database. On this basis, during the startup of the memory database, it is first necessary to try to read the status of the memory allocator, and after success, map the index data to the memory allocation area. At this time, the system operation status will be consistent with that before the snapshot was generated, but since the corresponding memory allocator status is lost, a huge waste of memory resources will be generated, and it will not be able to continue to run normally. Therefore, the status of the memory allocator needs to be restored.

[0092] In other words, you need to first register the memory allocation area with the memory allocator, then scan the allocation status based on the registration result; based on the scan result, update the allocation information of the memory blocks contained in the memory allocation area as a status update to the memory allocator. This is done by first re-registering the memory allocation area with the memory allocator, then scanning the saved allocation status, and modifying the allocator status of the corresponding memory block to allocated. This updates the status of each memory block in the memory allocation area to the memory allocator.

[0093] like Figure 2 As shown in the figure, after the in-memory database is restarted, it first checks whether the memory allocator snapshot information is available. If so, it loads the index data and allocation status, then modifies the allocator status of the corresponding area to restore the in-memory database. If the snapshot information is unavailable, the index data is initialized, and the log load data is rebuilt. The in-memory database restart is completed according to this logic on any recovery node, ensuring continuity before and after the in-memory database restart.

[0094] In summary, by re-registering the memory allocation area and restoring the allocation status, the state loss of the memory allocator can be avoided, the utilization rate of memory resources can be greatly improved, and the waste of resources in the case of state loss can be avoided.

[0095] In addition, to avoid the hole problem, the memory allocation area can be scanned again after the allocation state is restored to return the unallocated memory blocks. In this embodiment, the specific implementation is as follows:

[0096] The memory allocation area is scanned by the memory allocator after state update to determine unallocated free memory blocks in the memory allocation area; and release processing is performed on the free memory blocks.

[0097] Specifically, free memory blocks refer to unallocated memory blocks after the memory allocation area is re-registered with the memory allocator. Based on this, after the memory allocator's status is updated, the memory allocation area can be rescanned to identify unallocated free memory blocks within the memory allocation area. These free memory blocks can then be released, returning the unallocated areas to the in-memory database and improving memory utilization.

[0098] In summary, after restoring the memory database, scanning the memory allocation area again can reclaim the free memory blocks to improve the memory space utilization.

[0099] The data processing method provided in this specification can obtain the index data of the memory allocation area and the allocation status of the memory allocator after determining the memory allocation area corresponding to the area address and the memory allocator that manages the memory allocation area. At this time, the index data and the allocation status can be written to the disk file together to achieve persistence of the index and status through physical replication, avoiding the problem of being unable to restore the index due to the restart of the memory database. When the memory database containing the memory allocation area is restarted, the index data and allocation status can be read directly from the disk file, and the index data can be mapped to the memory allocation area according to the allocation address. At the same time, the status of the memory allocator is updated according to the allocation status, and the recovery of the index and status is completed in combination with the disk file. This can effectively reduce the restart time of the memory database so that the memory database can continue to run in the state before the restart.

[0100] The following combined Figure 3 , taking the application of the data processing method provided in this specification in the memory database recovery scenario as an example, the data processing method is further explained. Figure 3 A flowchart of a data processing method provided in one embodiment of this specification is shown, which specifically includes the following steps.

[0101] Step S302: When a backup event corresponding to the memory database is detected, at least two initial memory allocation areas are determined in the memory database.

[0102] Step S304: determining a region address according to the backup event, and selecting an index allocation region as the memory allocation region from at least two initial memory allocation regions according to the region address.

[0103] Step S306: Determine the memory allocator associated with the memory allocation area.

[0104] Step S308: obtaining index data of the memory allocation area, and creating a target memory allocation area associated with the memory allocation area.

[0105] Step S310: Switch the allocation operation of the memory allocation area to the target memory allocation area, and read the allocation state of the memory allocator according to the switching result.

[0106] Step S312: determining the running index data included in the index data.

[0107] Step S314: Delete the running index data in the indexed data, and write the index data containing the deleted running index data into a disk file.

[0108] Step S316: determining unallocated memory blocks to be released in the memory allocation area according to the allocation status, and performing release processing on the memory blocks to be released.

[0109] Step S318: Update the allocation status according to the release processing result, and write the updated allocation status into the disk file.

[0110] In this process, when a memory block release instruction submitted to a memory allocation area is received, a target memory block is determined; it is detected whether the target allocation state of the target memory block is saved to the disk file; if not, the memory release instruction is delayed until the target allocation state is written to the disk file, and the memory block mapped by the target memory block is released in the target memory allocation area according to the memory block release instruction; if so, the memory block mapped by the target memory block is released in the target memory allocation area according to the memory block release instruction.

[0111] Step S320: When the in-memory database is restarted, the restart configuration information preset for the in-memory database is obtained.

[0112] Step S322: Restart the memory database according to the restart configuration information, and detect snapshot information of the memory allocator based on the restart processing result of the memory database.

[0113] Step S324 : when the snapshot information is available, read the index data from the disk file, and map the index data to the memory allocation area according to the area address.

[0114] Step S326: register the memory allocation area in the memory allocator, and scan the allocation status in the disk file according to the registration result.

[0115] Step S328: Update the allocation information of the memory blocks contained in the memory allocation area according to the scanning result, as a status update of the memory allocator.

[0116] Step S330 , scanning the memory allocation area through the memory allocator after the state is updated, and releasing the unallocated free memory blocks in the memory allocation area.

[0117] In summary, after determining the memory allocation area corresponding to the area address and the memory allocator that manages the memory allocation area, the index data of the memory allocation area and the allocation status of the memory allocator can be obtained. At this time, the index data and the allocation status can be written to the disk file together to achieve persistence of the index and status through physical replication, avoiding the problem of being unable to restore the index due to the restart of the memory database. When the memory database containing the memory allocation area is restarted, the index data and allocation status can be read directly from the disk file, and the index data can be mapped to the memory allocation area according to the allocation address. At the same time, the status of the memory allocator is updated according to the allocation status, and the recovery of the index and status is completed in combination with the disk file. This can effectively reduce the restart time of the memory database so that the memory database can continue to run in the state before the restart.

[0118] Corresponding to the above method embodiment, this specification also provides a data processing device embodiment, Figure 4 FIG1 shows a schematic diagram of the structure of a data processing device provided by an embodiment of this specification. Figure 4 As shown, the device includes:

[0119] A determination module 402 is configured to determine a memory allocation region corresponding to a region address and a memory allocator associated with the memory allocation region;

[0120] The acquisition module 404 is configured to acquire index data of the memory allocation area and the allocation status of the memory allocator, and write the index data and the allocation status into a disk file;

[0121] A reading module 406 is configured to read the index data and the allocation status from the disk file when the memory database containing the memory allocation area is restarted;

[0122] The updating module 408 is configured to map the index data to the memory allocation area according to the area address, and update the state of the memory allocator according to the allocation state.

[0123] In an optional embodiment, the determining module 402 is further configured to:

[0124] When a backup event corresponding to the memory database is detected, at least two initial memory allocation areas are determined in the memory database, and the at least two initial memory allocation areas include an index allocation area and a temporary allocation area; the area address is determined according to the backup event, and the index allocation area is selected as the memory allocation area in the at least two initial memory allocation areas according to the area address.

[0125] In an optional embodiment, the acquisition module 404 is further configured to:

[0126] Determine unallocated memory blocks to be released in the memory allocation area according to the allocation status, and perform release processing on the memory blocks to be released; update the allocation status according to the release processing result, and write the updated allocation status into the disk file.

[0127] In an optional embodiment, the acquisition module 404 is further configured to:

[0128] Determine the running index data included in the index data; delete the running index data in the index data, and write the index data containing the deleted running index data into the disk file.

[0129] In an optional embodiment, the acquisition module 404 is further configured to:

[0130] A target memory allocation area associated with the memory allocation area is created, and an allocation operation of the memory allocation area is switched to the target memory allocation area; and the allocation state of the memory allocator is read according to the switching result.

[0131] In an optional embodiment, the acquisition module 404 is further configured to:

[0132] The memory allocation area is scanned according to the switching result to determine the memory blocks contained in the memory allocation area; allocation information of the memory blocks is determined according to the memory allocator; and the allocation state of the memory allocator is generated by integrating the allocation information of the memory blocks.

[0133] In an optional embodiment, the device further includes:

[0134] A release processing module is configured to determine a target memory block upon receiving a memory block release instruction submitted for the memory allocation area; detect whether a target allocation state of the target memory block is saved to the disk file; if not, delay processing of the memory release instruction until the target allocation state is written to the disk file, and release the memory block mapped by the target memory block in the target memory allocation area according to the memory block release instruction; if so, release the memory block mapped by the target memory block in the target memory allocation area according to the memory block release instruction.

[0135] In an optional embodiment, the reading module 406 is further configured to:

[0136] Obtain the restart configuration information preset for the memory database, and restart the memory database according to the restart configuration information; based on the restart processing result of the memory database, detect whether the snapshot information of the memory allocator is available; if so, execute the step of reading the index data and the allocation status in the disk file.

[0137] In an optional embodiment, the updating module 408 is further configured to:

[0138] The memory allocation area is registered in the memory allocator, and the allocation status is scanned according to the registration result; and allocation information of the memory blocks contained in the memory allocation area is updated according to the scanning result as a status update of the memory allocator.

[0139] In an optional embodiment, the device further includes:

[0140] The release module is configured to scan the memory allocation area through the memory allocator after the state is updated, determine the unallocated free memory blocks in the memory allocation area, and perform release processing on the free memory blocks.

[0141] The data processing device provided in this specification can obtain the index data of the memory allocation area and the allocation status of the memory allocator after determining the memory allocation area corresponding to the area address and the memory allocator that manages the memory allocation area. At this time, the index data and the allocation status can be written to the disk file together, so as to achieve persistence of the index and status through physical copying, thereby avoiding the problem of being unable to restore the index due to the restart of the memory database. When the memory database containing the memory allocation area is restarted, the index data and the allocation status can be read directly from the disk file, and the index data can be mapped to the memory allocation area according to the allocation address. At the same time, the status of the memory allocator is updated according to the allocation status, and the recovery of the index and status is completed in combination with the disk file. This can effectively reduce the restart time of the memory database so that the memory database can continue to run in the state before the restart.

[0142] The above is a schematic diagram of a data processing device according to this embodiment. It should be noted that the technical solution of the data processing device and the technical solution of the above-mentioned data processing method are based on the same concept. For details not described in detail in the technical solution of the data processing device, please refer to the description of the technical solution of the above-mentioned data processing method.

[0143] Figure 5 The block diagram of a computing device 500 according to one embodiment of the present disclosure is shown. Components of the computing device 500 include, but are not limited to, a memory 510 and a processor 520. The processor 520 is connected to the memory 510 via a bus 530, and a database 550 is used to store data.

[0144] The computing device 500 also includes an access device 540 that enables the computing device 500 to communicate via one or more networks 560. Examples of such networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 540 may include one or more of any type of network interface (e.g., a network interface card (NIC)), whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0145] In one embodiment of the present specification, the above components of the computing device 500 and Figure 5 Other components not shown in the figure may also be connected to each other, for example, via a bus. Figure 5The computing device structure block diagram shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art may add or replace other components as needed.

[0146] Computing device 500 can be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook computer, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smartwatch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or PC. Computing device 500 can also be a mobile or stationary server.

[0147] The processor 520 is configured to execute the following computer-executable instructions, which implement the steps of the above-mentioned data processing method when executed by the processor.

[0148] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above-mentioned data processing method are of the same concept. For details not described in detail in the technical scheme of the computing device, please refer to the description of the technical scheme of the above-mentioned data processing method.

[0149] An embodiment of the present specification further provides a computer-readable storage medium storing computer-executable instructions, which implement the steps of the above-mentioned data processing method when executed by a processor.

[0150] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above-mentioned data processing method are based on the same concept. For details not described in detail in the technical scheme of the storage medium, please refer to the description of the technical scheme of the above-mentioned data processing method.

[0151] An embodiment of the present specification further provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned data processing method.

[0152] The above is an illustrative solution of a computer program of this embodiment. It should be noted that the technical solution of the computer program and the technical solution of the above-mentioned data processing method are of the same concept. For details not described in detail in the technical solution of the computer program, please refer to the description of the technical solution of the above-mentioned data processing method.

[0153] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0154] The computer instructions include computer program code, which may be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0155] It should be noted that for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0156] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0157] The preferred embodiments disclosed above are intended only to help illustrate this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific embodiments described. Obviously, many modifications and variations can be made based on the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that those skilled in the art can better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A data processing method, comprising: Determine a memory allocation region corresponding to the region address and a memory allocator associated with the memory allocation region; Obtaining index data of the memory allocation area and allocation status of the memory allocator, and writing the index data and allocation status to a disk file; Writing the allocation status into the disk file comprises: determining unallocated memory blocks to be released in the memory allocation area according to the allocation status, performing release processing on the memory blocks to be released; updating the allocation status according to the release processing result, and writing the updated allocation status into the disk file; saving the allocation status of each memory block contained in the memory allocation area by using a bitmap, and based on the fact that each memory block is allocated for use, using 1 bit to indicate the allocation status of each memory block being allocated for use, and deleting the bitmap; When the in-memory database including the memory allocation area is restarted, reading the index data and the allocation status from the disk file; Mapping the index data to the memory allocation area according to the area address, and updating the state of the memory allocator according to the allocation state; Upon receiving a memory block release instruction submitted for the memory allocation area, determine the target memory block; detect whether the target allocation status of the target memory block is saved to the disk file; if not, delay processing of the memory release instruction until the target allocation status is written to the disk file, and release the memory block mapped by the target memory block in the target memory allocation area according to the memory block release instruction; if so, release the memory block mapped by the target memory block in the target memory allocation area according to the memory block release instruction.

2. The method according to claim 1, wherein determining the memory allocation area corresponding to the area address comprises: In a case where a backup event corresponding to the memory database is detected, determining at least two initial memory allocation areas in the memory database, the at least two initial memory allocation areas including an index allocation area and a temporary allocation area; The area address is determined according to the backup event, and the index allocation area is selected as the memory allocation area from among the at least two initial memory allocation areas according to the area address.

3. The method according to claim 1, wherein writing the index data to a disk file comprises: determining running index data included in the index data; The running index data in the index data is deleted, and the index data containing the running index data is written into the disk file.

4. The method according to claim 1, wherein obtaining the allocation status of the memory allocator comprises: Creating a target memory allocation area associated with the memory allocation area, and switching the allocation operation of the memory allocation area to the target memory allocation area; The allocation state of the memory allocator is read according to the switching result.

5. The method according to claim 4, wherein the step of reading the allocation state of the memory allocator according to the switching result comprises: Scan the memory allocation area according to the switching result to determine the memory blocks contained in the memory allocation area; Determining allocation information of the memory block according to the memory allocator; The allocation state of the memory allocator is generated by integrating the allocation information of the memory blocks.

6. The method according to claim 1, wherein when the in-memory database containing the memory allocation area is restarted, reading the index data and the allocation status from the disk file comprises: Obtaining the restart configuration information preset for the memory database, and restarting the memory database according to the restart configuration information; Detecting whether the snapshot information of the memory allocator is available according to a restart processing result of the memory database; If so, execute the step of reading the index data and the allocation status in the disk file.

7. The method according to claim 1, wherein updating the state of the memory allocator according to the allocation state comprises: Registering the memory allocation area in the memory allocator, and scanning the allocation status according to the registration result; The allocation information of the memory blocks contained in the memory allocation area is updated according to the scanning result, as a status update of the memory allocator.

8. The method according to claim 1, further comprising: after executing the step of updating the state of the memory allocator according to the allocation state: Scan the memory allocation area by using the memory allocator with updated state to determine unallocated free memory blocks in the memory allocation area; Release processing is performed on the free memory block.

9. A data processing device comprising: a determination module configured to determine a memory allocation region corresponding to a region address and a memory allocator associated with the memory allocation region; an acquisition module configured to acquire index data of the memory allocation area and an allocation status of the memory allocator, and write the index data and the allocation status to a disk file; when the acquisition module is configured to write the allocation status to the disk file, the module is specifically configured to: determine unallocated memory blocks to be released in the memory allocation area according to the allocation status, and perform release processing on the memory blocks to be released; update the allocation status according to the release processing result, and write the updated allocation status to the disk file; The device is further configured to: save the allocation status of each memory block contained in the memory allocation area through a bitmap, and based on the fact that each memory block is allocated for use, use 1 bit to indicate the allocation status of each memory block being allocated for use, and delete the bitmap; a reading module configured to read the index data and the allocation status from the disk file when the memory database containing the memory allocation area is restarted; an updating module configured to map the index data to the memory allocation area according to the area address, and update a state of the memory allocator according to the allocation state; A release processing module is configured to determine a target memory block upon receiving a memory block release instruction submitted for the memory allocation area; detect whether a target allocation state of the target memory block is saved to the disk file; if not, delay processing of the memory release instruction until the target allocation state is written to the disk file, and release the memory block mapped by the target memory block in the target memory allocation area according to the memory block release instruction; if so, release the memory block mapped by the target memory block in the target memory allocation area according to the memory block release instruction.

10. A computing device comprising: memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the method according to any one of claims 1 to 8 are implemented.

11. A computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the steps of the method according to any one of claims 1 to 8.

Images