File storage method and device, electronic equipment and storage medium

Abstract

The invention discloses a file storage method and device, electronic equipment and a storage medium, and relates to the technical field of computers. The method specifically comprises the steps that first data block information in a data file is split into first metadata and first bitmap data, header information in the data file is stored in a first storage unit, the first metadata is stored in a second storage unit, and the first bitmap data is stored in a third storage unit; storing the metadata memory address corresponding to the second storage unit and the first metadata in a metadata hash table, and storing the bitmap data memory address corresponding to the third storage unit and the first bitmap data in a bitmap data hash table; and storing the target block information corresponding to the first data block information into a memory. According to the method and the device, different data contents in the data file are respectively stored, and the different data contents are pointed through the pointers, so that useless data loading in a memory is reduced.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a file storage method, device, electronic equipment and storage medium. Background Art

[0002] A physical transaction is a collection of operations that modify data pages within the database. Each physical transaction corresponds to a log sequence number. When a physical transaction modifies a data page, the system records the log sequence number corresponding to the physical transaction in the header of the data page.

[0003] When storing data files, the data pages that need to be backed up this time need to be determined based on the log sequence number. The current existing method is usually to filter by file. However, when there are fewer data pages that need to be backed up in a data file, filtering by file will load all the data pages in the data file, which will load many useless data pages into the memory, thereby reducing the overall efficiency of data file storage. Therefore, how to refine the filtering granularity, reduce the loading of useless data pages in the memory, and thus improve the data file storage efficiency is currently an urgent problem to be solved. Summary of the invention

[0004] The present invention provides a file storage method, device, electronic device and storage medium to complete the separate storage of different data contents in a data file and realize pointing to different data contents through pointers, thereby reducing useless data loading in the memory.

[0005] According to one aspect of the present invention, there is provided a file storage method, comprising:

[0006] Splitting the first data block information in the data file into first metadata and first bitmap data, wherein the first metadata includes a first key value and a first page log sequence number of a data page in the first data block information, the first key value identifies the first data block information, and the first bitmap data stores bitmap information associated with the data page in the first data block information;

[0007] storing the header information in the data file in a first storage unit, storing the first metadata in a second storage unit, and storing the first bitmap data in a third storage unit;

[0008] The metadata memory address corresponding to the second storage unit is stored in the metadata hash table, and the bitmap data memory address corresponding to the third storage unit is stored in the bitmap data hash table;

[0009] storing the first metadata stored in the second storage unit in a metadata hash table, and storing the first bitmap data stored in the third storage unit in a bitmap data hash table;

[0010] The target block information corresponding to the first data block information is stored in the memory, wherein the target block information includes a first metadata pointer pointing to the first metadata in the metadata hash table, and a first bitmap data pointer pointing to the first bitmap data in the bitmap data hash table.

[0011] According to another aspect of the present invention, there is provided a file storage device, comprising:

[0012] a splitting module, configured to split the first data block information in the data file into first metadata and first bitmap data, wherein the first metadata includes a first key value and a first page log sequence number of a data page in the first data block information, the first key value identifies the first data block information, and the first bitmap data stores bitmap information associated with the data page in the first data block information;

[0013] A first storage module, used for storing the header information in the data file in a first storage unit, storing the first metadata in a second storage unit, and storing the first bitmap data in a third storage unit;

[0014] A second storage module, used to store the metadata memory address corresponding to the second storage unit in a metadata hash table, and store the bitmap data memory address corresponding to the third storage unit in a bitmap data hash table;

[0015] A third storage module, configured to store the first metadata stored in the second storage unit in a metadata hash table, and store the first bitmap data stored in the third storage unit in a bitmap data hash table;

[0016] A fourth storage module is used to store the target block information corresponding to the first data block information in the memory, and the target block information includes a first metadata pointer pointing to the first metadata in the metadata hash table, and a first bitmap data pointer pointing to the first bitmap data in the bitmap data hash table.

[0017] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0018] at least one processor; and

[0019] a memory communicatively connected to the at least one processor; wherein,

[0020] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the file storage method described in any embodiment of the present invention.

[0021] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the file storage method described in any embodiment of the present invention when executed.

[0022] The technical solution of the embodiment of the present invention separates the block information in the data file into metadata and bitmap data, thereby completing the separate storage of different data contents in the data file and improving the storage efficiency of the data file. Metadata pointers and bitmap data pointers are used in the memory to point to the metadata and bitmap data stored in the metadata hash table and the bitmap data hash table, thereby reducing useless data loading in the memory.

[0023] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0025] Figure 1A is a flowchart of a file storage method provided according to Embodiment 1 of the present invention;

[0026] Figure 1B is a structural diagram of first data block information after splitting provided according to Embodiment 1 of the present invention;

[0027] Figure 1C A physical storage structure diagram of a divided data file provided according to the first embodiment of the present invention;

[0028] Figure 1D A memory structure diagram for storing first metadata and first bitmap data provided according to the first embodiment of the present invention;

[0029] Figure 1E This is a metadata hash table structure diagram provided according to the first embodiment of the present invention;

[0030] Figure 1F is a diagram of a bitmap data hash table structure provided according to Embodiment 1 of the present invention;

[0031] Figure 1G is a target block information structure diagram provided according to the first embodiment of the present invention;

[0032] Figure 2 is a flowchart of a bitmap persistence method provided according to Embodiment 2 of the present invention;

[0033] Figure 3 is a schematic diagram of the structure of a file storage device provided according to Embodiment 3 of the present invention;

[0034] Figure 4 It is a block diagram of an electronic device provided according to a fourth embodiment of the present invention. DETAILED DESCRIPTION

[0035] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0036] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0037] Embodiment 1

[0038] Figure 1 is a flow chart of a file storage method provided in Embodiment 1 of the present invention. This embodiment is applicable to the situation where different data contents in a data file are stored separately, and pointers are used to point to different data contents. The method can be executed by a file storage device, which can be implemented in the form of hardware and / or software, and can be configured in an electronic device.

[0039] As shown in FIG1 , the method includes:

[0040] S110. Split the first data block information in the data file into first metadata and first bitmap data, wherein the first metadata includes a first key value and a first page log sequence number of a data page in the first data block information, the first key value identifies the first data block information, and the first bitmap data stores bitmap information associated with the data page in the first data block information.

[0041] In this embodiment, the data file can be understood as a carrier for storing data, which contains multiple data block information. The database system will periodically write the data block information in the memory to the data file. The first data block information can be used to maintain the disk flushing information of all data pages in a data block, and the first data block information is used in the file storage stage. The data page can be understood as the basic unit for storing data in the database, which is usually composed of a storage area of ​​a fixed size. The disk flushing is the operation of writing the data in the memory to the disk to ensure the security and reliability of the data.

[0042] Metadata is information that describes data attributes and is used to record structured information about data, such as tablespace identifiers, file identifiers, data block identifiers, and page log sequence numbers. It only takes up a small amount of storage space and is used to identify data blocks and record statistical information about data pages registered in data blocks. The first metadata is metadata used in the file storage phase. A tablespace consists of one or more data files and is a logical concept in a database. Tablespace identifiers are used to uniquely identify different tablespaces in data block information.

[0043] The bitmap data stores bitmap information, and the bitmap data can be a set of binary bits, and each binary bit can be considered as bitmap information associated with the data page in the first data block. The bitmap information associated with the data page can be considered as information indicating whether the data page has been flushed to disk. For example, each binary bit has two values, 0 or 1, 0 indicates that the corresponding data page has not been flushed to disk, and 1 indicates that the corresponding data page has been flushed to disk, and one of the data pages corresponds to a binary bit in the bitmap data. The first bitmap data is the bitmap data used for the file storage stage. The bitmap information occupies a large amount of memory and is used to record the characteristics of the data.

[0044] The key value can be used to locate the address of the data block information, for example, the combination of the tablespace identifier, the file identifier, and the data block identifier is used as the key value to locate the data block information, wherein the first key value is used to locate the address of the first data block information, which is used in the file storage stage. The page log sequence number can be understood as a unique number assigned by the database to the data page. It is an automatically maintained integer value with a global incrementing feature. The page log sequence number is continuously updated as the data page is modified. The first page log sequence number is the page log sequence number in the file storage stage.

[0045] Specifically, when storing a data file, the first data block information in the data file is first split into the first metadata and the first bitmap data. At this time, the first metadata and the first bitmap data are no longer continuous in physical storage, but are only associated in a logical relationship. The first metadata stores fixed-value attributes such as the first key value composed of the table space identifier, file identifier, and data block identifier, as well as dynamically changing attributes such as the first page log sequence number. The first bitmap data stores the bitmap information associated with the data page in the first data block information.

[0046] Exemplarily, the first data block information in the data file is split into the first metadata and the first bitmap data, such as Figure 1B It is a structural diagram of the first data block information after splitting provided according to the first embodiment of the present invention. The first metadata includes a table space identifier, a file identifier, a data block identifier and a first page log sequence number, the first page log sequence number includes a maximum page log sequence number and a minimum page log sequence number, the minimum page log sequence number is the smallest of the page log sequence numbers of all data pages in the data block, and similarly the maximum page log sequence number is the largest one. The first bitmap data only includes bitmap information.

[0047] S120: Store the header information in the data file in a first storage unit, store the first metadata in a second storage unit, and store the first bitmap data in a third storage unit.

[0048] In this embodiment, the header information can be understood as a specific piece of data located at the beginning of the data file, which is used to uniquely identify a data file. The first storage unit can be understood as a storage unit for storing file header information. The storage unit is the smallest unit for reading and writing data files, and its size can be equal to the size of the data page. Each storage unit is uniquely identified by a storage unit address. The second storage unit is used to store metadata in the data file, and one of the second storage units can store multiple metadata, and the specific number depends on the size of the storage unit and the space occupied by the metadata. The third storage unit is used to store bitmap data in the data file, and one of the third storage units can only store one bitmap data, and the bitmap data corresponds to the metadata in the previous second storage unit.

[0049] Specifically, first, the physical storage structure of the data file is divided into a plurality of storage units of equal size, each storage unit having a unique storage unit address. Then, the header information of the data file is stored in the first storage unit, which may be the first storage unit in the physical storage structure. Then, the first metadata obtained in S110 is stored in the second storage unit, and the first bitmap data obtained in S110 is stored in the third storage unit, wherein the first metadata and the first bitmap data correspond to a set rule, and the set rule may be: determining the first bitmap data corresponding to the first metadata stored in the second storage unit, and sequentially storing the first bitmap data in the third storage unit adjacent to the second storage unit.

[0050] Exemplarily, the physical storage structure of the data file is divided into a plurality of storage units of equal size, such as Figure 1C This is a diagram of a physical storage structure of divided data files provided according to the first embodiment of the present invention. Figure 1C Storage unit 0 is the first storage unit, which is used to store the header information of the data file. Figure 1C Storage unit 1 can be used to store the first metadata, and storage unit 2 can be used to store the first bitmap data corresponding to the first metadata in storage unit 1. Specific storage method: Assuming that one storage unit can store M first metadata, storage unit 1 stores M first metadata, storage unit 2-storage unit 1+M are respectively used to store the first bitmap data corresponding to the first metadata, the order of the first bitmap data matches the order of the first metadata, storage unit 2+M stores M first metadata, and so on.

[0051] Optionally, storing the header information in the data file in a first storage unit, storing the first metadata in a second storage unit, and storing the first bitmap data in a third storage unit includes: dividing the physical storage structure of the data file into multiple storage units of equal size, the storage unit is uniquely identified by a storage unit address, the multiple storage units include one first storage unit, multiple second storage units and multiple third storage units, the first storage unit, the second storage unit and the third storage unit are storage units after the physical storage structure of the data file is divided, and the data file includes multiple first data block information; storing the header information in the first storage unit; storing each of the first metadata as a group according to each set number of first metadata, and storing them in the corresponding second storage unit in sequence; for each group of first metadata, determining the first bitmap data corresponding to each first metadata in the group of first metadata, and storing the determined first bitmap data in sequence in the corresponding third storage unit, the third storage unit corresponding to the group of first metadata and the second storage unit corresponding to the group of first metadata are adjacent storage units.

[0052] Specifically, a data file includes a plurality of first data block information, and the physical storage structure of a data file is divided into a plurality of storage units of equal size, the storage unit size is equal to the data page size, and each storage unit is uniquely identified by a storage unit address. After the division, the physical storage structure of the data file includes a first storage unit, a plurality of second storage units, and a plurality of third storage units. First, the header information of the data file is stored in the first storage unit, and then the plurality of first metadata corresponding to the plurality of first data block information in the data file is grouped according to a set number, and stored in sequence in the plurality of second storage units, wherein the set number depends on the size of the storage unit and the size of the space occupied by the first metadata. For each group of first metadata in the second storage unit, the bitmap data corresponding to each group of first metadata is determined, and the bitmap data is stored in sequence in the third storage unit adjacent to the second storage unit.

[0053] For example, Figure 1DThis is a memory structure diagram for storing first metadata and first bitmap data according to the first embodiment of the present invention. Assuming that one storage unit can store three first metadata, first metadata No. 1, first metadata No. 2 and first metadata No. 3 are stored in a second storage unit, that is, storage unit 1, first bitmap data No. 1 corresponding to first metadata No. 1 is stored in an adjacent third storage unit, that is, storage unit 2, first bitmap data No. 2 corresponding to first metadata No. 2 is stored in storage unit 3, and first bitmap data No. 3 corresponding to first metadata No. 3 is stored in storage unit 4. Similarly, first metadata No. 4, first metadata No. 5 and first metadata No. 6 are stored in another second storage unit, that is, storage unit 5, and the corresponding first bitmap data are stored in an adjacent third storage unit.

[0054] S130. Store the metadata memory address corresponding to the second storage unit in a metadata hash table, and store the bitmap data memory address corresponding to the third storage unit in a bitmap data hash table.

[0055] In this embodiment, the metadata memory address can be understood as the address of the memory unit storing metadata in the memory, which is the same as the address of the second storage unit storing metadata. The metadata hash table can be considered as a hash table for managing metadata stored in the memory unit. The metadata hash table can be used to store metadata and metadata memory addresses, etc. The hash table can be understood as a structure for maintaining data block information in the current database. The memory unit address storing bitmap data in the bitmap data memory address is the same as the address of the third storage unit storing bitmap data. The bitmap data hash table can be considered as a hash table for managing bitmap data stored in the memory unit. The bitmap data hash table can be used to store bitmap data and bitmap data memory addresses, etc.

[0056] Specifically, a metadata hash table and a bitmap data hash table are created in the memory, wherein one hash table can manage multiple memory units, the content in the memory unit corresponds to the content in the storage unit one by one, and the memory unit address corresponds to the storage unit address one by one. The storage unit address corresponding to the second storage unit is determined, and it is used as the metadata memory address and stored in the metadata hash table. Similarly, the storage unit address corresponding to the third storage unit is determined, and it is used as the bitmap data memory address and stored in the bitmap data hash table.

[0057] For example, Figure 1E is a structure of a metadata hash table, in which a unique memory unit address is stored in a memory unit of the metadata hash table, wherein the memory unit address corresponds to the second storage unit address one by one. Figure 1FIt is a structure of a bitmap data hash table. Similarly, a unique memory unit address is stored in a memory unit of the bitmap data hash table, wherein the memory unit address corresponds to the third storage unit address one by one.

[0058] S140: Store the first metadata stored in the second storage unit in a metadata hash table, and store the first bitmap data stored in the third storage unit in a bitmap data hash table.

[0059] Specifically, a memory unit can store multiple first metadata, and the first metadata stored in a memory unit in the metadata hash table corresponds one-to-one to the first metadata stored in the above-mentioned second storage unit. Similarly, the first bitmap data stored in a memory unit in the bitmap data hash table corresponds to the first bitmap data stored in the above-mentioned third storage unit.

[0060] For example, Figure 1E , a unique memory unit address and M first metadata are stored in a memory unit of the metadata hash table, corresponding to the M first metadata stored in a second storage unit, such as Figure 1D The M first metadata stored in the storage unit 1. Figure 1F , a unique memory unit address and a first bitmap data are stored in a memory unit of the bitmap data hash table, corresponding to a first bitmap data stored in a third storage unit, such as Figure 1D A first bitmap data is stored in the storage unit 2.

[0061] S150. Store the target block information corresponding to the first data block information in the memory, wherein the target block information includes a first metadata pointer pointing to the first metadata in the metadata hash table, and a first bitmap data pointer pointing to the first bitmap data in the bitmap data hash table.

[0062] In this embodiment, the target block information can be understood as the information in the first data block information that needs to be stored in the memory. The pointer describes the location of the data in the memory, and the data pointed to can be quickly accessed and operated in the memory through the pointer. The first metadata pointer can be understood as a pointer pointing to the first metadata, which can be used to quickly locate the first metadata. The first bitmap data pointer can be understood as a pointer pointing to the first bitmap data, which can be used to quickly locate the first bitmap data.

[0063] Specifically, the target block information to be stored in the memory is determined according to the attribute field of the first data block information, the attribute field may include the tablespace identifier, file identifier, data block identifier and page log sequence number of the data block information, and the target block information includes a first metadata pointer pointing to the first metadata in the metadata hash table and a first bitmap data pointer pointing to the first bitmap data in the bitmap data hash table, as well as a first key value, a first page log sequence number, etc. Finally, the located target block information is stored in the memory.

[0064] For example, Figure 1G A target block information structure diagram is provided according to the first embodiment of the present invention. The target block information to be stored can be determined according to the attribute field. The metadata pointer of the target block information points to the first metadata stored in the metadata hash table, and the bitmap data pointer points to the first bitmap data stored in the bitmap data hash table.

[0065] The technical solution of the embodiment of the present invention separates the block information in the data file into metadata and bitmap data, thereby completing the separate storage of different data contents in the data file and improving the storage efficiency of the data file. Metadata pointers and bitmap data pointers are used in the memory to point to the metadata and bitmap data stored in the metadata hash table and the bitmap data hash table, thereby reducing useless data loading in the memory.

[0066] Based on the above embodiment, a variant embodiment of the above embodiment is proposed. It should be noted that in order to make the description concise, only the differences from the above embodiment are described in the variant embodiment.

[0067] In one embodiment, the file storage method also includes: searching whether there is matching second data block information in the memory according to the second key value of the data page to be registered; if so, updating the second page log sequence number of the second data block information; if not, creating corresponding second data block information for the data page to be registered in the memory, setting the second key value, second page log sequence number, second metadata pointer and second bitmap data pointer corresponding to the second data block information; allocating second metadata and second bitmap data of the second data block information, pointing the second metadata pointer to the allocated second metadata, and pointing the second bitmap data pointer to the allocated second bitmap data.

[0068] In this embodiment, during the operation of the database, after the data page is flushed, the data block information to which the data page belongs needs to be updated in the memory. This process is called registering the flushing data page. The data page that needs to be registered in this process is the data page to be registered. The second data block information can be used to maintain the flushing information of all data pages in a data block. The second data block information is used in the process of registering the flushing data page in this embodiment. There is no necessary connection between the first data block information and the second data block information.

[0069] The second key value is used to locate the address of the second data block information. The second key value is used in the registration and flushing data page process of this embodiment. There is no necessary connection between the first key value and the second key value. The characteristics of the second page log sequence number are the same as the first page log sequence number. The second page log sequence number is used in the registration and flushing data page process of this embodiment. There is no necessary connection between the first page log sequence number and the second page log sequence number.

[0070] The second metadata is used in the process of registering and flushing data pages in this embodiment, and is used to identify the second data block information and record the statistical information of the data pages in the second data block information. There is no necessary connection between the first metadata and the second metadata. The second bitmap data is used in the process of registering and flushing data pages in this embodiment, and is used to store the bitmap information associated with the data pages in the second data block information. There is no necessary connection between the first bitmap data and the second bitmap data.

[0071] The second metadata pointer can be understood as a pointer to the second metadata, which can be used to quickly locate the second metadata. The second metadata pointer is used in the process of registering and flushing disk data pages in this embodiment. There is no necessary connection between the first metadata pointer and the second metadata pointer. The second bitmap data pointer can be understood as a pointer to the second bitmap data, which can be used to quickly locate the second bitmap data. It is used in the process of registering and flushing disk data pages in this embodiment. There is no necessary connection between the first bitmap data pointer and the second bitmap data pointer.

[0072] Specifically, according to the second key value composed of the table space identifier, file identifier, and data block identifier in the data page to be registered, it is searched whether there is matching second data block information in the memory, that is, whether there is second data block information whose table space identifier, file identifier, and data block identifier completely match. If there is, the page log sequence number of the current data page to be registered is read, and the page log sequence number is updated to the second page log sequence number of the second data block information; if not, the corresponding second data block information is created in the memory for the data page to be registered, and firstly, the second key value composed of the table space identifier, file identifier, and data block identifier, the second page log sequence number, the second metadata pointer, and the second bitmap data pointer are set for the second data block information, and according to the above method, the second metadata pointer is made to point to the allocated second metadata, and the second bitmap data pointer is made to point to the allocated second bitmap data.

[0073] Exemplarily, according to the second key value of the data page to be registered, it is searched whether there is a matching second data block information in the memory. If it exists, the page log sequence number of the current data page to be registered is read: if the page log sequence number is less than the minimum page log sequence number of the second data block information or the minimum page log sequence number is -1, the minimum page log sequence number of the second data block information is set to the page log sequence number of the current data page to be registered; if the page log sequence number is greater than the maximum page log sequence number of the second data block information or the maximum page log sequence number is 0, the maximum page log sequence number of the second data block information is set to the page log sequence number of the current data page to be registered. If it does not exist, the corresponding second data block information is created in the memory for the data page to be registered, and the second key value corresponding to the second data block information, the minimum page log sequence number is set to an invalid value -1, the maximum page log sequence number is set to 0, the second metadata pointer and the second bitmap data pointer are set, and according to the above method, the second metadata pointer is made to point to the allocated second metadata, and the second bitmap data pointer is made to point to the allocated second bitmap data.

[0074] Optionally, the allocation of the second metadata and the second bitmap data of the second data block information, pointing the second metadata pointer to the allocated second metadata, and pointing the second bitmap data pointer to the allocated second bitmap data, includes: if there are excess memory units in the metadata hash table, storing the second metadata of the second data block information in the memory unit; otherwise, allocating a new memory unit in the metadata hash table, generating a unique metadata memory address, and setting the second metadata pointer of the second data block information to point to the second metadata; allocating a new memory unit in the bitmap data hash table, generating a unique bitmap data memory address, and setting the second bitmap data pointer of the second data block information to point to the second bitmap data.

[0075] Specifically, according to the method in the above embodiment, the second data block information is divided into the second metadata and the second bitmap data. First, determine whether there are redundant memory units in the metadata hash table. If so, store the second metadata in the existing memory unit; if there is no memory unit in the metadata hash table or the memory units therein have been fully used, allocate a new memory unit in the metadata hash table, generate a unique metadata memory address for the memory unit, register the second metadata in the newly created memory unit, and finally point the second metadata pointer to the second metadata. Since a memory unit in the bitmap data hash table can only store one second bitmap data, allocate a new memory unit in the bitmap data hash table, generate a unique bitmap data memory address for the memory unit, register the second bitmap data in the newly created memory unit, and finally point the second bitmap data pointer to the second bitmap data.

[0076] Exemplarily, first determine whether there are redundant memory units in the metadata hash table, if so, store the second metadata in the existing memory unit; if there is no memory unit in the metadata hash table or the memory units therein have been fully used, allocate a new memory unit in the metadata hash table, and generate a unique metadata memory address for the memory unit, the generation method is: if there is no memory unit in the metadata hash table, set the metadata memory unit address to 1; otherwise, obtain the maximum address value in the metadata hash table and the bitmap data hash table memory address, and the metadata memory address is the maximum address value plus one. Then register the second metadata in the newly created memory unit, the registered content includes the table space identifier, file identifier, data block identifier, second page log sequence number, etc., and finally point the second metadata pointer to the second metadata. Allocate a new memory unit in the bitmap data hash table, and generate a unique bitmap data memory address for the memory unit, the generation method is: obtain the maximum address value in the metadata hash table and the bitmap data hash table memory unit address, and the metadata memory address is the maximum address value plus one. Then register the second bitmap data in the newly created memory unit, and finally point the second bitmap data pointer to the second bitmap data.

[0077] Embodiment 2

[0078] Figure 2 This is a flowchart of a bitmap persistence method provided in Embodiment 2 of the present invention. The bitmap persistence method provided in this embodiment is implemented by the file storage method of the above embodiment. Figure 2 As shown, the method includes:

[0079] S210: Write the third bitmap data in the bitmap data hash table into the file to be stored.

[0080] In this embodiment, the database periodically writes the data block information in the memory to the data file, and the file to be stored is the data file to which the data block information is to be written. The third bitmap data is used in the bitmap persistence process of this embodiment, and is used to store the bitmap information associated with the data page in the third data block information. There is no necessary connection between the first bitmap data, the second bitmap data and the third bitmap data. The third data block information can be used to maintain the disk flushing information of all data pages in a data block. The third data block information is used in the bitmap persistence process of this embodiment. There is no necessary connection between the first data block information, the second data block information and the third data block information.

[0081] Specifically, the process of writing data block information into a data file by the in-memory database is called bitmap persistence. To realize bitmap persistence, it is first necessary to divide the third data block information into third metadata and third bitmap data according to the file storage method of the above embodiment, and write the third metadata and the third bitmap data into the metadata hash table and the memory data hash table respectively. Then, all memory units in the bitmap data hash table are traversed to determine whether the data in the memory is updated. If an update occurs, the content of the memory unit needs to be written into the data file, that is, the third bitmap data of the bitmap data hash table in the memory unit is written into the file to be stored.

[0082] Exemplarily, the third bitmap data in the bitmap data hash table is written into the file to be stored. The matching offset position on the data file can be found based on the memory unit address. The offset position is the product of the memory unit address and the storage unit size. The third bitmap data in the bitmap data hash table is written into the file to be stored based on the offset address.

[0083] S220: Update the third metadata in the metadata hash table according to the third page log sequence number of the third data block information in the memory, and update the update mark.

[0084] In this embodiment, the characteristics of the third page log sequence number are the same as those of the first page log sequence number. The third page log sequence number is used in the bitmap persistence process of this embodiment. There is no necessary connection between the first page log sequence number, the second page log sequence number and the third page log sequence number. The third metadata is used in the bitmap persistence process of this embodiment to identify the third data block information and record the statistical information of the data page in the third data block information. There is no necessary connection between the first metadata, the second metadata and the third metadata.

[0085] Specifically, the third data block information in the memory is traversed, the third metadata in the metadata hash table is synchronously adjusted according to the third page log sequence number of the third data block information, and the update count of the memory unit update mark corresponding to the third metadata is updated.

[0086] Optionally, updating the third metadata in the metadata hash table according to the third page log sequence number of the third data block information in the memory, and updating the update mark, includes: traversing the third data block information in the memory, and if the update mark in the third data block information indicates that an update has occurred, searching the metadata hash table for the corresponding third metadata according to the third metadata pointer in the third data block information; synchronously adjusting the third page log sequence number included in the third metadata according to the third page log sequence number of the third data block information, and updating the update mark of the memory unit where the third metadata is located.

[0087] Specifically, the third data block information in the memory is traversed to find the third data block information whose update mark indicates that an update has occurred, and the corresponding third metadata is found according to the third metadata pointer of the third data block information. Then, the maximum page log sequence number and the minimum page log sequence number included in the third metadata are synchronously adjusted according to the maximum page log sequence number and the minimum page log sequence number of the third data block information, and the update mark of the memory unit where the third metadata is located is updated.

[0088] Exemplarily, when the update mark of the memory unit where the third metadata is located is updated, the update count on the update mark may be increased by 1.

[0089] S230: Write the third metadata in the metadata hash table into the file to be stored.

[0090] Specifically, all memory units in the metadata hash table are traversed to determine whether the data in the memory is updated. If an update occurs, the content of the memory unit needs to be written into the data file, and the third metadata in the metadata hash table is written into the file to be stored.

[0091] Optionally, writing the third metadata in the metadata hash table into the file to be stored includes: traversing each memory unit in the metadata hash table; if the update flag of the memory unit indicates that an update has occurred, writing the third metadata in the memory unit into the file to be stored until all memory units in the metadata hash table are traversed.

[0092] Specifically, all memory units in the metadata hash table are traversed to determine that an updated memory unit has been updated, indicating that bitmap persistence is required, and the third metadata of the metadata hash table in the memory unit is written into the file to be stored.

[0093] Exemplarily, the third metadata in the metadata hash table is written into the file to be stored. The matching offset position on the data file can be found based on the memory unit address. The offset position is the product of the memory unit address and the storage unit size. The third metadata in the metadata hash table is written into the file to be stored based on the offset address.

[0094] S240: Update header information of the file to be stored.

[0095] Specifically, at this time, the third metadata and the third bitmap data have been stored in the file to be stored, and finally the header information of the file to be stored is updated to complete the bitmap persistence.

[0096] The technical solution of the embodiment of the present invention divides the data block information that needs to be written into the file to be stored into metadata and bitmap data. Through the file storage method of the above embodiment, the bitmap data stored in the bitmap data hash table and the metadata stored in the metadata hash table are respectively written into the file to be stored, thereby realizing bitmap persistence and preventing data loss in the memory.

[0097] Embodiment 3

[0098] Figure 3 This is a schematic diagram of the structure of a file storage device provided by Embodiment 3 of the present invention. Figure 3 As shown, the device comprises:

[0099] A splitting module 310 is used to split the first data block information in the data file into first metadata and first bitmap data, wherein the first metadata includes a first key value and a first page log sequence number of a data page in the first data block information, the first key value identifies the first data block information, and the first bitmap data stores bitmap information associated with the data page in the first data block information;

[0100] A first storage module 320, configured to store the header information in the data file in a first storage unit, store the first metadata in a second storage unit, and store the first bitmap data in a third storage unit;

[0101] A second storage module 330, configured to store the metadata memory address corresponding to the second storage unit in a metadata hash table, and store the bitmap data memory address corresponding to the third storage unit in a bitmap data hash table;

[0102] A third storage module 340, configured to store the first metadata stored in the second storage unit in a metadata hash table, and store the first bitmap data stored in the third storage unit in a bitmap data hash table;

[0103] The fourth storage module 350 is used to store the target block information corresponding to the first data block information in the memory, and the target block information includes a first metadata pointer pointing to the first metadata in the metadata hash table, and a first bitmap data pointer pointing to the first bitmap data in the bitmap data hash table.

[0104] The file storage device provided by the present invention splits the block information in the data file into metadata and bitmap data through a splitting module, and makes the metadata pointer and the bitmap data pointer point to the metadata and bitmap data stored in the metadata hash table and the bitmap data hash table through the cooperation of the first storage module, the second storage module, the third storage module and the fourth storage module. The separate storage of different data contents in the data file is completed, the storage efficiency of the data file is improved, and different data contents are pointed to by pointers, which reduces the useless data loading in the memory.

[0105] In one embodiment, the first storage module 320 is specifically used to: divide the physical storage structure of the data file into multiple storage units of equal size, the storage unit is uniquely identified by a storage unit address, the multiple storage units include one first storage unit, multiple second storage units and multiple third storage units, the first storage unit, the second storage unit and the third storage unit are storage units after the physical storage structure of the data file is divided, and the data file includes multiple first data block information; store the header information in the first storage unit; group each of the first metadata into a group according to each set number of first metadata, and store them in the corresponding second storage unit in sequence; for each group of first metadata, determine the first bitmap data corresponding to each first metadata in the group of first metadata, and store the determined first bitmap data in the corresponding third storage unit in sequence, the third storage unit corresponding to the group of first metadata and the second storage unit corresponding to the group of first metadata are adjacent storage units.

[0106] In one embodiment, the file storage device further includes a registration module, including:

[0107] A searching unit, used for searching whether there is matching second data block information in the memory according to the second key value of the data page to be registered;

[0108] an updating unit, configured to update the second page log sequence number of the second data block information if it exists;

[0109] A creation unit is used to create corresponding second data block information for the data page to be registered in the memory if it does not exist, set the second key value, the second page log sequence number, the second metadata pointer and the second bitmap data pointer corresponding to the second data block information; allocate the second metadata and the second bitmap data of the second data block information, point the second metadata pointer to the allocated second metadata, and point the second bitmap data pointer to the allocated second bitmap data.

[0110] In one embodiment, a creation unit is specifically used to: if there are excess memory units in the metadata hash table, store the second metadata of the second data block information in the memory unit; otherwise, allocate a new memory unit in the metadata hash table, generate a unique metadata memory address, and set the second metadata pointer of the second data block information to point to the second metadata; allocate a new memory unit in the bitmap data hash table, generate a unique bitmap data memory address, and set the second bitmap data pointer of the second data block information to point to the second bitmap data.

[0111] In one embodiment, the file storage device further includes a writing module, including:

[0112] A bitmap data writing unit, used for writing the third bitmap data in the bitmap data hash table into a file to be stored;

[0113] A data updating unit, configured to update the third metadata in the metadata hash table according to the third page log sequence number of the third data block information in the memory, and to update the update mark;

[0114] A metadata writing unit, used for writing the third metadata in the metadata hash table into the file to be stored;

[0115] The header information updating unit is used to update the header information of the file to be stored.

[0116] In one embodiment, the data update unit is specifically used to: traverse the third data block information in the memory, and if the update mark in the third data block information indicates that an update has occurred, search the corresponding third metadata from the metadata hash table according to the third metadata pointer in the third data block information; synchronously adjust the third page log sequence number included in the third metadata according to the third page log sequence number of the third data block information, and update the update mark of the memory unit where the third metadata is located.

[0117] In one embodiment, the metadata writing unit is specifically used to: traverse each memory unit in the metadata hash table; if the update flag of the memory unit indicates that an update has occurred, write the third metadata in the memory unit to the file to be stored until all memory units in the metadata hash table are traversed.

[0118] The file storage device provided in the embodiment of the present invention can execute the file storage method provided in any embodiment of the present invention, completes the storage of files through the mutual cooperation and collaboration between the modules, and has the functional modules and beneficial effects corresponding to the execution method.

[0119] Embodiment 4

[0120] According to an embodiment of the present invention, the present invention also provides an electronic device and a computer-readable storage medium.

[0121] Figure 4 It is a block diagram of an electronic device provided according to Embodiment 4 of the present invention, and the electronic device can implement the file storage method described in the embodiment of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0122] like Figure 4 As shown, the electronic device 410 includes at least one processor 411, and a memory connected to the at least one processor 411 in communication, such as a read-only memory (ROM) 412, a random access memory (RAM) 413, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 411 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 412 or the computer program loaded from the storage unit 418 to the random access memory (RAM) 413. In the RAM 413, various programs and data required for the operation of the electronic device 410 can also be stored. The processor 411, the ROM 412, and the RAM 413 are connected to each other via a bus 414. An input / output (I / O) interface 415 is also connected to the bus 414.

[0123] A number of components in the electronic device are connected to the I / O interface 415, including: an input unit 416, such as a keyboard, a mouse, etc.; an output unit 417, such as various types of displays, speakers, etc.; a storage unit 418, such as a disk, an optical disk, etc.; and a communication unit 419, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 419 allows the electronic device to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0124] The processor 411 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 411 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 411 performs the various methods and processes described above, such as a file storage method.

[0125] In some embodiments, the file storage method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 418. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 410 via the ROM 412 and / or the communication unit 419. When the computer program is loaded into the RAM 413 and executed by the processor 411, one or more steps of the file storage method described above may be performed. Alternatively, in other embodiments, the processor 411 may be configured to perform the file storage method in any other appropriate manner (e.g., by means of firmware).

[0126] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0127] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0128] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0129] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0130] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0131] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0132] The technical solution of the embodiment of the present invention, through a file storage method, device, electronic device and storage medium, splits the block information in the data file into metadata and bitmap data, completes the separate storage of different data contents in the data file, and improves the storage efficiency of the data file. Metadata pointers and bitmap data pointers are used in the memory to point to the metadata and bitmap data stored in the metadata hash table and the bitmap data hash table, which reduces the useless data loading in the memory and improves the storage efficiency of the data file.

[0133] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0134] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A file storage method, characterized in that: include: Splitting the first data block information in the data file into first metadata and first bitmap data, wherein the first metadata includes a first key value and a first page log sequence number of a data page in the first data block information, the first key value identifies the first data block information, and the first bitmap data stores bitmap information associated with the data page in the first data block information; storing the header information in the data file in a first storage unit, storing the first metadata in a second storage unit, and storing the first bitmap data in a third storage unit; The metadata memory address corresponding to the second storage unit is stored in the metadata hash table, and the bitmap data memory address corresponding to the third storage unit is stored in the bitmap data hash table; storing the first metadata stored in the second storage unit in a metadata hash table, and storing the first bitmap data stored in the third storage unit in a bitmap data hash table; The target block information corresponding to the first data block information is stored in the memory, wherein the target block information includes a first metadata pointer pointing to the first metadata in the metadata hash table, and a first bitmap data pointer pointing to the first bitmap data in the bitmap data hash table.

2. The method according to claim 1, characterized in that: The step of storing the header information in the data file in a first storage unit, storing the first metadata in a second storage unit, and storing the first bitmap data in a third storage unit includes: Dividing the physical storage structure of the data file into a plurality of storage units of equal size, wherein the storage unit is uniquely identified by a storage unit address, wherein the plurality of storage units include one first storage unit, a plurality of second storage units, and a plurality of third storage units, wherein the first storage unit, the second storage unit, and the third storage unit are storage units after the physical storage structure of the data file is divided, and the data file includes a plurality of first data block information; storing the header information in the first storage unit; storing the first metadata in the corresponding second storage unit in sequence according to a set number of first metadata as a group; For each group of first metadata, the first bitmap data corresponding to each first metadata in the group of first metadata is determined, and the determined first bitmap data are stored in the corresponding third storage unit in sequence, and the third storage unit corresponding to the group of first metadata and the second storage unit corresponding to the group of first metadata are adjacent storage units.

3. The method according to claim 1, characterized in that Also includes: Searching whether there is matching second data block information in the memory according to the second key value of the data page to be registered; If it exists, updating the second page log sequence number of the second data block information; If not, creating corresponding second data block information for the data page to be registered in the memory, and setting a second key value, a second page log sequence number, a second metadata pointer, and a second bitmap data pointer corresponding to the second data block information; Allocate second metadata and second bitmap data of the second data block information, point the second metadata pointer to the allocated second metadata, and point the second bitmap data pointer to the allocated second bitmap data.

4. The method according to claim 3, characterized in that The allocating the second metadata and the second bitmap data of the second data block information, pointing the second metadata pointer to the allocated second metadata, and pointing the second bitmap data pointer to the allocated second bitmap data, comprises: If there are extra memory units in the metadata hash table, storing the second metadata of the second data block information in the memory units; Otherwise, a new memory unit is allocated in the metadata hash table, and a unique metadata memory address is generated, and the second metadata pointer of the second data block information is set to point to the second metadata; A new memory unit is allocated in the bitmap data hash table, and a unique bitmap data memory address is generated, and the second bitmap data pointer of the second data block information is set to point to the second bitmap data.

5. The method according to claim 1, characterized in that Also includes: Writing the third bitmap data in the bitmap data hash table into the file to be stored; Update the third metadata in the metadata hash table according to the third page log sequence number of the third data block information in the memory, and update the update mark; Writing the third metadata in the metadata hash table into the file to be stored; Update the header information of the file to be stored.

6. The method according to claim 5, characterized in that The updating of the third metadata in the metadata hash table according to the third page log sequence number of the third data block information in the memory, and updating the update mark, includes: Traversing the third data block information in the memory, if the update mark in the third data block information indicates that an update has occurred, searching the metadata hash table for the corresponding third metadata according to the third metadata pointer in the third data block information; The third page log sequence number included in the third metadata is synchronously adjusted according to the third page log sequence number of the third data block information, and the update mark of the memory unit where the third metadata is located is updated.

7. The method according to claim 5, characterized in that The step of writing the third metadata in the metadata hash table into the file to be stored includes: Traversing each memory cell in the metadata hash table; If the update mark of the memory unit indicates that an update has occurred, the third metadata in the memory unit is written into the file to be stored until all memory units in the metadata hash table are traversed.

8. A file storage device, characterized in that: include: a splitting module, configured to split the first data block information in the data file into first metadata and first bitmap data, wherein the first metadata includes a first key value and a first page log sequence number of a data page in the first data block information, the first key value identifies the first data block information, and the first bitmap data stores bitmap information associated with the data page in the first data block information; A first storage module, used for storing the header information in the data file in a first storage unit, storing the first metadata in a second storage unit, and storing the first bitmap data in a third storage unit; A second storage module, used to store the metadata memory address corresponding to the second storage unit in a metadata hash table, and store the bitmap data memory address corresponding to the third storage unit in a bitmap data hash table; A third storage module, configured to store the first metadata stored in the second storage unit in a metadata hash table, and store the first bitmap data stored in the third storage unit in a bitmap data hash table; A fourth storage module is used to store the target block information corresponding to the first data block information in the memory, and the target block information includes a first metadata pointer pointing to the first metadata in the metadata hash table, and a first bitmap data pointer pointing to the first bitmap data in the bitmap data hash table.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the file storage method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the file storage method according to any one of claims 1 to 7 when executed.

Images