Data storage method, device, computer equipment and storage medium
By obtaining the memory usage of memory blocks, performing data reorganization and generating verification data in the data storage method, the file corruption or data loss caused by untimely data writeback is solved, and the integrity of data writeback is achieved and reliability.
Patent Information
- Application Number
- CN202010676941.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-07-14
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2040-07-14
AI Technical Summary
In the prior art, data write back may not be timely, resulting in problems such as file corruption or data loss when the device is powered off or restarted.
By obtaining the memory usage of the memory block, data reorganization is performed when the threshold is reached, data is determined to be written back, and a file header for verification data is written to the target file is generated, and system functions are called in time to synchronize memory data with file data. Ensure the integrity of data write back to the file.
Improves the integrity of the writeback data written to the file and reduces the corruption and loss rate during the data writeback process.
Smart Images

Figure CN113934362B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a data storage method, device, computer equipment and storage medium. Background Art
[0002] With the rapid development of Internet technology, a large amount of data can be processed through various applications and operating systems to achieve various functions. In data processing, I / O operations on memory and disk files, i.e., read operations and / or write operations, are often involved.
[0003] In the current method, data is usually written to the memory for caching first, and then written back to the disk file to improve data read and write performance. However, in this method, the operating system usually determines when to write back the data in the memory-mapped file, and the data may not be written back in time. If the device is powered off or restarted, it may cause file damage or data loss. Summary of the invention
[0004] Based on this, it is necessary to provide a data storage method, device, computer equipment and storage medium to address the above technical problems, which can promptly and effectively write the data to be written back into the file, thereby improving the integrity of the write-back data in the written file, and thus effectively reducing the damage rate during the data write-back process.
[0005] A data storage method, the method comprising:
[0006] Get the memory usage of the memory block;
[0007] When the memory occupancy of the memory block reaches a memory threshold, reorganizing the data of the memory block to determine the data to be written back in the memory block;
[0008] Obtaining verification data corresponding to the data to be written back;
[0009] Writing the verification data into the file header of the target file, and immediately calling a system function for synchronizing the memory data with the file data, and writing the data to be written back to the target file;
[0010] The verification data in the file header is used to perform data integrity verification when reading data in the target file.
[0011] A data storage device, comprising:
[0012] A data acquisition module is used to obtain the memory usage of a memory block;
[0013] A data reorganization module, used for reorganizing the data of the memory block when the memory occupancy of the memory block reaches a memory threshold, so as to determine the data to be written back in the memory block;
[0014] The data acquisition module is also used to acquire verification data corresponding to the data to be written back;
[0015] A data write-back module is used to write the verification data into the file header of the target file, and immediately call the system function used to synchronize the memory data with the file data to write the data to be written back to the target file; wherein the verification data in the file header is used to perform data integrity verification when reading the data in the target file.
[0016] In one of the embodiments, the data acquisition module is also used to obtain a data write request triggered by the application when the application writes data to the memory block; the data write request is used to request to write data to the memory block based on a memory mapping file; according to the data write request, the memory mapping file is parsed to obtain the memory occupancy of the memory block.
[0017] In one of the embodiments, the memory data in the memory block includes a key-value pair, and the key-value pair includes a keyword and a corresponding value; the data reorganization module is also used to deduplicate the key-value pairs in the memory block based on the keyword of the key-value pairs in the memory block to obtain valid data in the memory block; and determine the valid data as the data to be written back in the memory block.
[0018] In one of the embodiments, the data reorganization module is also used to obtain operation records of memory data in the memory block; based on the operation records, filter out the latest value of the value corresponding to the keyword of the key-value pair in the memory block after the operation from the memory data; and determine the keyword and the corresponding latest value as valid data in the memory block.
[0019] In one of the embodiments, the data acquisition module is also used to determine the memory occupancy value corresponding to the data to be written back; encode the data to be written back to obtain a verification code corresponding to the data to be written back; and generate verification data corresponding to the data to be written back based on the memory occupancy value and the verification code.
[0020] In one of the embodiments, the data write-back module is also used to call a system function for synchronizing memory data with file data immediately after successfully writing the verification data into the file header of the target file; determine function parameters corresponding to the system function based on the data to be written back in the memory block; and synchronize the data to be written back in the memory block through the system function and the corresponding function parameters so as to write the data to be written back to the target file.
[0021] In one of the embodiments, the data write-back module is also used to determine the data sequence and mapping area of the data to be written back in the memory block based on the memory mapping file; through the system function and the corresponding function parameters, the data to be written back is synchronized to the storage location corresponding to the mapping area in the target file according to the data sequence.
[0022] In one of the embodiments, the data storage device also includes a data cache module, which is used to obtain a data read request triggered by the application when the application reads data from the target file; according to the data read request, the file data corresponding to the data read request in the target file is cached to the memory block based on the memory mapping file.
[0023] In one of the embodiments, the data storage device also includes a data writing module, which is used to obtain a write lock of the memory-mapped file through a key-value pair storage component; based on the write lock of the memory-mapped file, write file data into the memory block, and release the write lock when the file data writing is completed.
[0024] In one of the embodiments, the key-value pair storage component is a cross-process storage component based on memory mapping, which is used to simultaneously access the key-value pairs corresponding to each running process to read and process the accessed key-value pairs.
[0025] In one of the embodiments, the data storage device also includes a data reading module, which is used to obtain a data reading request triggered by the application when the application reads data from the target file again; determine the data to be accessed according to the data reading request, and obtain verification data corresponding to the data to be accessed from the file header of the target file; based on the verification data, perform data integrity verification on the data to be accessed; when the data integrity verification passes, cache the data to be accessed in the target file to the memory block, and read the data to be accessed from the memory block.
[0026] In one of the embodiments, the data reading module is also used to read the key-value pairs corresponding to the data to be accessed in the target file through the data storage component; cache the key-value pairs corresponding to the data to be accessed into the hash table of the memory block according to the memory sequence of the memory block; and read the key-value pairs corresponding to the data to be accessed from the hash table of the memory block based on the data reading request.
[0027] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:
[0028] Get the memory usage of the memory block;
[0029] When the memory occupancy of the memory block reaches a memory threshold, reorganizing the data of the memory block to determine the data to be written back in the memory block;
[0030] Obtaining verification data corresponding to the data to be written back;
[0031] Writing the verification data into the file header of the target file, and immediately calling a system function for synchronizing the memory data with the file data, and writing the data to be written back to the target file;
[0032] The verification data in the file header is used to perform data integrity verification when reading data in the target file.
[0033] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:
[0034] Get the memory usage of the memory block;
[0035] When the memory occupancy of the memory block reaches a memory threshold, reorganizing the data of the memory block to determine the data to be written back in the memory block;
[0036] Obtaining verification data corresponding to the data to be written back;
[0037] Writing the verification data into the file header of the target file, and immediately calling a system function for synchronizing the memory data with the file data, and writing the data to be written back to the target file;
[0038] The verification data in the file header is used to perform data integrity verification when reading data in the target file.
[0039] The above-mentioned data storage method, device, computer equipment and storage medium obtain the memory usage of the memory block when the application needs to write data into the memory; when the memory usage of the memory block reaches the memory threshold, the memory block is reorganized to determine the data to be written back in the memory block and determine the verification data corresponding to the data to be written back. The verification data is further written into the file header of the target file for data integrity verification when reading the data in the target file to ensure the integrity of the data. By instantly calling the system function for synchronizing the memory data with the file data, the data to be written back can be written into the file instantly and effectively, thereby improving the integrity of the write-back data written into the file, and further effectively reducing the damage rate and loss rate of the data during the write-back process. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Figure 1An application environment diagram of a data storage method in one embodiment;
[0041] Figure 2 A schematic diagram of a flow chart of a data storage method in one embodiment;
[0042] Figure 3 A schematic diagram of storing key values in a memory block in one embodiment;
[0043] Figure 4 A schematic diagram of a flow chart of a data storage method in another embodiment;
[0044] Figure 5 A schematic flow chart of the steps of reading data in one embodiment;
[0045] Figure 6 It is a flowchart of a data storage method in a specific embodiment;
[0046] Figure 7 is a structural block diagram of a data storage device in one embodiment;
[0047] Figure 8 is a structural block diagram of a data storage device in another embodiment;
[0048] Fig. 9 is a structural block diagram of a data storage device in yet another embodiment;
[0049] Fig.10 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION
[0050] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0051] The data storage method provided in this application can be applied to Figure 1 In the application environment shown in the figure, Figure 1The application environment in is based on the operating environment in the computer device. The computer device includes a disk and a memory. The disk includes a data storage structure corresponding to the disk file, and the memory includes a memory block based on memory mapping. When the application needs to write data to the memory, the operating system obtains the memory occupancy of the memory block; when the memory occupancy of the memory block reaches the memory threshold, the operating system reorganizes the data of the memory block to determine the data to be written back in the memory block and determines the verification data corresponding to the data to be written back. The operating system further writes the verification data to the file header of the target file to ensure the integrity of the data. By instantly calling the system function for synchronizing the memory data with the file data, the data to be written back is instantly written back to the target file. Among them, the computer device can be but is not limited to various personal computers, laptops, smart phones, tablets and portable wearable devices.
[0052] In one embodiment, Figure 2 As shown, a data storage method is provided, which is described by taking the method applied to a computer device as an example, and includes the following steps:
[0053] S202, obtaining the memory usage of the memory block.
[0054] Among them, memory is one of the important components in the computer, also known as internal memory and main memory, and is the bridge for communication between external memory and CPU (Central Processing Unit). The operation of memory also determines the stable operation of computer equipment. Memory block is the memory space in the memory. Among them, memory usage refers to the current occupied capacity in the memory block.
[0055] All programs in a computer device are run in memory, which is used to temporarily store the calculation data in the CPU and the data exchanged with external storage devices such as hard disks. When the computer device is running, the operating system will transfer the data to be calculated from the memory to the CPU for calculation. When the calculation is completed, the CPU will transmit the result. When the computer device is running an application, the operating system of the computer device will write the required data into the memory. Specifically, the application running in the computer device applies to the operating system for the memory space of the mapped file and writes the required data into the mapped memory space. The application is responsible for writing data into the memory, and the operating system completes the write back from the memory to the file to achieve the effect of permanent storage.
[0056] When a computer device writes data to a memory block, it first obtains the memory occupancy of the memory block to determine whether the capacity of the memory block can guarantee successful data writing.
[0057] S204: When the memory usage of the memory block reaches the memory threshold, reorganize the data of the memory block to determine the data to be written back in the memory block.
[0058] Among them, writeback refers to updating the value of a multidimensional dataset cell, member, or member attribute. The operating system and the application on the platform need to write temporary data to the disk when running. In the write-back state, data is written to the disk only when it is to be cleared from the cache in the memory. As the data read from the main memory increases, it is necessary to write data back from the cache to the disk file, and write the data updated during the operation to the cache. The data to be written back refers to the part of the data in the memory block that needs to be written back to the disk file.
[0059] For example, when there is insufficient space in the memory block to write new data, it is determined that the memory occupancy of the memory block reaches the memory threshold, and the computer device reorganizes the memory block to determine the data to be written back in the memory block, so that there is enough space in the memory block to write new memory data.
[0060] Specifically, after the computer device obtains the memory usage of the memory block, when it is identified that the memory usage of the memory block reaches the memory threshold, it means that the data in the memory block needs to be written back to free up sufficient memory space in the memory block for writing new data. The computer device reorganizes the data in the memory block, and then determines the data to be written back in the memory block based on the reorganized data. The data to be written back can be all the reorganized memory data in the memory block, or it can be a specified part of the memory data. For example, the part of the memory data that needs to be written back can be specified based on the data capacity or storage time point.
[0061] In one embodiment, after the computer device reorganizes the memory data in the memory block, the memory usage of the memory block still reaches the memory threshold, indicating that the memory space of the memory block is still insufficient. The computer device then doubles the memory capacity of the memory block until the memory space of the memory block is sufficient to write new data.
[0062] S206, obtaining verification data corresponding to the data to be written back.
[0063] Data verification is a verification operation performed to ensure the integrity of data. Usually, a verification value is calculated for the original data using a specified algorithm. The receiver uses the same algorithm to calculate the verification value once. If the verification values calculated twice are the same, it means that the data is complete. For example, data verification can include parity check, CRC (Cyclic Redundancy Check), MD5 check, data signature and other verification methods.
[0064] The verification data corresponding to the data to be written back refers to the verification data obtained after the data to be written back is processed. Each item of the data to be written back includes a corresponding verification data to perform data integrity verification on each specific data.
[0065] The verification data corresponding to the data to be written back may only include the verification code of the data to be written back. In another implementation, the verification data may also include the memory size and the verification code of the data to be written back.
[0066] Specifically, the computer device can obtain the verification code corresponding to the data to be written back by performing data verification calculation on the data to be written back, and use the verification code obtained by calculation as the verification data of the data to be written back. Therefore, when the part of the data to be written back is read later, the data integrity of the data to be written back can be effectively verified.
[0067] S208, write the verification data into the file header of the target file, and immediately call the system function used to synchronize the memory data with the file data, and write the data to be written back to the target file; wherein, the verification data in the file header is used to perform data integrity verification when reading the data in the target file.
[0068] Among them, the target file refers to the disk file in the computer device, which is a data storage structure that organizes, stores and names file data. There is a corresponding file system in the computer system. The file system is the format for storing information on the hard disk, which stipulates various standards and mechanisms for computer devices to operate and process files. The information stored in the disk file will not be affected by factors such as power outages.
[0069] Files in the disk are usually stored in the form of files, which usually include file headers and block table structures. The total number of data in the block table structure is specified by the data in the file header. The data in the file header defines the alignment value of the disk block, which is used to map the storage location of the data in the file. Each block is stored from an offset position that is a multiple of the alignment value.
[0070] The system function for synchronizing memory data with file data refers to a function that can be called to make the disk file content consistent with the content in the shared memory area, that is, to achieve synchronization of memory data in the memory with file data in the disk file. For example, the system function can be a refresh change function, that is, an msync() function.
[0071] Specifically, after the computer device obtains the verification data of the data to be written back, it first writes the verification data corresponding to the data to be written back into the file header of the target file in the disk. When the verification data is successfully written into the file header of the target file, the computer device immediately calls the system function for synchronizing the memory data with the file data, and synchronizes the data to be written back in the memory block to the target file on the disk through the system function, so that the data to be written back can be written back to the target file immediately and effectively.
[0072] Furthermore, after the computer device writes the data to be written back in the memory block to the target file on the disk, the check data in the file header is used to perform data integrity check when reading the data in the target file. Specifically, when the computer device accesses this part of the data again, it first reads the check data corresponding to this part of the data to be accessed in the file header, and obtains the check result by parsing the check data. The check result can determine the integrity of the data to be accessed. When it is determined that the check result of the data to be accessed is that the data is complete, the corresponding data to be accessed is read from the file, thereby ensuring the integrity of the data.
[0073] In the above data storage method, when the application of the computer device needs to write data into the memory, the memory occupancy of the memory block is obtained; when the memory occupancy of the memory block reaches the memory threshold, the memory block is reorganized to determine the data to be written back in the memory block and the verification data corresponding to the data to be written back. The computer device further writes the verification data into the file header of the target file, which is used to perform data integrity verification when reading the data in the target file to ensure the integrity of the data; the computer device immediately calls the system function for synchronizing the memory data with the file data, and writes the data to be written back to the target file, so that the data to be written back can be written into the file immediately and effectively, thereby improving the integrity of the write-back data written into the file, and then effectively reducing the damage rate and loss rate of the data during the write-back process.
[0074] In one embodiment, the memory usage of a memory block is obtained, including: when an application writes data to the memory block, obtaining a data write request triggered by the application; the data write request is used to request to write data to the memory block based on a memory mapping file; according to the data write request, parsing the memory mapping file to obtain the memory usage of the memory block.
[0075] Among them, a memory-mapped file refers to a mapping from a file to a piece of memory, so that the application processes the file as if it were accessing memory. Computer devices can pre-build memory-mapped files according to preset rules. The memory-mapped file is managed by the memory manager of the operating system, so the application only needs to write the content to the memory and let the operating system complete the memory-to-file write-back operation, which will not cause data loss that may be caused by program exit.
[0076] The operating system provides a function that allows applications to map files to a process. Memory-mapped files are similar to virtual memory. Through memory-mapped files, an area of address space can be reserved and physical memory can be submitted to this area. The physical memory mapped by the memory file comes from a file that already exists on the disk, and the file needs to be mapped before operating on the file. Specifically, the memory mapping function can be used to pre-build a memory-mapped file. For example, the mmap() function can be used to map file data to memory. The mmap() system call function enables processes to share memory by mapping the same ordinary file.
[0077] When the application is running, it requests the operating system to map the file's memory space, which is then allocated to the application by the operating system. The memory space allocated by the operating system can be composed of memory blocks, each of which can be represented by the memory start address + memory offset. The shared data written by the process to the memory-mapped file can occupy one memory block or several memory blocks.
[0078] When an application running in a computer device writes data to a memory block, the application initiates a data write request to the memory. The computer device then detects the memory usage of the memory block to determine whether data write-back processing is required. Specifically, the computer device parses the memory mapping file to obtain a mapping relationship between the file and the memory to determine the data mapped in the memory block, such as the shared data read by each process in the memory block, so as to determine the memory usage of the data in the current memory block.
[0079] In this embodiment, when the application writes data to the memory block, it is determined whether to perform data write-back processing, so that the data in the memory can be written back in time. Instead of the operating system completely deciding when to write back the data, if the operating system decides when to write back, it may cause the data to be written back untimely. By instantly calling the system function for synchronizing the memory data with the file data, the data to be written back can be written into the file instantly and effectively, thereby effectively reducing the damage rate and loss rate of the file data during the write-back process.
[0080] In one embodiment, memory data in a memory block includes a key-value pair, and the key-value pair includes a keyword and a corresponding value; data in the memory block is reorganized to determine the data to be written back in the memory block, including: based on the keyword of the key-value pair in the memory block, deduplication processing is performed on the key-value pairs in the memory block to obtain valid data in the memory block; and the valid data is determined as the data to be written back in the memory block.
[0081] Among them, key-value pair storage is a relatively simple organizational form of the database, namely key-value pairs. Key is the keyword, value is the value, also known as key-value pairs. Among them, the key is the keyword, which refers to the number of the stored data value; the corresponding value refers to the data to be stored. The data is stored as a key-value pair, and the corresponding value can be read, stored, and updated by accessing the specified key.
[0082] When the computer device reorganizes the memory data in the memory block, the data is reorganized according to the key-value pairs of the memory data. Specifically, the computer device determines whether there are repeated or redundant values corresponding to the key words according to the key words of each key-value pair in the memory block. If there are repeated or redundant values, the corresponding key words and corresponding values are deduplicated according to the key words. After deduplicating the key-value pairs in the memory block, valid key-value pair data is obtained, and the computer device determines the valid data as the data to be written back in the memory block, so as to perform a write-back operation on this part of the data.
[0083] In this embodiment, by reorganizing the memory data in the memory block, invalid data in the memory block can be effectively cleaned up, and valid data can be accurately written back, thereby ensuring the validity and accuracy of the data to be written back in the memory block.
[0084] In one embodiment, based on the keyword of the key-value pair in the memory block, the key-value pairs in the memory block are deduplicated to obtain valid data in the memory block, including: obtaining the operation record of the memory data in the memory block; according to the operation record, filtering out the latest value of the value corresponding to the keyword of the key-value pair in the memory block after the operation from the memory data; and determining the keyword and the corresponding latest value as the valid data in the memory block.
[0085] The operation record refers to the trace of the operation on the memory data in the memory block, and the operation record may include the record data of the operation modes such as adding, deleting, modifying and searching data.
[0086] Specifically, during the process of reorganizing the memory data in the memory block, the computer device can obtain the operation record of the memory data in the memory block. The operation record reflects what operations the user has performed on the accessed memory data during the operation of the application. The computer device can identify the valid data in the memory data based on the operation record. Since the operation record reflects all the records of the memory data, there may be duplicate or redundant data in the memory data after these operations. The computer device then filters out the latest value of the value corresponding to the keyword of the key-value pair in the memory block after the operation from the memory data based on the operation record, and determines the keyword and the corresponding latest value as the valid data in the memory block, thereby effectively filtering out duplicate or redundant data in the memory block.
[0087] like Figure 3 FIG. 1 is a schematic diagram of storing key values in a memory block in one embodiment. Figure 3 , key1 and value1 are a key-value pair, key2 and value2, key1 and value11, and key1 and value12 are all corresponding key-value pair data. Among them, the keyword key1 corresponds to three key-value pair data, and value12 is the latest value corresponding to the keyword key1 after the operation. Therefore, after the data is reorganized, only the latest keyword key1 and value12 and the corresponding latest value are retained, and the key-value pair is determined as valid data.
[0088] For example, when updating data, if the method of inserting new data into a memory block is used, for the same key, there may be several key values in the memory, and only the last key value in the memory is the latest and valid. Therefore, when performing a read operation, only the last key value is used as valid data. That is, when reading data, for any key, if there are multiple key values corresponding to the key stored in the memory-mapped file, the key value with the largest memory offset is read as the key value corresponding to the key.
[0089] After the application completes the insertion or update operation, the computer device takes the last written value of each key as the valid value, and then writes all valid key-value pairs back to the disk file. This can effectively remove invalid key-value pair data, thereby freeing up corresponding memory space in the memory block.
[0090] In this embodiment, by reorganizing the memory data in the memory block, the keyword and the corresponding latest value are determined as valid data in the memory block, thereby effectively filtering out duplicate or redundant data in the memory block, so that the valid data can be further accurately written back.
[0091] In one embodiment, obtaining verification data corresponding to the data to be written back includes: determining a memory occupancy value corresponding to the data to be written back; encoding the data to be written back to obtain a verification code corresponding to the data to be written back; and generating verification data corresponding to the data to be written back based on the memory occupancy value and the verification code.
[0092] Among them, the memory usage value refers to the memory size of each data, that is, the size of the data, that is, the number of bytes occupied by the data.
[0093] When the computer device writes back the data to be written back in the memory block, it is necessary to calculate the memory usage value corresponding to the data to be written back. The computer device further encodes the data to be written back to calculate the check code corresponding to the data to be written back. Specifically, the computer device can use the CRC32 cyclic redundancy check method to encode the data to be written back, calculate the corresponding CRC32 check value, and use the CRC32 check value as the check code corresponding to the data to be written back.
[0094] After the computer device calculates the memory occupancy data and the check code corresponding to the data to be written back, the memory occupancy value and the check code are used together as the check data corresponding to the data to be written back, thereby generating the check data corresponding to the data to be written back.
[0095] In this embodiment, the memory occupancy value and the check code are determined as the check data corresponding to the data to be written back, and the check data corresponding to the data to be written back is written into the file header of the target file. This can effectively map the storage position of the data to be written back in the target file, thereby effectively ensuring that the data is successfully written back to the target file.
[0096] In one embodiment, the verification data is written into the file header of the target file, and a system function for synchronizing the memory data with the file data is immediately called to write the data to be written back to the target file, including: after the verification data is successfully written into the file header of the target file, the system function for synchronizing the memory data with the file data is immediately called; based on the data to be written back in the memory block, the function parameters corresponding to the system function are determined; and through the system function and the corresponding function parameters, the data to be written back in the memory block are synchronized to write the data to be written back to the target file.
[0097] Among them, the system function used to synchronize memory data with file data also includes corresponding function parameters. The function parameters corresponding to the system function can be pre-configured, or they can be configured according to the attributes of the data to be written back when calling, so that the system function can be effectively called and accurately implement the synchronization operation of memory data and file data. For example, the function parameters may include parameters such as the memory area returned during memory mapping, the length specified during memory mapping, and the synchronization method of memory and disk.
[0098] After the computer device obtains the verification data corresponding to the data to be written back, it first writes the verification data corresponding to the data to be written back into the file header of the target file to verify the integrity of the data to ensure the integrity and validity of the data written into the file.
[0099] When the check data corresponding to the data to be written back is successfully written into the file header of the target file, the computer device immediately calls the system function for synchronizing the memory data with the file data, and determines the function parameters corresponding to the system function according to the data to be written back. Among them, the function parameters can be automatically configured by the operating system according to the attribute information of the data to be written back. The computer device then synchronizes the data to be written back in the memory block to the target file by calling the system function and the corresponding function parameters, thereby realizing the synchronization operation of the memory data and the file data, thereby being able to write the data to be written back to the target file immediately and effectively.
[0100] In this embodiment, by determining the data to be written back in the memory block and writing the verification data of the data to be written back into the file header of the target file, the system function for synchronizing the memory data with the file data is immediately called to synchronize the data to be written back to the target file. In this way, the data to be written back can be written back to the target file in a timely and effective manner, effectively improving the integrity of the write-back data in the written file, thereby effectively reducing the damage rate of the file data.
[0101] In one embodiment, a synchronization operation is performed on the data to be written back in the memory block through a system function and corresponding function parameters to write the data to be written back to a target file, including: determining a data sequence and a mapping area of the data to be written back in the memory block based on a memory mapping file; synchronizing the data to be written back to a storage location corresponding to the mapping area in the target file according to the data sequence through a system function and corresponding function parameters.
[0102] The data sequence refers to the order of the corresponding memory locations of the memory data in the memory block. Specifically, the data sequence may be a sequence after the memory data in the memory block is reorganized. The mapping area refers to the mapping location area of the file data and the memory data, which refers to the one-to-one correspondence between the location of the file on the disk and the same area in the process logical address space. When constructing a memory-mapped file, the relevant data structure and mapping area are established and initialized.
[0103] A memory reorganization sequence number may be stored in the memory mapping file and the process respectively. The memory reorganization sequence number increases monotonically. The memory reorganization sequence number stored in the memory mapping file may be the first memory reorganization sequence number, and the memory reorganization sequence number cached in the process may be the second memory reorganization sequence number. After the data in the memory block is reorganized, the memory reorganization sequence number is incremented. In this way, each process can determine whether another process has triggered the memory reorganization by comparing the first memory reorganization sequence number and the second memory reorganization sequence number.
[0104] In one embodiment, when a process finds that the memory has been reorganized, a data synchronization operation of the process can be triggered. Specifically, each process can compare the second memory reorganization sequence number of the cache with the first memory reorganization sequence number, and if the second memory reorganization sequence number is different from the first memory reorganization sequence number, all shared data stored in the memory mapping file are reloaded into the cache.
[0105] When the computer device performs synchronization operation on the data to be written back through the system function for synchronizing memory data with file data, the data sequence and mapping area of the data to be written back are determined according to the memory mapping file, and then the data to be written back is synchronized to the storage location corresponding to the mapping area in the target file according to the data sequence.
[0106] In this embodiment, a system function is used to synchronize memory data with file data, and the data to be written back is synchronized to the storage location corresponding to the mapping area in the target file according to the data sequence, thereby enabling the data to be written back from the memory area to the corresponding file area accurately and effectively.
[0107] In one embodiment, Figure 4 As shown, another data storage method is provided, which specifically includes the following steps:
[0108] S402, when the application reads data from the target file, a data reading request triggered by the application is obtained.
[0109] S404: According to the data read request, the file data corresponding to the data read request in the target file is cached to the memory block based on the memory mapping file.
[0110] S406, obtaining the memory usage of the memory block.
[0111] S408, when the memory usage of the memory block reaches the memory threshold, reorganize the data of the memory block to determine the data to be written back in the memory block.
[0112] S410, obtaining verification data corresponding to the data to be written back.
[0113] S412, write the verification data into the file header of the target file, and immediately call the system function used to synchronize the memory data with the file data, and write the data to be written back to the target file; wherein, the verification data in the file header is used to perform data integrity verification when reading the data in the target file.
[0114] The data read request refers to a read request triggered when the computer device runs an application and the application requests to read data from a disk file through the operating system.
[0115] When an application accesses file data, the operating system of the computer device first maps the data to be accessed from the file system structure based on the data read request triggered by the application, and then loads the data to be accessed from the disk into the memory according to the byte offset of the data to be accessed.
[0116] Specifically, the operating system of the computer device determines the data attribute information such as the byte offset, data length and data position of the data to be accessed according to the data read request, and can find the corresponding physical address of the data to be accessed in the disk file through the memory mapping file. The operating system sends instructions to the disk to read the corresponding data to be accessed from the disk, and then maps the data to be accessed to the corresponding memory area in the corresponding memory block according to the data attribute information, so as to cache the data to be accessed in the memory.
[0117] After the application accesses data or writes data into the memory, when the process of the computer device writes data into the memory again, the memory occupancy of the memory block is obtained. When the memory occupancy of the memory block reaches the memory threshold, the memory block is reorganized to determine the data to be written back in the memory block, and then the data to be written back in the memory block is written back.
[0118] In this embodiment, when an application accesses data during operation, the operating system first caches the data to be accessed from the disk to the memory, so that the application directly reads the corresponding data to be accessed from the memory, thereby effectively improving data access efficiency.
[0119] In one embodiment, according to a data read request, the file data corresponding to the data read request in the target file is cached to a memory block based on a memory mapping file, and a write lock of the memory mapping file is obtained through a key-value pair storage component; based on the write lock of the memory mapping file, the file data is written to the memory block, and the write lock is released when the file data writing is completed.
[0120] The key-value pair storage component is a data storage component for storing key-value pairs, such as an MMKV component, which can realize simultaneous access to multiple processes.
[0121] File lock is a technology to achieve multi-process access synchronization restrictions, which is divided into shared locks (also known as read locks) and exclusive locks (also known as write locks). When a process adds an exclusive lock to a certain area of a file, other processes must wait for this process to unlock it before they can obtain access to this file area; when a process adds a shared lock to a certain area of a file, other processes can obtain the shared lock of this area, but cannot obtain the exclusive lock.
[0122] The operating system of the computer device maps file data to memory, and when writing data to the memory block, the data is written to the memory block through the key-value pair storage component.
[0123] Specifically, when the application writes data to the memory block corresponding to the memory-mapped file during operation, it applies to the operating system for the memory space of the mapped file, which is allocated to the application by the operating system. Before a process in the application writes data to the memory-mapped file, it needs to apply to the operating system for a write lock of the memory-mapped file, that is, to add a write lock to the memory-mapped file. After receiving the request of the process, the operating system can mark the memory block corresponding to the operation area applied by the process as a write operation. If other processes apply to the operating system for a read lock or a write lock for the same area during the process writing data, the operating system will reject the application until the previous process releases the write lock of the area. In other words, after a process obtains the write lock of an area in the memory-mapped file, other processes need to wait for the process to unlock it before they can obtain access rights to the corresponding area, that is, to obtain the read lock or the write lock of the area.
[0124] After a process obtains a write lock on a certain operation area of a memory-mapped file, it can write shared data to the memory-mapped file. If a process fails to obtain a write lock on a certain operation area of a memory-mapped file, it means that other processes have obtained the write lock or read lock of the area. The process needs to wait for other processes to release the write lock of the area before applying to the system to obtain the write lock of the area. After a process completes the data writing operation to a certain area of a memory-mapped file, it releases the corresponding write lock so that other processes can access the area.
[0125] When writing shared data to a memory-mapped file, the shared data can be inserted into the memory-mapped file in the order in which the shared data is written, that is, the newly added data is written into the memory in the order of increasing memory offset.
[0126] In this embodiment, the file data is written into the memory block through the key-value pair storage component based on the memory mapping file and the file lock, so that the data can be written into the memory efficiently.
[0127] In one embodiment, the key-value pair storage component is a cross-process storage component based on memory mapping, which is used to simultaneously access the key-value pairs corresponding to each running process to perform read processing on the accessed key-value pairs.
[0128] The key-value pair storage component may also be a cross-process storage component based on memory mapping, such as an MMKV component, which is a key-value component based on mmap memory mapping. This storage component can be accessed by multiple processes at the same time, and the data update generated by one process can be effective for multiple processes at the same time. In this way, the key-value pairs corresponding to each running process can be accessed at the same time to read and process the accessed key-value pairs.
[0129] In this embodiment, a cross-process storage component based on memory mapping is used to read data, which can support data access operations of multiple processes and effectively improve the data reading efficiency.
[0130] In one embodiment, after writing the data to be written back to the target file, the above data storage method also includes: when the application reads data from the target file again, obtaining a data read request triggered by the application; determining the data to be accessed according to the data read request, and obtaining verification data corresponding to the data to be accessed from the file header of the target file; based on the verification data, performing a data integrity check on the data to be accessed; when the data integrity check passes, caching the data to be accessed in the target file to a memory block, and reading the data to be accessed from the memory block.
[0131] The verification data is used to perform data integrity verification when the corresponding data is accessed again to verify the integrity and validity of this part of the data.
[0132] After the memory data in the memory block is written back to the disk file, when the application accesses the data again, the operating system still first loads the data to be accessed from the disk into the memory to cache the data to be accessed in the memory. The application then reads the corresponding data to be accessed from the memory.
[0133] Specifically, when the operating system of the computer device reads the data to be accessed from the target file on the disk, the application triggers a data read request, and after determining the data to be accessed according to the data read request, the operating system in the computer device first obtains the verification data corresponding to the data to be accessed from the file header of the target file. The computer device then performs a data integrity check on the data to be accessed based on the verification data to determine whether the data to be accessed is complete. When the data integrity check passes, it means that this part of the data to be accessed is complete and accurate.
[0134] The computer device then caches the to-be-accessed data in the target file into the memory block, so that when the application reads the to-be-accessed data, it can directly and quickly read the required to-be-accessed data from the memory block.
[0135] In this embodiment, by writing the verification data corresponding to the data to be written back into the file header of the target file, the computer device can effectively perform data integrity verification when accessing the data again, thereby effectively ensuring the integrity of the data.
[0136] In one embodiment, Figure 5 FIG. 1 is a flow chart of the steps of reading data in one embodiment, which specifically includes the following steps:
[0137] S502: When the application reads data from the target file again, a data reading request triggered by the application is obtained.
[0138] S504: Determine the data to be accessed according to the data read request, and obtain verification data corresponding to the data to be accessed from the file header of the target file.
[0139] S506: Perform data integrity verification on the data to be accessed based on the verification data.
[0140] S508, when the data integrity check passes, the key-value pair corresponding to the data to be accessed in the target file is read through the data storage component.
[0141] S510 , according to the memory sequence of the memory block, cache the key-value pairs corresponding to the to-be-accessed data into a hash table of the memory block.
[0142] S512: Based on the data read request, read the key-value pair corresponding to the data to be accessed from the hash table of the memory block.
[0143] Among them, the memory sequence is the organization method of data in the memory space, that is, the data sequence created when data is written into the memory.
[0144] When the application process in the computer device accesses data, it first determines the corresponding data to be accessed in the target file according to the mapping relationship corresponding to the memory mapping file, and performs data integrity check on the data to be accessed according to the verification data. After the verification is passed, the computer device reads the key-value pairs corresponding to the data to be accessed in the target file through the built data storage component, and caches the data to be accessed in the target file to the memory block. Among them, when the operating system of the computer device caches the data to be accessed to the memory, it creates a memory sequence of the data to be accessed in the memory according to the current storage status of the memory block, and then caches the key-value pairs corresponding to the data to be accessed into the hash table of the memory block according to the memory sequence of the memory block.
[0145] Furthermore, for data read requests triggered by applications, applications can directly read key-value pairs corresponding to the data to be accessed from the hash table of the memory block. Since the data access speed in memory is higher than that of disk files, the application can quickly and efficiently read data from the memory when it is running.
[0146] For example, when an application process in a computer device reads data for the first time after startup, it first mmaps the entire file data to be accessed into memory, that is, maps the file data to memory. Then, a data integrity check is performed according to the check data of the file data, that is, the memory size and CRC32 check code. If the check passes, each key-value pair corresponding to the file data is read in memory order and cached into the hash table in memory.
[0147] When the application accesses each key-value data subsequently, it directly finds the corresponding key-value pair from the hash table. Before the data cached in the memory block is written back to the disk file, if the application accesses the corresponding key-value data again, it can still directly find the corresponding key-value pair from the hash table to read the required data.
[0148] In this embodiment, the check data of the accessed data is first verified and then cached in a hash table of the memory, so that when the application reads the data, it can directly access the required data from the hash table of the memory block, thereby being able to quickly and efficiently obtain the required complete data.
[0149] In a specific embodiment, Figure 6 As shown, a specific data storage method is provided, comprising the following steps:
[0150] S602, when the application writes data to the memory block, a data write request triggered by the application is obtained; the data write request is used to request to write data to the memory block based on the memory mapping file.
[0151] S604, parsing the memory mapping file according to the data write request to obtain the memory usage of the memory block.
[0152] S606, when the memory usage of the memory block reaches the memory threshold, based on obtaining the operation record of the memory data in the memory block; the memory data in the memory block includes a key-value pair, and the key-value pair includes a keyword and a corresponding value.
[0153] S608, according to the operation record, filter out from the memory data the latest value of the value corresponding to the keyword of the key-value pair in the memory block after the operation.
[0154] S610, determining the keyword and the corresponding latest value as valid data in the memory block.
[0155] S612, determining the memory usage value corresponding to the data to be written back.
[0156] S614, encode the data to be written back to obtain a check code corresponding to the data to be written back.
[0157] S616, generating verification data corresponding to the data to be written back based on the memory usage value and the verification code.
[0158] S618, after the verification data is successfully written into the file header of the target file, a system function for synchronizing the memory data with the file data is immediately called; wherein the verification data in the file header is used to perform data integrity verification when reading the data in the target file.
[0159] S620: Determine function parameters corresponding to the system function based on the data to be written back in the memory block.
[0160] S622, based on the memory mapping file, determine the data sequence and mapping area of the data to be written back in the memory block.
[0161] S624, using the system function and the corresponding function parameters, synchronize the data to be written back to the storage location corresponding to the mapping area in the target file according to the data sequence.
[0162] In this embodiment, when the application of the computer device needs to write data to the memory, the memory occupancy of the memory block is obtained; when the memory occupancy of the memory block reaches the memory threshold, the memory block is reorganized to determine the data to be written back in the memory block and the verification data corresponding to the data to be written back. The computer device further writes the verification data into the file header of the target file, and synchronizes the data to be written back to the target file by immediately calling the system function for synchronizing the memory data with the file data, so that the data to be written back can be written into the file immediately and effectively, thereby improving the integrity of the write-back data written into the file, and further effectively reducing the damage rate of the file.
[0163] The present application also provides an application scenario, which applies the above data storage method. Specifically, the application of the data storage method in the application scenario is as follows:
[0164] The computer device is installed with an Android operating system and an Android application. The Android application runs corresponding processes through the Android operating system to access data and implement data writing or reading.
[0165] When an Android application writes data to memory, it requests memory space for a mapped file from the Android operating system, which is then allocated to the application. All processes within the Android application share the memory space allocated by the Android operating system. Each process can write data to or read data from the memory space allocated by the Android operating system, thus supporting multi-process data access.
[0166] Furthermore, when the Android application writes data to the memory, the memory usage of the memory block is also obtained; when the memory usage of the memory block reaches the memory threshold, the Android operating system of the computer device reorganizes the data of the memory block to determine the data to be written back in the memory block, and obtains the verification data corresponding to the data to be written back. The Android operating system then writes the verification data into the file header of the target file, and the verification data in the file header is used to perform data integrity verification when reading the data in the target file. The Android operating system immediately calls the system function for synchronizing the memory data with the file data to write the data to be written back to the target file immediately, thereby effectively ensuring that the data is written back to the target file immediately and completely, effectively improving the integrity of the write-back data in the written file, and thus effectively reducing the damage rate of the data during the write-back process.
[0167] The present application also provides an application scenario, which applies the above data storage method. Specifically, the application of the data storage method in the application scenario is as follows:
[0168] The computer device is installed with the iOS operating system and iOS applications. The iOS applications run corresponding processes through the iOS operating system to access data and implement data writing or reading.
[0169] When an iOS application writes data to memory, it requests the iOS operating system to map the file's memory space, which is then allocated to the application by the iOS operating system. All processes within the iOS application share the memory space allocated by the iOS operating system, and each process can write data to or read data from the memory space allocated by the iOS operating system, thus supporting multi-process data access.
[0170] Furthermore, when the iOS application writes data to the memory, the memory usage of the memory block is also obtained; when the memory usage of the memory block reaches the memory threshold, the iOS operating system of the computer device reorganizes the data of the memory block to determine the data to be written back in the memory block, and obtains the verification data corresponding to the data to be written back. The iOS operating system then writes the verification data into the file header of the target file, and the verification data in the file header is used to perform data integrity verification when reading the data in the target file. The iOS operating system immediately calls the system function for synchronizing the memory data with the file data to write the data to be written back to the target file immediately, thereby effectively ensuring that the data is written back to the target file immediately and completely, effectively improving the integrity of the write-back data in the written file, and thus effectively reducing the data damage rate during the write-back process.
[0171] It should be understood that although Figure 2 , 4 The steps in the flowcharts of , 5, and 6 are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2 , 4 At least part of the steps in , 5, and 6 may include multiple steps or multiple stages. These steps or stages do not necessarily have to be executed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily one by one, but can be executed in turn or alternately with other steps or at least part of the steps or stages in other steps.
[0172] In one embodiment, Figure 7 As shown, a data storage device 700 is provided. The device can adopt a software module or a hardware module, or a combination of the two to become a part of a computer device. The device specifically includes: a data acquisition module 702, a data reorganization module 704 and a data write-back module 706, wherein:
[0173] The data acquisition module 702 is used to acquire the memory usage of the memory block;
[0174] The data reorganization module 704 is used to reorganize the data of the memory block when the memory occupancy of the memory block reaches the memory threshold, so as to determine the data to be written back in the memory block;
[0175] The data acquisition module 702 is also used to acquire verification data corresponding to the data to be written back;
[0176] The data write-back module 706 is used to write the verification data into the file header of the target file, and immediately call the system function used to synchronize the memory data with the file data to write the data to be written back to the target file; wherein, the verification data in the file header is used to perform data integrity verification when reading the data in the target file.
[0177] In one embodiment, the data acquisition module 702 is also used to obtain a data write request triggered by an application when the application writes data to a memory block; the data write request is used to request to write data to the memory block based on a memory mapping file; according to the data write request, the memory mapping file is parsed to obtain the memory occupancy of the memory block.
[0178] In one embodiment, the memory data in the memory block includes a key-value pair, and the key-value pair includes a keyword and a corresponding value; the data reorganization module 704 is also used to deduplicate the key-value pairs in the memory block based on the keywords of the key-value pairs in the memory block to obtain valid data in the memory block; and determine the valid data as the data to be written back in the memory block.
[0179] In one embodiment, the data reorganization module 706 is also used to obtain operation records of memory data in the memory block; based on the operation records, filter out the latest value of the value corresponding to the keyword of the key-value pair in the memory block after the operation from the memory data; and determine the keyword and the corresponding latest value as valid data in the memory block.
[0180] In one embodiment, the data acquisition module 702 is also used to determine the memory occupancy value corresponding to the data to be written back; encode the data to be written back to obtain the verification code corresponding to the data to be written back; and generate the verification data corresponding to the data to be written back based on the memory occupancy value and the verification code.
[0181] In one embodiment, the data write-back module 706 is also used to call a system function for synchronizing memory data with file data immediately after successfully writing the verification data into the file header of the target file; determine the function parameters corresponding to the system function based on the data to be written back in the memory block; and synchronize the data to be written back in the memory block through the system function and the corresponding function parameters so as to write the data to be written back to the target file.
[0182] In one embodiment, the data write-back module 706 is also used to determine the data sequence and mapping area of the data to be written back in the memory block based on the memory mapping file; through the system function and the corresponding function parameters, the data to be written back is synchronized to the storage location corresponding to the mapping area in the target file according to the data sequence.
[0183] In one embodiment, the above-mentioned data storage device also includes a data cache module, which is used to obtain a data read request triggered by an application when the application reads data from a target file; according to the data read request, the file data corresponding to the data read request in the target file is cached to a memory block based on a memory mapping file.
[0184] In one embodiment, Figure 8 As shown, the above-mentioned data storage device 700 also includes a data writing module 701, which is used to obtain the write lock of the memory-mapped file through the key-value pair storage component; based on the write lock of the memory-mapped file, write file data into the memory block, and release the write lock when the file data writing is completed.
[0185] In one embodiment, the key-value pair storage component is a cross-process storage component based on memory mapping, which is used to simultaneously access the key-value pairs corresponding to each running process to perform read processing on the accessed key-value pairs.
[0186] In one embodiment, Fig. 9 As shown, the above-mentioned data storage device 700 also includes a data reading module 708, which is used to obtain a data reading request triggered by the application when the application reads data from the target file again; determine the data to be accessed according to the data reading request, and obtain the verification data corresponding to the data to be accessed from the file header of the target file; based on the verification data, perform data integrity verification on the data to be accessed; when the data integrity verification passes, cache the data to be accessed in the target file to the memory block, and read the data to be accessed from the memory block.
[0187] In one embodiment, the data reading module 708 is also used to read the key-value pairs corresponding to the data to be accessed in the target file through the data storage component; cache the key-value pairs corresponding to the data to be accessed into the hash table of the memory block according to the memory sequence of the memory block; and read the key-value pairs corresponding to the data to be accessed from the hash table of the memory block based on the data reading request.
[0188] For the specific definition of the data storage device, please refer to the definition of the data storage method above, which will not be repeated here. Each module in the above data storage device can be implemented in whole or in part by software, hardware and a combination thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.
[0189] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.10As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a data storage method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.
[0190] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0191] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.
[0192] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above method embodiments when executed by a processor.
[0193] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps in the above-mentioned method embodiments.
[0194] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).
[0195] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0196] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.
Claims
1. A data storage method, characterized in that: The method comprises: Get the memory usage of the memory block; When the memory occupancy of the memory block reaches a memory threshold, reorganizing the data of the memory block to determine the data to be written back in the memory block; Obtaining verification data corresponding to the data to be written back; Writing the verification data into the file header of the target file, and immediately calling a system function for synchronizing the memory data with the file data, and writing the data to be written back to the target file; The verification data in the file header is used to perform data integrity verification when reading data in the target file.
2. The method according to claim 1, characterized in that The obtaining of the memory usage of the memory block includes: When an application writes data to a memory block, a data write request triggered by the application is obtained; the data write request is used to request to write data to the memory block based on a memory-mapped file; According to the data write request, the memory mapping file is parsed to obtain the memory occupancy of the memory block.
3. The method according to claim 1, characterized in that The memory data in the memory block includes a key-value pair, and the key-value pair includes a keyword and a corresponding value; The reorganizing the data of the memory block to determine the data to be written back in the memory block includes: Based on the keywords of the key-value pairs in the memory block, deduplication processing is performed on the key-value pairs in the memory block to obtain valid data in the memory block; The valid data is determined as the data to be written back in the memory block.
4. The method according to claim 3, characterized in that The deduplication processing of the key-value pairs in the memory block based on the keywords of the key-value pairs in the memory block to obtain valid data in the memory block includes: Obtaining an operation record of memory data in the memory block; According to the operation record, the latest value of the value corresponding to the keyword of the key-value pair in the memory block after the operation is filtered out from the memory data; The keyword and the corresponding latest value are determined as valid data in the memory block.
5. The method according to claim 1, characterized in that The obtaining verification data corresponding to the data to be written back includes: Determine a memory usage value corresponding to the data to be written back; Encoding the data to be written back to obtain a check code corresponding to the data to be written back; Based on the memory occupancy value and the verification code, verification data corresponding to the data to be written back is generated.
6. The method according to claim 1, characterized in that The step of writing the verification data into the file header of the target file and immediately calling a system function for synchronizing the memory data with the file data to write the data to be written back to the target file includes: After the verification data is successfully written into the file header of the target file, a system function for synchronizing the memory data with the file data is immediately called; Determining function parameters corresponding to the system function based on the data to be written back in the memory block; The system function and the corresponding function parameters are used to synchronize the data to be written back in the memory block, so as to write the data to be written back to the target file.
7. The method according to claim 6, characterized in that The step of performing a synchronization operation on the data to be written back in the memory block through the system function and the corresponding function parameter to write the data to be written back to the target file includes: Determine, based on the memory mapping file, a data sequence and a mapping area of the data to be written back in the memory block; The data to be written back is synchronized to a storage location corresponding to the mapping area in the target file according to the data sequence through the system function and the corresponding function parameters.
8. The method according to claim 1, characterized in that Before writing the to-be-written-back data back to the target file, the method further includes: When the application reads data from the target file, obtaining a data reading request triggered by the application; According to the data read request, the file data in the target file corresponding to the data read request is cached to the memory block based on a memory mapping file.
9. The method according to claim 1, characterized in that: The method further comprises: Obtain the write lock of the memory-mapped file through the key-value pair storage component; Based on the write lock of the memory-mapped file, file data is written into the memory block, and the write lock is released when the writing of the file data is completed.
10. The method according to claim 9, characterized in that The key-value pair storage component is a cross-process storage component based on memory mapping, which is used to simultaneously access the key-value pairs corresponding to each running process to read and process the accessed key-value pairs.
11. The method according to any one of claims 1 to 10, characterized in that: After writing the to-be-written-back data back to the target file, the method further includes: When the application reads data from the target file again, obtaining a data reading request triggered by the application; Determine the data to be accessed according to the data read request, and obtain verification data corresponding to the data to be accessed from the file header of the target file; Based on the verification data, performing data integrity verification on the data to be accessed; When the data integrity check passes, the data to be accessed in the target file is cached in the memory block, and the data to be accessed is read from the memory block.
12. The method according to claim 11, characterized in that The step of caching the to-be-accessed data in the target file into the memory block and reading the to-be-accessed data from the memory block comprises: Reading the key-value pairs corresponding to the data to be accessed in the target file through the data storage component; According to the memory sequence of the memory block, cache the key-value pairs corresponding to the to-be-accessed data into a hash table of the memory block; Based on the data read request, a key-value pair corresponding to the to-be-accessed data is read from a hash table of the memory block.
13. A data storage device, characterized in that: The device comprises: A data acquisition module is used to obtain the memory usage of a memory block; A data reorganization module, used for reorganizing the data of the memory block when the memory occupancy of the memory block reaches a memory threshold, so as to determine the data to be written back in the memory block; The data acquisition module is also used to acquire verification data corresponding to the data to be written back; A data write-back module is used to write the verification data into the file header of the target file, and immediately call the system function used to synchronize the memory data with the file data to write the data to be written back to the target file; wherein the verification data in the file header is used to perform data integrity verification when reading the data in the target file.
14. The device according to claim 13, characterized in that The obtaining of the memory usage of the memory block includes: When an application writes data to a memory block, a data write request triggered by the application is obtained; the data write request is used to request to write data to the memory block based on a memory-mapped file; According to the data write request, the memory mapping file is parsed to obtain the memory occupancy of the memory block.
15. The device according to claim 13, characterized in that The memory data in the memory block includes a key-value pair, and the key-value pair includes a keyword and a corresponding value; The reorganizing the data of the memory block to determine the data to be written back in the memory block includes: Based on the keywords of the key-value pairs in the memory block, deduplication processing is performed on the key-value pairs in the memory block to obtain valid data in the memory block; The valid data is determined as the data to be written back in the memory block.
16. The device according to claim 15, characterized in that The deduplication processing of the key-value pairs in the memory block based on the keywords of the key-value pairs in the memory block to obtain valid data in the memory block includes: Obtaining an operation record of memory data in the memory block; According to the operation record, the latest value of the value corresponding to the keyword of the key-value pair in the memory block after the operation is filtered out from the memory data; The keyword and the corresponding latest value are determined as valid data in the memory block.
17. The device according to claim 13, characterized in that The obtaining verification data corresponding to the data to be written back includes: Determine a memory usage value corresponding to the data to be written back; Encoding the data to be written back to obtain a check code corresponding to the data to be written back; Based on the memory occupancy value and the verification code, verification data corresponding to the data to be written back is generated.
18. The device according to claim 13, characterized in that The step of writing the verification data into the file header of the target file and immediately calling a system function for synchronizing the memory data with the file data to write the data to be written back to the target file includes: After the verification data is successfully written into the file header of the target file, a system function for synchronizing the memory data with the file data is immediately called; Determining function parameters corresponding to the system function based on the data to be written back in the memory block; The system function and the corresponding function parameters are used to synchronize the data to be written back in the memory block, so as to write the data to be written back to the target file.
19. The device according to claim 18, characterized in that The step of performing a synchronization operation on the data to be written back in the memory block through the system function and the corresponding function parameter to write the data to be written back to the target file includes: Determine, based on the memory mapping file, a data sequence and a mapping area of the data to be written back in the memory block; The data to be written back is synchronized to a storage location corresponding to the mapping area in the target file according to the data sequence through the system function and the corresponding function parameters.
20. The device according to claim 13, characterized in that Before writing the to-be-written-back data back to the target file, the device further includes: When the application reads data from the target file, obtaining a data reading request triggered by the application; According to the data read request, the file data in the target file corresponding to the data read request is cached to the memory block based on a memory mapping file.
21. The device according to claim 13, characterized in that The device also includes: Obtain the write lock of the memory-mapped file through the key-value pair storage component; Based on the write lock of the memory-mapped file, file data is written into the memory block, and the write lock is released when the writing of the file data is completed.
22. The device according to claim 21, characterized in that The key-value pair storage component is a cross-process storage component based on memory mapping, which is used to simultaneously access the key-value pairs corresponding to each running process to read and process the accessed key-value pairs.
23. The device according to any one of claims 13 to 22, characterized in that After writing the to-be-written-back data back to the target file, the device further includes: When the application reads data from the target file again, obtaining a data reading request triggered by the application; Determine the data to be accessed according to the data read request, and obtain verification data corresponding to the data to be accessed from the file header of the target file; Based on the verification data, performing data integrity verification on the data to be accessed; When the data integrity check passes, the data to be accessed in the target file is cached in the memory block, and the data to be accessed is read from the memory block.
24. The device according to claim 23, characterized in that The step of caching the to-be-accessed data in the target file into the memory block and reading the to-be-accessed data from the memory block comprises: Reading the key-value pairs corresponding to the data to be accessed in the target file through the data storage component; According to the memory sequence of the memory block, cache the key-value pairs corresponding to the to-be-accessed data into a hash table of the memory block; Based on the data read request, a key-value pair corresponding to the to-be-accessed data is read from a hash table of the memory block.
25. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 12 are implemented.
26. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.
27. A computer program product comprising computer instructions, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 12.
Citation Information
Patent Citations
Data access method and system in cache system
CN105677579A
Data storage method and device, terminal equipment and storage medium
CN109597568A
Data processing method and device
CN109710185A
Data access method and device and storage medium
CN110399227A
Managing memory health
US20060236165A1
