Data processing method and device, electronic equipment and storage medium
By adding identification fields to data objects and using circular buffer technology, the problems of low read and write efficiency and low flash space utilization in NVM data processing are solved, and wear equalization and efficient data processing are achieved.
Patent Information
- Application Number
- CN202411943201.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-24
- Publication Date
- 2025-05-16
AI Technical Summary
In the existing NVM data processing methods, there are problems such as low reading and writing efficiency of data, low flash memory space utilization, and wear balance.
By adding identification fields to the data object and using the available space for the flash memory as a circular buffer, when receiving the data storage instructions, the data object and identification fields are written to the circular buffer, and a data object information table is maintained in memory, recording the identification fields and storage location information of each data object, and recycling the flash memory space.
It improves the usage rate of flash memory space, ensures wear balance, and improves data reading and writing efficiency.
Smart Images
Figure CN120010765A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the technical field of data storage, and in particular, relates to a data processing method, device, electronic device and storage medium. Background Art
[0002] Non-volatile memory (NVM) is a memory that can persist data during power failure, which is widely used in embedded systems. Currently, the most common ones are Nor flash and Nand flash. The former has small capacity, high price, fast reading speed, and supports on-chip execution of instructions, while the latter has large capacity, low price, and relatively fast writing and erasing speed. Both can be saved during power failure and can be read and written (usually in units of 4 bytes), but the entire block (usually in units of KB) needs to be erased before writing, and the number of erasable times of each block is limited. In most embedded systems, flash memory is not only used to store execution programs, but also to store data that needs to be frequently read and written when the program is running. Therefore, for more complex embedded systems, the data reading and writing efficiency and the service life of flash memory need to be considered in software design to ensure wear leveling. In the related NVM data processing methods, there are problems that cannot be taken into account, such as low reading and writing efficiency of NVM data, low flash memory space utilization, and wear leveling. Summary of the invention
[0003] The present application proposes a data processing method, device, electronic device and storage medium to solve the problems existing in the related art such as low read and write efficiency of NVM data, low flash memory space utilization, wear leveling and the like that cannot be taken into account at the same time.
[0004] In a first aspect, the present application provides a data processing method, comprising:
[0005] receiving a data storage instruction, wherein the data storage instruction is used to instruct to store a first data object;
[0006] In response to the data storage instruction, obtaining an identification field of the first data object, and writing the first data object and the identification field into a circular buffer, wherein the identification field includes a version number and identification information;
[0007] The storage location information of the first data object in the circular buffer is obtained, and the storage location information and the identification field are stored in a data object information table in the memory.
[0008] In an embodiment of the present application, an identification field is added to a data object. By using the available space of the flash memory as a circular buffer, when a data storage instruction is received, the data object and the identification field of the data object are written into the circular buffer, and a data object information table is maintained in the memory. The data object information table records the identification field of each data object and the storage location information of the data object in the flash memory space. By combining the data object information table and the circular buffer, the flash memory space can be recycled, thereby improving the utilization rate of the flash memory space.
[0009] According to some embodiments of the present application, in response to the data storage instruction, obtaining the identification field of the first data object, and writing the first data object and the identification field into a circular buffer includes:
[0010] Acquire identification information of the first data object, and query the data object information table according to the identification information whether the first data object exists;
[0011] If not, initialize the version number of the first data object, and write the first data object and the identification field into a blank position of the circular buffer;
[0012] If it exists, the version number of the first data object is increased by one, and the first data object and the identification field are written into a blank position of the circular buffer.
[0013] In an embodiment of the present application, by combining a data object information table and a circular buffer, version information is added to the data object and sequentially written into blank locations of the flash memory, the entire flash memory space is recycled, the flash memory space utilization is improved, and wear leveling is ensured.
[0014] According to some embodiments of the present application, the circular buffer includes: a header and a tail, and a blank position between the header and the tail of the circular buffer can store a data object of a preset data length;
[0015] Writing the first data object and the identification field into a blank position of a circular buffer comprises:
[0016] The first data object and the identification field are written into a blank position at the head of the circular buffer.
[0017] In an embodiment of the present application, the entire available space of the flash memory is regarded as a circular buffer. When storing data, the data objects are written sequentially to the blank locations of the flash memory, that is, the head of the circular buffer. As the data is stored, the tail of the circular buffer is always the oldest version of the data, so wear leveling can be achieved.
[0018] According to some embodiments of the present application, the method further includes:
[0019] When the first data object cannot be written into the blank position at the head of the circular buffer, the target position at the tail of the circular buffer is erased.
[0020] According to some embodiments of the present application, the method further includes:
[0021] receiving a data erasing instruction, wherein the data erasing instruction is used to instruct erasing a target flash memory page;
[0022] In response to the data erase instruction, acquiring address information corresponding to the target flash memory page;
[0023] According to the address information, query the data object information table to determine whether there is a data object of the latest version in the target flash memory page;
[0024] If not present, erasing the target flash memory page;
[0025] Alternatively, if it exists, the latest version of the data object is copied to the new blank flash memory page in sequence, the storage location information corresponding to the latest version of the data object in the data object information table is updated, and the target flash memory page is erased.
[0026] In an embodiment of the present application, when a data erase instruction is received, it is first determined whether the latest version of the data object exists in the target flash memory page to be erased. If so, it is copied first and then erased. If not, it is erased directly. This ensures that the old version of the data object is always erased. In most cases, there is no need to copy or only a few data objects in the target flash memory page to be erased are copied, thereby ensuring write efficiency.
[0027] According to some embodiments of the present application, the data object information table further includes: a memory backup flag, the memory backup flag being used to indicate whether the data object is backed up in the memory;
[0028] After writing the first data object and the identification field into a circular buffer, the method further comprises:
[0029] storing the first data object in a memory;
[0030] The memory backup flag bit of the first data object in the data object information table is set to a first value.
[0031] In the embodiment of the present application, by adding a memory backup flag in the data object information table, the data reading efficiency can be improved.
[0032] According to some embodiments of the present application, the method further includes:
[0033] receiving a data read instruction, wherein the data read instruction is used to instruct to read a second data object;
[0034] In response to the data read instruction, determining whether the second data object exists in the memory through a memory backup flag in the data object information table;
[0035] If it exists, read the second data object from the memory;
[0036] or, if it does not exist, obtaining storage location information of the second data object based on the data object information table;
[0037] Based on the storage location information of the second data object, the second data object is read from the circular buffer, and the second data object is stored in the memory.
[0038] In an embodiment of the present application, when data needs to be read, the data object is located through the data object information table. If there is a copy in the memory, it is read directly from the memory. If not, it is read from the physical flash memory, which can improve data reading efficiency.
[0039] According to some embodiments of the present application, the method further includes:
[0040] When the number of data objects backed up in the memory exceeds a preset value, at least one data object that is least recently used and backed up in the memory is deleted based on a least recently used replacement algorithm.
[0041] In the embodiment of the present application, by retaining copies of some commonly used data objects in the memory, when reading the data object, the position of the data object is located by querying the data object information table, thereby improving the data reading efficiency.
[0042] In a second aspect, the present application provides a data processing device, the device comprising:
[0043] A receiving unit, configured to receive a data storage instruction, wherein the data storage instruction is used to instruct to store a first data object;
[0044] A data storage unit, configured to obtain an identification field of the first data object in response to the data storage instruction, and write the first data object and the identification field into a circular buffer, wherein the identification field includes a version number and identification information;
[0045] The information storage unit is used to obtain the storage location information of the first data object in the circular buffer, and store the storage location information and the identification field in a data object information table in a memory.
[0046] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the data processing method as described in the first aspect above is implemented.
[0047] In a fourth aspect, the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the data processing method as described in the first aspect above.
[0048] In a fifth aspect, the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the data processing method as described in the first aspect.
[0049] In a sixth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the data processing method as described in the first aspect above.
[0050] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0051] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:
[0052] Figure 1 is a flowchart of a data processing method provided by some embodiments of the present application;
[0053] Figure 2 is a schematic diagram of writing a data object into a circular buffer provided by some embodiments of the present application;
[0054] Figure 3 is a schematic diagram of a circular buffer provided in some embodiments of the present application;
[0055] Figure 4 is a schematic diagram of erasing a data object provided by some embodiments of the present application;
[0056] Figure 5 is a schematic diagram of the structure of a data processing device provided in some embodiments of the present application;
[0057] Figure 6 It is a schematic diagram of the structure of an electronic device provided in some embodiments of the present application. DETAILED DESCRIPTION
[0058] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.
[0059] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.
[0060] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments.
[0061] Non-volatile memory (NVM) is a type of memory that can persist data even during power-off periods, which is widely used in embedded systems. Currently, the most common types are Nor flash memory and Nand flash memory. The former has a small capacity, high price, fast read speed, and supports on-chip execution of instructions, while the latter has a large capacity, low price, and relatively fast write and erase speeds. Both can be saved during power-off and can be read and written (usually in units of 4 bytes), but the entire block (usually in units of KB) needs to be erased before writing, and the number of times each block can be erased is limited (tens to millions of times). In most embedded systems, flash memory is not only used to store execution programs, but also to store data that needs to be frequently read and written when the program is running. Therefore, for more complex embedded systems, the read and write efficiency of data and the service life of flash memory (flash memory) need to be considered in software design to ensure wear leveling.
[0062] Currently, there are multiple NVM data storage methods:
[0063] Method 1: Each data object is stored in a different flash page, so that each data object can be erased and rewritten without affecting other data objects;
[0064] Method 2: To utilize the flash memory space, multiple different data objects are stored sequentially in the flash memory pages. When a data object is updated, the page where the data object is located is located, and other data objects in the page are first copied to a new blank flash memory page, and then the new data object is written, and finally the original page is erased;
[0065] Method 3: Divide the available space of the flash memory into two equal blocks, each of which serves as a virtual page. When writing a data object, add version identification information to find the latest version of the data object. Write the new version of each object sequentially to a new blank position in the flash memory virtual page. When the remaining space of the flash memory virtual page is insufficient to write a new data object, copy the latest version of each data object in the original virtual page to a new backup virtual page, then write the new data object, and finally erase the original virtual page.
[0066] Usually, the size of data stored by users is much smaller than the target flash memory page, so method one will waste a lot of storage space. Method two writes new data to the blank position of the target flash memory page, which improves the utilization rate of the page, and adopts the write-before-erase method to avoid the impact of power failure or reset during writing. However, in the case of frequent updates of a data object, the page where the object is located is still frequently erased, and the wear balance of the flash memory cannot be guaranteed. Method three uses multiple flash memory physical pages as a virtual page, and the page is erased in units of virtual pages, which can increase the service life of the flash memory, but the maximum utilization rate of the flash memory space is only 50%, and when the virtual page is full, copying all data objects at once will increase the object writing time in the worst case. In addition, the above three methods do not consider the reading efficiency of NVM data. Especially for Nand flash memory with slow reading speed, it is important to improve the data reading speed.
[0067] In the related NVM data processing methods, there are problems that cannot be taken into account, such as low read and write efficiency of NVM data, low flash memory space utilization, and wear leveling.
[0068] In order to solve the above problems, the embodiments of the present application provide a data processing method, device, electronic device and storage medium. The data processing method, device, electronic device and storage medium provided by the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.
[0069] Figure 1 is a flowchart of a data processing method provided by some embodiments of the present application, such as Figure 1 As shown, the data processing method includes: step 110, step 120 and step 130.
[0070] Step 110: Receive a data storage instruction, where the data storage instruction is used to instruct to store a first data object;
[0071] It should be noted that the first data object may refer to one data object or multiple data objects, and this embodiment of the present application does not limit this.
[0072] Step 120: In response to the data storage instruction, obtain an identification field of the first data object, and write the first data object and the identification field into a circular buffer, wherein the identification field includes a version number and identification information;
[0073] It can be understood that the embodiment of the present application uses the available space of the flash memory as a circular buffer and adopts a circular buffer erase model to manage the entire flash memory physical space, so that the available space of the flash memory can be recycled.
[0074] When a data storage instruction is received, in response to the data storage instruction, the first data object and the identification field of the first data object are obtained, and the first data object and the identification field of the first data object are written into a circular buffer.
[0075] In the embodiment of the present application, in order to distinguish different data objects, an identification field is added to each data object, and the identification field and the data object are written into the circular buffer at the same time during storage.
[0076] The identification field includes identification information (which can be a key) and a version number for distinguishing different data objects. It should be noted that the key of each data object must be unique and cannot be repeated. The version number is used to distinguish different versions of the same data object. The data object with the largest version number is the latest data.
[0077] Optionally, the identification field also includes data length, that is, the identification field includes identification information, version number and data length.
[0078] Step 130: Obtain storage location information of the first data object in the circular buffer, and store the storage location information and the identification field in a data object information table in a memory.
[0079] In the embodiment of the present application, the storage location information of the first data object in the circular buffer, that is, the location information (address information) of the first data object in the flash memory is recorded, and the storage location information of the first data object in the circular buffer is obtained. Then, the storage location information and the identification field of the first data object are stored in the data object information table in the memory.
[0080] The embodiment of the present application maintains a data object information table in the memory, which records the identification field of each data object, the storage location information of the first data object in the circular buffer, whether the memory is backed up, etc. When the NVM is initialized, the data objects in the entire flash memory space are scanned and the data object information table is initialized. The data object information table always records the latest version information of all data objects.
[0081] The data processing method provided in the embodiment of the present application adds an identification field to a data object. By storing the available space of the flash memory as a circular buffer, the data object and the identification field of the data object are written into the circular buffer when a data storage instruction is received, and a data object information table is maintained in the memory. The data object information table records the identification field of each data object and the storage location information of the data object in the flash memory space. By combining the data object information table and the circular buffer, the flash memory space can be recycled, thereby improving the utilization rate of the flash memory space.
[0082] In some embodiments, in response to the data storage instruction, obtaining the identification field of the first data object, and writing the first data object and the identification field into a circular buffer comprises:
[0083] Acquire identification information of the first data object, and query the data object information table according to the identification information whether the first data object exists;
[0084] If not, initialize the version number of the first data object, and write the first data object and the identification field into a blank position of the circular buffer;
[0085] If it exists, the version number of the first data object is increased by one, and the first data object and the identification field are written into a blank position of the circular buffer.
[0086] It is understandable that the data object information table stores identification information and storage location information of the data object, so by obtaining the identification information of the first data object, it is possible to query whether the data object exists in the data object information table according to the identification information.
[0087] If the first data object does not exist in the data object information table, it means that the current data storage instruction is to store a new data object. At this time, the version number of the identification field of the first data object needs to be initialized, for example, the version number of the identification field of the first data object is initialized to 1, and then the first data object and the identification field of the first data object (including identification information and version number) are written into a blank position of the circular buffer. At the same time, the storage position information of the first data object in the circular buffer is obtained, and the storage position information and the identification field are stored in the data object information table in the memory, that is, the identification field of the first data object and the position information of the first data object in the flash memory are added to the data object information table.
[0088] If the first data object exists in the data object information table, it means that the current data storage instruction is to update the existing data object. At this time, the version number of the identification field of the first data object needs to be updated, that is, the version number of the identification field of the first data object is increased by one, and then the first data object and the identification field of the first data object (including identification information and version number) are written into the blank position of the circular buffer. At the same time, the storage location information of the first data object in the circular buffer is obtained, and the storage location information and the identification field are stored in the data object information table in the memory, that is, the identification field of the first data object and the location information in the flash memory are updated in the data object information table. Therefore, the data object information table maintains the latest version of the data object.
[0089] Figure 2 Schematic diagram of writing a data object into a circular buffer provided by some embodiments of the present application. Figure 2 As shown, the first data object includes data object A, data object B and data object C. Among them, data object A and data object B are existing data objects, so when storing data object A and data object B, their version numbers need to be updated, and data object C is a new data object, and its version number is initially set to 1. Data objects A, B, C and the corresponding identification fields (i.e. Figure 2 The ID and version number in the flash memory are written into the blank positions of the circular buffer in sequence, and the identification fields of data objects A and B and the location information of data objects A and B in the flash memory are updated in the data object information table, and the identification field of data object C and the location information of data object C in the flash memory are added to the data object information table.
[0090] The data processing method provided in the embodiment of the present application combines the data object information table and the circular buffer to add version information to the data object and write it sequentially into the blank positions of the flash memory, thereby recycling the entire flash memory space, improving the utilization rate of the flash memory space, and ensuring wear leveling.
[0091] In some embodiments, the circular buffer includes: a header and a tail, and a blank position between the header and the tail of the circular buffer can store a data object of a preset data length;
[0092] Writing the first data object and the identification field into a blank position of a circular buffer comprises:
[0093] The first data object and the identification field are written into a blank position at the head of the circular buffer.
[0094] To ensure wear leveling, the entire available flash memory space is regarded as a circular buffer. When storing data, the data objects are written sequentially to the blank locations of the flash memory, that is, the head of the circular buffer. As data is stored, the tail of the circular buffer is always the oldest version of the data.
[0095] In some embodiments, the method further comprises:
[0096] When the first data object cannot be written into the blank position at the head of the circular buffer, the target position at the tail of the circular buffer is erased.
[0097] When the blank space in the flash memory is insufficient to write a new data object, the page at the end of the circular buffer needs to be erased. The larger the available space in the flash memory, the more likely it is that a new version of the data object will be inserted at the head of the circular buffer. When the page at the end of the circular buffer is erased, fewer data objects need to be copied or no data objects need to be copied, which greatly improves the flash memory space utilization and write efficiency and ensures wear leveling.
[0098] It should be noted that in flash memory, a page is the basic unit for storage operations, representing a storage area of a fixed size. Page operations usually include reading, writing, and erasing, among which the erase operation needs to be performed at the block level, and the write operation is to write data to a new page.
[0099] Furthermore, in order to successfully store data, it is necessary to ensure that there are enough blank pages between the head and the tail of the circular buffer. To improve the utilization rate of the available space of the flash memory, the blank page can store the largest data object. Figure 3 Schematic diagram of a circular buffer provided by some embodiments of the present application. Figure 3 As shown, the blank page only needs to be able to store the largest data object, which can improve the utilization rate of the available space of the flash memory.
[0100] In some embodiments, the method further comprises:
[0101] receiving a data erasing instruction, wherein the data erasing instruction is used to instruct erasing a target flash memory page;
[0102] In response to the data erase instruction, acquiring address information corresponding to the target flash memory page;
[0103] According to the address information, query the data object information table to determine whether there is a data object of the latest version in the target flash memory page;
[0104] If not present, erasing the target flash memory page;
[0105] Alternatively, if it exists, the latest version of the data object is copied to the new blank flash memory page in sequence, the storage location information corresponding to the latest version of the data object in the data object information table is updated, and the target flash memory page is erased.
[0106] It can be understood that a data erase instruction is received, and the data erase instruction is used to indicate the erasure of the target flash memory page, indicating that the target flash memory page needs to be erased. It is determined based on the data information table whether the latest version of the data object exists in the target flash memory page. If so, the data object is copied to a new blank page in sequence, and the storage location information corresponding to the latest version of the data object is updated in the data object information table, and finally the page is erased.
[0107] The embodiment of the present application obtains the address information corresponding to the target flash memory page, queries the data object information table according to the address information, and determines whether the latest version of the data object exists in the target flash memory page.
[0108] If there is a data object in the data object information table whose storage location information is within the range of the address information, it means that the latest version of the data object exists in the target flash memory page. The latest version of the data object is copied to the new blank flash memory page in sequence, and the storage location information corresponding to the latest version of the data object in the data object information table is updated, and the target flash memory page is erased.
[0109] If there is no data object in the data object information table whose storage location information is within the range of the address information, it means that the latest version of the data object does not exist in the target flash memory page, and the target flash memory page is directly erased.
[0110] Figure 4 Schematic diagram of erasing data objects provided by some embodiments of the present application. Figure 4 As shown, the latest versions of data object C (V1), data object A (V3) and data object B (V3) exist in flash page 1. First, data object C (V1), data object A (V3) and data object B (V3) are copied to a blank flash page. Figure 4The flash memory page 2 is in the middle, the storage location information corresponding to the latest version of the data object in the data object information table is updated, and the flash memory page 1 is erased.
[0111] In an embodiment of the present application, when a data erase instruction is received, it is first determined whether the latest version of the data object exists in the target flash memory page to be erased. If so, it is copied first and then erased. If not, it is erased directly. This ensures that the old version of the data object is always erased. In most cases, there is no need to copy or only a few data objects in the target flash memory page to be erased are copied, thereby ensuring write efficiency.
[0112] In some embodiments, the data object information table further includes: a memory backup flag, the memory backup flag being used to indicate whether the data object is backed up in the memory;
[0113] After writing the first data object and the identification field into a circular buffer, the method further comprises:
[0114] storing the first data object in a memory;
[0115] The memory backup flag bit of the first data object in the data object information table is set to a first value.
[0116] To improve data reading efficiency, the value of the data object (that is, a backup of the data object itself) can be stored in memory.
[0117] Optionally, in order to save memory, after M (such as 32) data objects with a length less than N (such as 128 bytes) are written to the flash memory, the value of the data object is retained in the memory for backup. A memory backup flag is added to the data object information table to indicate whether the data object is retained in the memory for backup.
[0118] After writing the first data object and the identification field into the circular buffer, the method further includes: storing the first data object in the memory; setting the memory backup flag of the first data object in the data object information table to a first value, wherein the first value is used to indicate that the data object has a backup in the memory.
[0119] In the embodiment of the present application, by adding a memory backup flag in the data object information table, the data reading efficiency can be improved.
[0120] In some embodiments, the method further comprises:
[0121] receiving a data read instruction, wherein the data read instruction is used to instruct to read a second data object;
[0122] In response to the data read instruction, determining whether the second data object exists in the memory through a memory backup flag in the data object information table;
[0123] If it exists, read the second data object from the memory;
[0124] or, if it does not exist, obtaining storage location information of the second data object based on the data object information table;
[0125] Based on the storage location information of the second data object, the second data object is read from the circular buffer, and the second data object is stored in the memory.
[0126] It can be understood that when reading data, the position of the data object is located through the data object information table. If there is a copy in the memory, it is read directly from the memory. If not, it is read from the physical flash memory according to the flash address information, and a copy of the data object is backed up in the memory.
[0127] In an embodiment of the present application, when data needs to be read, the data object is located through the data object information table. If there is a copy in the memory, it is read directly from the memory. If not, it is read from the physical flash memory, which can improve data reading efficiency.
[0128] In some embodiments, the method further comprises:
[0129] When the number of data objects backed up in the memory exceeds a preset value, at least one data object that is least recently used and backed up in the memory is deleted based on a least recently used replacement algorithm.
[0130] When the number of data objects backed up in the memory exceeds the M value, a data object value needs to be replaced, and an LFU (Least Frequently Used) replacement algorithm can be used to delete at least one data object that has been least recently used in the memory backup.
[0131] A counter is set for each data object to record the frequency of the data object's value being used. Each time the data object is read or written, the counter value is increased by one. When the number of data objects backed up in memory exceeds the M value, the data object information table is searched to find the data object with the smallest counter value, its memory backup is deleted, and then the new data object is backed up in memory.
[0132] In the embodiment of the present application, by retaining copies of some commonly used data objects in the memory, when reading the data object, the position of the data object is located by querying the data object information table, thereby improving the data reading efficiency.
[0133] The data processing method provided by the embodiment of the present application uses the available space of the flash memory as a circular buffer, writes the data object and the identification field containing the version number into the blank area in sequence, the latest version of the data object is always stored at the head of the circular buffer, the oldest version of the data object is always at the tail of the circular buffer, and a blank page that can store the largest data object is reserved between the head and the tail. When the page at the tail of the circular buffer is erased, there is no need to copy or only a few latest versions of the data objects in the page are copied, which greatly improves the flash memory space utilization and writing efficiency, and ensures wear leveling. A latest data object information table is maintained in the memory, and M copies of the data object values are kept in the memory at the same time. When reading data, it is determined whether there is a copy in the memory by looking up the table. If there is a copy in the memory, it is read directly. If not, it is read from the physical flash memory according to the flash address information, and a copy of the data object value is backed up in the memory. Only when the number of data objects backed up in the memory exceeds the M value, a specific algorithm is used to replace it, which saves memory and improves reading efficiency.
[0134] The data processing method provided in the embodiment of the present application can be executed by a data processing device. In the embodiment of the present application, the data processing device provided in the embodiment of the present application is described by taking the data processing method executed by the data processing device as an example.
[0135] An embodiment of the present application also provides a data processing device. Figure 5 is a schematic diagram of the structure of a data processing device provided in some embodiments of the present application, such as Figure 5 As shown, the data processing device 500 includes: a receiving unit 510, a data storage unit 520 and an information storage unit 530, wherein:
[0136] A receiving unit 510 is configured to receive a data storage instruction, where the data storage instruction is used to instruct to store a first data object;
[0137] A data storage unit 520, configured to obtain an identification field of the first data object in response to the data storage instruction, and write the first data object and the identification field into a circular buffer, wherein the identification field includes a version number and identification information;
[0138] The information storage unit 530 is used to obtain the storage location information of the first data object in the circular buffer, and store the storage location information and the identification field in a data object information table in the memory.
[0139] Optionally, the data storage unit 520 is used to:
[0140] Acquire identification information of the first data object, and query the data object information table according to the identification information whether the first data object exists;
[0141] If not, initialize the version number of the first data object, and write the first data object and the identification field into a blank position of the circular buffer;
[0142] If it exists, the version number of the first data object is increased by one, and the first data object and the identification field are written into a blank position of the circular buffer.
[0143] Optionally, the circular buffer includes: a header and a tail, and a blank position between the header and the tail of the circular buffer can store a data object of a preset data length;
[0144] Writing the first data object and the identification field into a blank position of a circular buffer comprises:
[0145] The first data object and the identification field are written into a blank position at the head of the circular buffer.
[0146] In some embodiments, the apparatus further comprises a first processing unit configured to:
[0147] When the first data object cannot be written into the blank position at the head of the circular buffer, the target position at the tail of the circular buffer is erased.
[0148] In some embodiments, the apparatus further comprises a second processing unit configured to:
[0149] receiving a data erasing instruction, wherein the data erasing instruction is used to instruct erasing a target flash memory page;
[0150] In response to the data erase instruction, acquiring address information corresponding to the target flash memory page;
[0151] According to the address information, query the data object information table to determine whether there is a data object of the latest version in the target flash memory page;
[0152] If not present, erasing the target flash memory page;
[0153] Alternatively, if it exists, the latest version of the data object is copied to the new blank flash memory page in sequence, the storage location information corresponding to the latest version of the data object in the data object information table is updated, and the target flash memory page is erased.
[0154] In some embodiments, the data object information table further includes: a memory backup flag, the memory backup flag being used to indicate whether the data object is backed up in the memory;
[0155] The device further comprises a third processing unit, configured to:
[0156] storing the first data object in a memory;
[0157] The memory backup flag bit of the first data object in the data object information table is set to a first value.
[0158] In some embodiments, the third processing unit is further configured to:
[0159] receiving a data read instruction, wherein the data read instruction is used to instruct to read a second data object;
[0160] In response to the data read instruction, determining whether the second data object exists in the memory through a memory backup flag in the data object information table;
[0161] If it exists, read the second data object from the memory;
[0162] or, if it does not exist, obtaining storage location information of the second data object based on the data object information table;
[0163] Based on the storage location information of the second data object, the second data object is read from the circular buffer, and the second data object is stored in the memory.
[0164] In some embodiments, the third processing unit is further configured to:
[0165] When the number of data objects backed up in the memory exceeds a preset value, at least one data object that is least recently used and backed up in the memory is deleted based on a least recently used replacement algorithm.
[0166] The data processing device 500 in the embodiment of the present application can be an electronic device or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.
[0167] The data processing device 500 in the embodiment of the present application may be a device having an operating system. The operating system may be a Microsoft (Windows) operating system, an Android (Android) operating system, an IOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.
[0168] The data processing device 500 provided in the embodiment of the present application can realize Figures 1 to 4 To avoid repetition, the various processes implemented by the method embodiment are not described here.
[0169] In some embodiments, Figure 6 As shown, an embodiment of the present application also provides an electronic device 600, including a processor 601, a memory 602, and a computer program stored in the memory 602 and executable on the processor 601. When the program is executed by the processor 601, each process of the above-mentioned data processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.
[0170] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.
[0171] An embodiment of the present application also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned data processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
[0172] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.
[0173] An embodiment of the present application also provides a computer program product, including a computer program, which implements the above-mentioned data processing method when executed by a processor.
[0174] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.
[0175] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned data processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.
[0176] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.
[0177] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.
[0178] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the relevant technology, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.
[0179] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.
[0180] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.
[0181] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.
Claims
1. A data processing method, characterized in that: include: receiving a data storage instruction, wherein the data storage instruction is used to instruct to store a first data object; In response to the data storage instruction, obtaining an identification field of the first data object, and writing the first data object and the identification field into a circular buffer, wherein the identification field includes a version number and identification information; The storage location information of the first data object in the circular buffer is obtained, and the storage location information and the identification field are stored in a data object information table in the memory.
2. The data processing method according to claim 1, characterized in that: The step of obtaining the identification field of the first data object in response to the data storage instruction and writing the first data object and the identification field into a circular buffer comprises: Acquire identification information of the first data object, and query the data object information table according to the identification information whether the first data object exists; If not, initialize the version number of the first data object, and write the first data object and the identification field into a blank position of the circular buffer; If it exists, the version number of the first data object is increased by one, and the first data object and the identification field are written into a blank position of the circular buffer.
3. The data processing method according to claim 2, wherein the circular buffer comprises: A header and a tail, wherein the blank space between the header and the tail of the circular buffer can store a data object of a preset data length; Writing the first data object and the identification field into a blank position of a circular buffer comprises: The first data object and the identification field are written into a blank position at the head of the circular buffer.
4. The data processing method according to claim 3, characterized in that: The method further comprises: When the first data object cannot be written into the blank position at the head of the circular buffer, the target position at the tail of the circular buffer is erased.
5. The data processing method according to any one of claims 1 to 4, characterized in that: The method further comprises: receiving a data erasing instruction, wherein the data erasing instruction is used to instruct erasing a target flash memory page; In response to the data erase instruction, acquiring address information corresponding to the target flash memory page; According to the address information, query the data object information table to determine whether there is a data object of the latest version in the target flash memory page; If not present, erasing the target flash memory page; Alternatively, if it exists, the latest version of the data object is copied to the new blank flash memory page in sequence, the storage location information corresponding to the latest version of the data object in the data object information table is updated, and the target flash memory page is erased.
6. The data processing method according to claim 1, characterized in that: The data object information table also includes: a memory backup flag, the memory backup flag is used to indicate whether the data object is backed up in the memory; After writing the first data object and the identification field into a circular buffer, the method further comprises: storing the first data object in a memory; The memory backup flag bit of the first data object in the data object information table is set to a first value.
7. The data processing method according to claim 6, characterized in that: The method further comprises: receiving a data read instruction, wherein the data read instruction is used to instruct to read a second data object; In response to the data read instruction, determining whether the second data object exists in the memory through a memory backup flag in the data object information table; If it exists, read the second data object from the memory; or, if it does not exist, obtaining storage location information of the second data object based on the data object information table; Based on the storage location information of the second data object, the second data object is read from the circular buffer, and the second data object is stored in the memory.
8. The data processing method according to claim 6 or 7, characterized in that: The method further comprises: When the number of data objects backed up in the memory exceeds a preset value, at least one data object that is least recently used and backed up in the memory is deleted based on a least recently used replacement algorithm.
9. A data processing device, characterized in that: include: A receiving unit, configured to receive a data storage instruction, wherein the data storage instruction is used to instruct to store a first data object; A data storage unit, configured to obtain an identification field of the first data object in response to the data storage instruction, and write the first data object and the identification field into a circular buffer, wherein the identification field includes a version number and identification information; The information storage unit is used to obtain the storage location information of the first data object in the circular buffer, and store the storage location information and the identification field in a data object information table in the memory.
10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the data processing method according to any one of claims 1 to 8 is implemented.