Data writing method, data reading method, device, equipment and storage medium

Abstract

The present application provides a data writing method, a data reading method, a device, a device and a storage medium, relating to the field of data storage. The writing and reading methods are applied to an electronic device including a file system. The file system includes a first index block, a dynamic index block and user data blocks. The user data blocks are used to record user data; the index shards are used to record index information of the user data; the first index block is used to record the location information of the index shards. The writing method includes: obtaining user data and index information of the user data; determining a target user data block according to the user identifier of the user data; writing the user data into the target user data block, and writing the index information of the user data into a first index shard in the dynamic index block; the location information of the first index shard is recorded in the first index block corresponding to the target user data block. The writing method is applicable to the data writing process and is used to solve the problem of wasting storage space when writing small file data.

Description

Technical Field

[0001] This application relates to the field of data storage, and in particular, to a data writing method, a data reading method, a device, a device and a storage medium. Background Art

[0002] A file system refers to an abstract data type (Abstract data type) that implements operations such as data storage, hierarchical organization, access, and retrieval in a computer. The file system may include a data area and an index area. The data area includes a plurality of data blocks for storing user data, and the index area includes a plurality of index blocks for recording index information of the user data stored in the data blocks.

[0003] Currently, the size of the index area and the size of the index blocks are both fixed.

[0004] However, with the development of the intelligent Internet of Things, there will be a large demand for storing a large amount of real-time small file data. Since the size of the index blocks is fixed, each data block recording real-time small file data requires an index block to record the index information of the real-time small file data stored. There may be a situation where the space of the index blocks is full but the space of the data blocks is not full, resulting in a waste of storage space. Summary of the Invention

[0005] Based on the above technical problems, this application provides a data writing method, a data reading method, a device, a device and a storage medium. The writing method can use location information that occupies less storage space to replace index information, so as to make full use of the space of the index blocks and avoid the index blocks being filled before the user data blocks are full, resulting in a waste of data block space.

[0006] In a first aspect, this application provides a data writing method, which includes: obtaining user data and index information of the user data; determining a target user data block according to the user identifier of the user data; the target user data block is one of at least one user data block included in the user data area; the user data area is a partition in the file system for storing user data; the file system further includes an index area and a dynamic data area; the index area includes at least one first index block, the dynamic data area includes at least one dynamic index block, the dynamic index block includes one or more index shards; the index shards are used to record index information of the user data; the first index block is used to record the location information of the index shards; writing the user data into the target user data block, and writing the index information of the user data into the first index shard in the dynamic index block; the first index shard is one of the one or more index shards included in the dynamic index block, and the location information of the first index shard is recorded in the first index block corresponding to the target user data block.

[0007] In a possible implementation, determining a target user data block according to the user identifier of user data includes: querying whether there is an unfilled user data block corresponding to the user identifier according to the user identifier of the user data; if so, using the unfilled user data block as the target user data block; if so, allocating the target user data block to the user corresponding to the user identifier.

[0008] Optionally, the method further includes: determining the index shard corresponding to the user data block to be deleted according to the position information of the index shards recorded in the first index block; marking the position information of the index shard corresponding to the user data block to be deleted recorded in the first index block as deleted; for a dynamic index block, deleting the position information in the first index block according to the deletion marks of the index shards in the dynamic index block, and releasing the disk space corresponding to the position information.

[0009] Optionally, the file system further includes a second index block; the second index block is used to record the position information of the index shards that record the index information of the user data.

[0010] Optionally, the file system further includes a boot block and a backup boot block; both the boot block and the backup boot block are used to record the size of the data block, the size of the first index block, and the size of the index shard.

[0011] Optionally, the file system further includes a second index block; the second index block is used to record the position information of the index shards that record the index information of the user data, and the method further includes: obtaining formatting parameters; the formatting parameters include the size of the boot block, the size of the index block, and the upper limit number of data blocks in a single block group; the index block includes the first index block and the second index block; the data block is any one of the index block, the dynamic index block, and the user data block; obtaining the capacity of the memory; calculating the number of data blocks, the size of a single block group, and the number of block groups according to the formatting parameters and the capacity of the memory; determining the initial information of the index block according to the number of data blocks, the size of the block group, and the number of block groups; the initial information of the index block includes the unique identifier and the position information of the index block; writing the initial information of the index block into the storage spaces corresponding to the boot block and the backup boot block in the memory respectively to complete the formatting of the file system.

[0012] By using the data writing method provided in this application, after obtaining the user data and the index information of the user data, the user data can be written into the user data block, and the index information of the user data can be written into the index shards in the dynamic index block. The original first index block with a fixed size can record the position information of the index shards. Compared with the index information, the position information occupies less storage space. Using the position information that occupies less storage space to replace the index information can make full use of the space of the index block, avoid the index block from being filled before the user data block is full, and cause waste of the data block space.

[0013] In addition, in the file system provided by this application, the specific position of the dynamic index block is not fixed and can be set in different areas of the memory. When the electronic device performs read and write operations on the file system, it does not perform centralized operations on a certain point of the disk, thereby effectively avoiding the disk hot spot problem.

[0014] In a second aspect, this application provides a data writing device, which includes each module for the method described in the first aspect above.

[0015] In a third aspect, this application provides a data reading method, which includes: obtaining a target user identifier corresponding to the user data to be read; determining a target user data block according to the target user identifier; the target user data block is one of at least one user data block included in the user data area; the user data area is a partition in the file system for storing user data; the file system further includes an index area and a dynamic data area; the index area includes at least one first index block, the dynamic data area includes at least one dynamic index block, and the dynamic index block includes one or more index shards; the index shard is used to record the index information of the user data; the first index block is used to record the position information of the index shard; determining the user data to be read in the target user data block according to the target user identifier; determining the first index block corresponding to the target user data block according to the target user identifier; querying the first index block corresponding to the target user data block according to the target user identifier to determine the position information of the first index shard storing the index information of the user data to be read; the first index shard is one of the one or more index shards included in the dynamic index block, and the position information of the first index shard is recorded in the first index block corresponding to the target user data block; based on the position information of the first index shard, performing a read operation on the index information of the user data to be read and the user data to be read.

[0016] In a fourth aspect, this application provides a data reading device, which includes each module for the method described in the third aspect above.

[0017] In a fifth aspect, this application provides a computer program product, which, when running on a computer, causes the computer to execute the steps of the related method described in the first aspect above to implement the method described in the first aspect or the third aspect above.

[0018] In a sixth aspect, this application provides an electronic device, which includes: a processor and a memory; the memory stores instructions executable by the processor; when the processor is configured to execute the instructions, the electronic device implements the method described in the first aspect or the third aspect above.

[0019] In a seventh aspect, the present application provides a computer-readable storage medium, which includes computer software instructions; when the computer software instructions run in an electronic device, the electronic device implements the method described in the first aspect or the third aspect above.

[0020] The beneficial effects of the second aspect to the seventh aspect above can be referred to those described in the first aspect and will not be elaborated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a schematic structural diagram of a file system;

[0022] Figure 2 It is a schematic diagram of the relationship between an index block and a data block;

[0023] Figure 3 It is a schematic structural diagram of a file system provided by an embodiment of the present application;

[0024] Figure 4 It is a schematic diagram of the relationship between an index block and a data block provided by an embodiment of the present application;

[0025] Figure 5 It is another schematic structural diagram of a file system provided by an embodiment of the present application;

[0026] Figure 6 It is yet another schematic structural diagram of a file system provided by an embodiment of the present application;

[0027] Figure 7 It is yet another schematic structural diagram of a file system provided by an embodiment of the present application;

[0028] Figure 8 It is yet another schematic composition diagram of a file system provided by an embodiment of the present application;

[0029] Figure 9 It is a schematic diagram of the composition of an electronic device provided by an embodiment of the present application;

[0030] Figure 10 It is a schematic flowchart of a method for formatting a file system of an electronic device provided by an embodiment of the present application;

[0031] Figure 11 It is another schematic flowchart of a formatting method provided by an embodiment of the present application;

[0032] Figure 12 It is yet another schematic flowchart of a formatting method provided by an embodiment of the present application;

[0033] Figure 13 It is a schematic flowchart of a data writing method provided by an embodiment of the present application;

[0034] Figure 14Schematic flowchart of the data reading method provided by an embodiment of this application;

[0035] Figure 15 Schematic flowchart of the data deletion method provided by an embodiment of this application;

[0036] Figure 16 Schematic diagram of the composition of the data writing device provided by an embodiment of this application;

[0037] Figure 17 Schematic diagram of the composition of the data reading device provided by an embodiment of this application. Detailed implementation manners

[0038] Hereinafter, terms such as "first", "second", and "third" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first", "second", or "third" etc. may explicitly or implicitly include one or more of such features.

[0039] For ease of understanding, first, the terms involved in an embodiment of this application are briefly explained.

[0040] 1. Data block: A data unit for data transfer between the main memory and input / output devices, or between the main memory and external memories. The size of the data block can be fixed or variable.

[0041] 2. Large file data: Refers to file data that, taking a certain data size (such as 64 megabytes (M) or 32M, etc.) as a boundary, is larger than this boundary, such as wmv files, rmvb files, mp4 files, avi files, or flv files, etc.

[0042] 3. Small file data: In contrast to large file data, file data that is smaller than a certain data size boundary, such as jpg files, txt files, doc files, html files, or js files, etc.

[0043] Figure 1 Is a schematic diagram of the file system structure. As Figure 1 shown, the file system may include an index area and a data area. The index area may include multiple index blocks. The data area may include multiple data blocks. The data blocks may include user data blocks. The user data blocks may be used to record user data. The index blocks may be used to record index information of user data. The index information may be used to indicate the attributes of user data. The data blocks and the index blocks are in one-to-one correspondence.

[0044] It should be noted that the above Figure 1The regions therein are logical regions, and the blocks are logical blocks. In practice, there is a mapping relationship between the logical regions (and logical blocks) in the file system and the physical storage space in the storage medium. Thus, the data stored in the logical regions (and logical blocks) is actually stored in the physical storage space in the storage medium that has a mapping relationship with the logical regions (and logical blocks).

[0045] Exemplarily, Figure 2 is a schematic diagram of the relationship between the index block and the data block. As Figure 2 shown, taking the special behavior monitoring image with user data as the crossroads as an example, and the index information of the special behavior monitoring image includes the type, location, license plate number of the behavior subject, and time of the special behavior. Suppose the index area includes index block 1 and index block 2, the data area includes data block 1 and data block 2, and both data block 1 and data block 2 are user data blocks for recording user data. Then, data block 1 can record special behavior monitoring images Picture 1, Picture 2, Picture 3, and Picture 4. In index block 1 corresponding to data block 1, it can record that the type of the special behavior corresponding to Picture 1 is running a red light, the location where the special behavior occurs is intersection 1, the license plate number of the behavior subject of the special behavior is license plate number 1, and the occurrence time of the special behavior is time 1; the type of the special behavior corresponding to Picture 2 is running a red light, the location where the special behavior occurs is intersection 2, the license plate number of the behavior subject of the special behavior is license plate number 2, and the time is time 2; the type of the special behavior corresponding to Picture 3 is running a red light, the location where the special behavior occurs is intersection 3, the license plate number of the behavior subject of the special behavior is license plate number 3, and the time is time 3; the type of the special behavior corresponding to Picture 4 is running a red light, the location where the special behavior occurs is intersection 4, the license plate number of the behavior subject of the special behavior is license plate number 4, and the time is time 4. Data block 2 can record special behavior monitoring images Picture 5, Picture 6, Picture 7, and Picture 8. In index block 2 corresponding to data block 2, it can record that the type of the special behavior corresponding to Picture 5 is crossing the solid line, the location where the special behavior occurs is intersection 5, the license plate number of the behavior subject of the special behavior is license plate number 5, and the time is time 5; the type of the special behavior corresponding to Picture 6 is crossing the solid line, the location where the special behavior occurs is intersection 6, the license plate number of the behavior subject of the special behavior is license plate number 6, and the time is time 6; the type of the special behavior corresponding to Picture 7 is crossing the solid line, the location where the special behavior occurs is intersection 7, the license plate number of the behavior subject of the special behavior is license plate number 7, and the time is time 7; the type of the special behavior corresponding to Picture 8 is crossing the solid line, the location where the special behavior occurs is intersection 8, the license plate number of the behavior subject of the special behavior is license plate number 8, and the time is time 8.

[0046] When all the user data recorded in the file system is large file data, the size of the large file data recorded in the data block is much larger than the size of the index information of the large file data recorded in the index block. There is no problem using a fixed-size index block to record the index information of the large file data.

[0047] However, with the development of the intelligent Internet of Things, there will be a large demand for storing a large number of real-time small file data. Since the size of the index block is fixed, each real-time small file data requires an index block to record the index information for storing the real-time small file data. There may be a situation where the space of the index block is full but the space of the data block corresponding to the index block is not full, resulting in a waste of space.

[0048] On this basis, the embodiment of the present application provides a file system. This file system can use dynamic index sharding to record the index information of user data, and use index blocks to record the location information of the dynamic index sharding. The location information recorded in the fixed-size index block is relatively small. Compared with the index information, the index block can record more location information, thus avoiding the waste of data block space caused by the index block being filled up in advance.

[0049] The following is an introduction in conjunction with the accompanying drawings.

[0050] Figure 3 It is a schematic structural diagram of the file system provided by the embodiment of the present application. As Figure 3 shown, the file system may include an index area and a data area. The index area may include one or more first index blocks. The data area may include one or more data blocks. The data blocks can be of two types, one is a user data block, and the other is a dynamic index block ( Figure 3 using the dotted arrows as examples to point out these two types of data blocks respectively). The dynamic index block may include one or more index shards. The user data block can be used to record user data, such as the above-mentioned large file data or small file data, etc. The index shard can be used to record the index information of the user data. The first index block can be used to record the location information of the index shard that has recorded the index information of the user data.

[0051] For example, assume that 1 kilobyte (KB) is reserved in the first index block to record the location information of the index shard that has recorded the index information of the user data, and each location information of the index shard that has recorded the index information of the user data requires 8 bytes (B) for storage. Then each first index block can record at most 128 (1024 / 8) index shards.

[0052] Exemplarily, Figure 4 It is a schematic diagram of the relationship between the index block and the data block provided by the embodiment of the present application. As Figure 4As shown, taking the special behavior monitoring image with user data as the crossroads as an example, assume that data block 1 in the data area is a user data block, and the user data block records user data pictures 9, 10, 11, and 12. Also, data block 2 in the data area is a dynamic index block, and the dynamic index block includes a total of N index shards, namely index shard 1, index shard 2, index shard 3, …, and index shard N. Then, in the first index block 1 in the index area, it can record that the type of the special behavior corresponding to picture 9 is running a red light, the location where the special behavior occurs is intersection 9, the license plate number of the behavior subject of the special behavior is license plate number 9, and the occurrence time of the special behavior is time 9; the type of the special behavior corresponding to picture 10 is running a red light, the location where the special behavior occurs is intersection 10, the license plate number of the behavior subject of the special behavior is license plate number 10, and the occurrence time of the special behavior is time 10; the index shard corresponding to picture 11 is index shard 1 in data block 2 (dynamic index block); the index shard corresponding to picture 12 is index shard 2 in data block 2 (dynamic index block). In index shard 1 in data block 2 (dynamic index block), it can record that the type of the special behavior corresponding to picture 11 is running a red light, the location where the special behavior occurs is intersection 11, the license plate number of the behavior subject of the special behavior is license plate number 11, and the occurrence time of the special behavior is time 11; the type of the special behavior corresponding to picture 12 is running a red light, the location where the special behavior occurs is intersection 12, the license plate number of the behavior subject of the special behavior is license plate number 12, and the occurrence time of the special behavior is time 12.

[0053] In some possible embodiments, based on Figure 3 the file system shown, the file system may further include a backup index area. Referring to Figure 5 , Figure 5 is another structural schematic diagram of the file system provided by this application embodiment. As shown in Figure 5 , the backup index area may include one or more second index blocks. The second index block can also be used to record the location information of the index shard that records the index information of the user data. When the location information of the index shard that records the index information of the user data cannot be queried through the first index block (for example, the physical storage space corresponding to the first index block is damaged), the location information of the index shard that records the index information of the user data in the second index block can be queried. In this case, the first index block can be regarded as the main index block, and the second index block can be regarded as the backup index block.

[0054] In some other possible embodiments, based on Figure 3 or Figure 5 the file system shown, the file system may further include one or more block groups, and a boot block. The above-mentioned index area and data area can be two partitions in a block group. Taking Figure 3 as an example, referring toFigure 6 , Figure 6 is another schematic structural diagram of the file system provided by the embodiment of the present application. As Figure 6 shown, the boot block can be used to record the basic information of the disk, the sizes of the data blocks (the above-mentioned user data blocks and dynamic index blocks), the size of the first index block, the size of the index shard, and the division rules of the block group, etc. Among them, the basic information of the disk can include the capacity and unique identifier of the disk (such as the serial number (SN)), etc. The division rules of the block group are used to divide the sizes of the index area and the data area.

[0055] In some other possible embodiments, based on the Figure 6 file system shown, the file system may further include a backup boot block. Referring to Figure 7 , Figure 7 is another schematic structural diagram of the file system provided by the embodiment of the present application. As Figure 7 shown, the backup boot block can also be used to record the basic information of the disk, the sizes of the data blocks (the above-mentioned user data blocks and dynamic index blocks), the size of the first index block, the size of the index shard, and the division rules of the block group, etc. When the basic information of the disk, the sizes of the data blocks, the size of the first index block, the size of the index shard, and the division rules of the block group cannot be queried through the boot block (for example, the physical storage space corresponding to the boot block is damaged), the backup boot block can be used for query.

[0056] In some other possible embodiments, after combining the Figures 3 to 7 file system shown, the file system may simultaneously include a second index block, a boot block, a block group, and a backup boot block. Figure 8 is another schematic composition diagram of the file system provided by the embodiment of the present application. Figure 8 The structural description in Figures 3 to 7 can be referred to as described above at

[0057] and will not be elaborated here. Figures 3 to 8 It should be noted that the above takes an example where the dynamic index block is one of the data blocks set in the data area for introduction. Optionally, the dynamic index block can also be set in other partitions, such as the index area, or the file system may further include other partitions (for example, this other partition can be called a reserved area), and the dynamic index block can also be set in this other partition. The above Figure 8The dynamic data area in it can be regarded as a partition in the data area. The partition where the user data block is located can be called the user data area. The user data area can be regarded as another partition in the data area, or it can be understood as the partition for storing user data.

[0058] In an exemplary embodiment, the embodiment of the present application further provides a data writing method, which can be applied to an electronic device including the Figure 8 file system shown above. The electronic device can be an electronic device such as a computer or a server with storage and computing functions. Among them, the server can be a single server, or it can also be a server cluster composed of multiple servers. In some embodiments, the server cluster can also be a distributed cluster.

[0059] Optionally, the execution subject of the above data writing method can also be an application (APP) installed in the above electronic device that provides the data writing function; or, the central processing unit (CPU) in the above electronic device; or, the functional module in the above electronic device for executing the data writing method. The embodiment of the present application does not limit this.

[0060] Hereinafter, the execution subject of the data writing method is taken as the above electronic device as an example for introduction.

[0061] Figure 9 It is a schematic diagram of the composition of the electronic device provided by the embodiment of the present application. As Figure 9 shown, the electronic device may include a processor 10, a memory 20, a communication line 30, and a communication interface 40.

[0062] Among them, the processor 10, the memory 20, and the communication interface 40 can be connected through the communication line 30.

[0063] The processor 10 is used to execute the instructions stored in the memory 20 to implement the management method related to the file system provided in the following embodiments of the present application. The processor 10 can be a central processing unit (CPU), a general-purpose processor network processor (NP), a digital signal processor (DSP), a microprocessor, a microcontroller, a programmable logic device (PLD), or any combination thereof. The processor 10 can also be any other device with processing functions, such as a circuit, a device, or a software module. The embodiment of the present application does not limit this. In one example, the processor 10 may include one or more CPUs, for exampleFigure 9 CPU0 and CPU1 therein. As an alternative implementation, the first electronic device may include multiple processors. For example, in addition to processor 10, it may further include processor 50.

[0064] Memory 20, which is used to store instructions. The instructions may be computer programs. Optionally, memory 20 may be a read-only memory (ROM) or other types of static storage devices that can store static information and / or instructions, or it may be a random access memory (RAM) or other types of dynamic storage devices that can store information and / or instructions. It may also be an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM), or other optical disc storage, optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media, or other magnetic storage devices, etc. The embodiments of the present application do not limit this.

[0065] It should be noted that memory 20 may exist independently of processor 10 or may be integrated with processor 10. Memory 20 may be located inside the electronic device or outside the electronic device. The embodiments of the present application do not limit this.

[0066] In the embodiments of the present application, memory 20 may provide physical storage space for any of the Figures 3 to 8 file systems shown above.

[0067] Communication line 30, which is used to transmit information between the components included in the electronic device.

[0068] Communication interface 40, which is used to communicate with other devices or other communication networks. The other communication network may be an Ethernet, a radio access network (RAN), a wireless local area network (WLAN), etc. Communication interface 40 may be a module, a circuit, a transceiver, or any device capable of implementing communication.

[0069] It should be noted that Figure 9 the structure shown in Figure 9 does not constitute a limitation on the electronic device. In addition to the

[0070] First, before describing the data writing method provided by the embodiments of the present application, the process of formatting any file system by an electronic device is introduced first, so that the formatted file system has the structure described above. Figure 8 shown. Refer to Figure 10 , Figure 10 which is a schematic flowchart of a method for formatting a file system by an electronic device provided by an embodiment of the present application. As Figure 10 shown, the method may include S101 to S105.

[0071] S101. The electronic device obtains formatting parameters.

[0072] Among them, the formatting parameters may include the size of the boot block, the size of the data block, the size of the index block, and the upper limit number of data blocks in a single block group.

[0073] In a possible implementation manner, the above S101 may specifically include: the electronic device receives the formatting parameters configured by the user.

[0074] In another possible implementation manner, the above S101 may specifically include: the electronic device receives a formatting operation of the user; the electronic device obtains the pre-stored formatting parameters in response to the formatting operation.

[0075] S102. The electronic device obtains the memory capacity.

[0076] S103. The electronic device calculates the number of data blocks, the size of the block group, and the number of block groups according to the formatting parameters and the memory capacity.

[0077] Figure 11 which is another schematic flowchart of the formatting method provided by an embodiment of the present application. As Figure 11 shown, in a possible implementation manner, the above S103 may include the following S201 to step S204.

[0078] S201. The electronic device calculates the size of a single block group according to the size of the data block, the size of the index block, and the upper limit number of data blocks in a single block group in the formatting parameters.

[0079] Optionally, in combination with Figure 8 , when the block group includes a first index block (main) and a second index block (backup), the specific calculation formula for the electronic device to calculate the size of a single block group is: the size of a single block group = (the size of the data block + the size of the index block * 2) * the upper limit number of data blocks in a single block group.

[0080] S202. The electronic device calculates the capacity of the data space according to the memory capacity and the size of the boot block in the formatting parameters.

[0081] Optionally, in combination with Figure 8 , when the file system includes a boot block and a backup boot block, the specific calculation formula for the electronic device to calculate the capacity of the data space is: Capacity of the data space = Memory capacity - Size of the boot block * 2.

[0082] S203. The electronic device calculates the number of complete block groups based on the capacity of the data space and the size of a single block group.

[0083] Optionally, the specific calculation formula for the electronic device to calculate the number of complete block groups is: Number of complete block groups = Capacity of the data space / Size of a single block group.

[0084] Wherein, when the calculation result includes a decimal, the electronic device rounds down the calculation result. For example, when the calculation result is 10.23, the number of complete block groups is 10.

[0085] S204. The electronic device calculates the number of data blocks based on the number of complete block groups and the upper limit of the number of data blocks in a single block group.

[0086] Optionally, the electronic device can directly use the total number of data blocks in the complete block groups as the number of data blocks.

[0087] Optionally, the electronic device can also calculate the remaining space capacity and determine whether the remaining space capacity meets the sum of the size of a complete index area and the size of one data block. In this case, Figure 12 is another flowchart of the formatting method provided by the embodiments of the present application. As Figure 12 shown, the above S204 may specifically include S301 to S305.

[0088] S301. The electronic device calculates the remaining space capacity based on the data space capacity, the number of complete block groups, and the size of a single block group.

[0089] Optionally, the specific formula for the electronic device to calculate the remaining space capacity is: Remaining space capacity = Data space capacity - Number of complete block groups * Size of a single block group.

[0090] S302. The electronic device determines whether the remaining space capacity meets the size of a complete index area and the size of at least one data block.

[0091] Wherein, since the index blocks and the data blocks are in one-to-one correspondence, the size of a complete index area = Size of the index block * Upper limit of the number of data blocks in a single block group.

[0092] If not satisfied, execute S303; if satisfied, execute S304.

[0093] S303. The electronic device takes all the remaining space as reserved space.

[0094] Among them, the reserved space refers to the invalid storage space that does not meet the storage requirements of the data block.

[0095] S304. The electronic device calculates the number of data blocks in the unfilled block group according to the remaining space capacity, the size of the index block, the upper limit of the data blocks in a single block group, and the size of the data block.

[0096] Optionally, the specific formula for the electronic device to calculate the number of data blocks in the unfilled block group is: the number of data blocks in the unfilled block group = (remaining space - size of the index block * 2 * upper limit of the data blocks in a single block group) / size of the data block.

[0097] It can be understood that when the electronic device executes S303, that is, the number of data blocks in the unfilled block group is 0.

[0098] S305. The electronic device calculates the number of data blocks according to the number of complete block groups, the upper limit number of data blocks in a single block group, and the number of data blocks in the unfilled data block group.

[0099] Optionally, the specific formula is: the number of data blocks = number of complete block groups * upper limit number of data blocks in a single block group + number of data blocks in the unfilled data block group.

[0100] S104. The electronic device determines the initial information of the index block according to the number of data blocks, the size of the block group, and the number of block groups.

[0101] Among them, the initial information of the index block may include the unique identifier and location information of the index block, etc.

[0102] S105. The electronic device writes the initial information of the index block into the corresponding storage space of the boot block in the memory to complete the formatting of the file system.

[0103] Optionally, as described above, the file system may further include a backup boot block. In this case, S105 may specifically include: the electronic device writes the initial information of the index block into the corresponding storage spaces of the boot block and the backup boot block in the memory respectively to complete the formatting of the file system.

[0104] It can be understood that taking the file system shown above Figure 8 as an example, after specifying the location information of the boot block in the memory, the location information of the index area, data area, and backup index area in the block group after the boot block can also be determined according to the number of data blocks. For example, Figure 8For the first block group, the location information of the index area of the block group can be determined by adding the location information of the boot block to the product of the size of the index block and the number of data blocks. The location information of the data area of the block group can be determined by adding the location information of the index area of the block group to the product of the size of the data block and the number of data blocks. The location information of other block groups and the location information of the backup boot block can be referred to that of the first block group, which will not be elaborated here.

[0105] It can be understood that user data can be aggregated in different user data blocks according to different aggregation rules. Taking the user data shown above Figure 2 as an example, the monitoring images of running a red light are recorded in data block 1, and the monitoring images of crossing the solid line are recorded in data block 2. In this case, the aggregation rule can be understood as aggregating according to the types of special behaviors. User data can also be aggregated in different user data blocks according to users. One or more of the above user data blocks are used to record the user data of a user, that is, a user can correspond to one or more of the above user data blocks. Hereinafter, taking the aggregation rule as aggregating according to users as an example, the data writing method provided in the embodiments of the present application will be described. Refer to Figure 13 , Figure 13 which is a schematic flowchart of the data writing method provided in the embodiments of the present application. Optionally, this method can be applied to the above Figure 9 shown electronic device. As Figure 13 shown, this method may include S401 to S403.

[0106] S401. The electronic device obtains user data and index information of the user data.

[0107] Among them, the user data may carry a user identifier.

[0108] In a possible implementation manner, the electronic device may receive the user data and the index information of the user data input by the user.

[0109] In another possible implementation manner, the electronic device may receive the user data and the index information of the user data sent by other devices.

[0110] S402. The electronic device determines a target user data block according to the user identifier of the user data.

[0111] Among them, the target user data block is one of at least one user data block included in the user data area.

[0112] Optionally, the electronic device may query whether there is an unfilled user data block corresponding to the user identifier of the user data according to the user identifier. If there is, the electronic device uses the unfilled user data block as the target user data block; if not, the electronic device allocates a target user data block (new or not written with user data) corresponding to the user identifier.

[0113] S403. The electronic device writes the user data into the target user data block and writes the index information of the user data into the first index shard in the dynamic index block.

[0114] Wherein, the first index shard is one of the one or more index shards included in the dynamic index block, and the location information of the first index shard is recorded in the first index block corresponding to the target user data block.

[0115] Optionally, the electronic device may continuously obtain the user data and the index information of the user data. After an index shard is full, the electronic device writes the index information of the user data into other index shards.

[0116] Optionally, for a dynamic index block, after all the index shards in the dynamic index block are full, the electronic device may divide a new dynamic index block and write the index information of the user data into the index shards in the newly divided dynamic index block.

[0117] By using the data writing method provided in the embodiments of the present application, after obtaining the user data and the index information of the user data, the user data can be written into the user data block, and the index information of the user data can be written into the index shards in the dynamic index block. The original fixed-size first index block can record the location information of the index shards. Compared with the index information, the storage space occupied by the location information is smaller. Using the location information that occupies less storage space to replace the index information can make full use of the space of the index block and avoid the index block from being full before the user data block is full, resulting in waste of the data block space.

[0118] In addition, in the file system provided by the embodiments of the present application, the specific location of the dynamic index block is not fixed and can be set in different areas of the memory. When the electronic device performs read and write operations on the file system, it does not perform centralized operations on a certain point of the memory, thereby effectively avoiding the memory hot spot problem.

[0119] In an exemplary embodiment, the embodiments of the present application further provide a data reading method. Optionally, this method may also be executed by an electronic device having the above Figure 9 shown hardware structure. Figure 14 It is a schematic flowchart of the data reading method provided by the embodiments of the present application. As Figure 14 shown, this method may include S501 to S5 / 6.

[0120] S501. The electronic device obtains the target user identifier corresponding to the user data to be read.

[0121] For example, the electronic device can obtain the target user identifier by receiving the target user identifier input by the user through the above input / output interface, or can also receive the target user identifier sent by other devices through the above communication interface, etc. The embodiments of the present application do not limit this.

[0122] S502. The electronic device determines the target user data block according to the target user identifier.

[0123] Optionally, a corresponding relationship between the user identifier and the user data block can be preset in the electronic device, and one user identifier can correspond to one or more user data blocks. The electronic device can traverse the corresponding relationship between the user identifier and the user data block according to the target user identifier, and determine the user data block corresponding to the target user identifier as the target user data block.

[0124] S503. The electronic device determines the user data to be read in the target user data block according to the target user identifier.

[0125] S503 can refer to that described in S502 above, and will not be elaborated here.

[0126] S504. The electronic device determines the first index block corresponding to the target user data block according to the target user identifier.

[0127] S504 can refer to that described in S502 above, and will not be elaborated here.

[0128] S505. The electronic device queries the first index block corresponding to the target user data block according to the target user identifier, and determines the position information of the first index shard storing the index information of the user data to be read.

[0129] S506. The electronic device performs a read operation on the index information of the user data to be read and the user data to be read based on the position information of the first index shard.

[0130] Among them, performing a read operation on the file system based on the position information can refer to that described in the related art, and will not be elaborated here.

[0131] In an exemplary embodiment, the embodiments of the present application further provide a data deletion method. Optionally, this method can be applied to the above Figure 9 shown electronic device. Figure 15 is a schematic flowchart of the data deletion method provided by the embodiments of the present application. As Figure 15 shown, this method may include S601 to S603.

[0132] S601. The electronic device determines the index shard corresponding to the user data block to be deleted according to the position information of the index shards recorded in the first index block.

[0133] Optionally, before S601, the electronic device may also obtain a deletion instruction for indicating the user data block to be deleted.

[0134] S602. The electronic device marks the position information of the index shard corresponding to the user data block to be deleted recorded in the first index block as deleted.

[0135] S603. For a dynamic index block, the electronic device deletes the position information in the first index block according to the deletion marks of the index shards in the dynamic index block, and releases the disk space corresponding to the position information.

[0136] In a possible implementation, for a dynamic index block, when there is an index shard marked as deleted in the dynamic index block, the electronic device deletes the position information of the index shard recorded in the first index block, and releases the disk space corresponding to the recorded position information.

[0137] In another possible implementation, for a dynamic index block, when the number of index shards marked as deleted in the dynamic index block reaches a preset quantity threshold, the electronic device deletes the position information of the preset quantity threshold of index shards recorded in the first index block, and releases the disk space corresponding to the recorded position information.

[0138] In still another possible implementation, for a dynamic index block, when all the index shards in the dynamic index block are marked as deleted, the electronic device deletes the position information of all the index shards in the dynamic index block recorded in the first index block, and releases the disk space corresponding to the recorded position information.

[0139] The above mainly introduces the solution provided by the embodiments of the present application from the perspective of methods. To implement the above functions, it includes the corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should easily realize that, combining the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in the way of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Skilled professionals can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0140] In an exemplary embodiment, the embodiments of the present application further provide a data writing device, which can be applied to the above Figure 9The electronic device shown. Figure 16 It is a schematic diagram of the composition of the data writing device provided by the embodiment of the present application. As Figure 16 shown, the device includes an acquisition module 1601 and a processing module 1602. The acquisition module 1601 and the processing module 1602 are connected.

[0141] The acquisition module 1601 is used to acquire user data and index information of the user data. The processing module 1602 is used to determine a target user data block according to the user identifier of the user data; the target user data block is one of at least one user data block included in the user data area; the user data area is a partition in the file system for storing user data; the file system further includes an index area and a dynamic data area; the index area includes at least one first index block, the dynamic data area includes at least one dynamic index block, the dynamic index block includes one or more index shards; the index shard is used to record the index information of the user data; the first index block is used to record the location information of the index shard; write the user data into the target user data block, and write the index information of the user data into the first index shard in the dynamic index block; the first index shard is one of the one or more index shards included in the dynamic index block, and the location information of the first index shard is recorded in the first index block corresponding to the target user data block.

[0142] In some possible embodiments, the processing module 1602 is specifically used to query whether there is an unfilled user data block corresponding to the user identifier according to the user identifier of the user data; if so, use the unfilled user data block as the target user data block; if so, allocate a target user data block for the user corresponding to the user identifier.

[0143] In some other possible embodiments, the processing module 1602 is further used to determine the index shard corresponding to the user data block to be deleted according to the location information of the index shard recorded in the first index block; mark the location information of the index shard corresponding to the user data block to be deleted recorded in the first index block as deleted; for a dynamic index block, according to the deletion mark of the index shard in a dynamic index block, delete the location information in the first index block and release the disk space corresponding to the location information.

[0144] In still some other possible embodiments, the file system further includes a second index block; the second index block is used to record the location information of the index shard that records the index information of the user data.

[0145] In still some other possible embodiments, the file system further includes a boot block and a backup boot block; both the boot block and the backup boot block are used to record the size of the data block, the size of the first index block, and the size of the index shard.

[0146] In some other possible embodiments, the file system further includes a second index block; the second index block is used to record the location information of the index shards that record the index information of the user data. The obtaining module 1601 is further configured to obtain formatting parameters; the formatting parameters include the size of the boot block, the size of the index block, and the upper limit number of data blocks in a single block group; the index block includes a first index block and a second index block; the data block can be any one of an index block, a dynamic index block, and a user data block; obtain the capacity of the memory. The processing module 1602 is further configured to calculate the number of data blocks, the size of a single block group, and the number of block groups according to the formatting parameters and the capacity of the memory; determine the initial information of the index block according to the number of data blocks, the size of the block group, and the number of block groups; the initial information of the index block includes the unique identifier and location information of the index block; write the initial information of the index block into the storage spaces corresponding to the boot block and the backup boot block in the memory respectively to complete the formatting of the file system.

[0147] In an exemplary embodiment, the embodiment of the present application further provides a data reading device, which can also be applied to the above Figure 9 shown electronic device. Figure 17 It is a schematic diagram of the composition of the data reading device provided by the embodiment of the present application. As Figure 17 shown, the device includes an obtaining module 1701 and a processing module 1702. The obtaining module 1701 is connected to the processing module 1702.

[0148] The obtaining module 1701 is configured to obtain a target user identifier corresponding to the user data to be read.

[0149] The processing module 1702 is configured to determine a target user data block according to the target user identifier; the target user data block is one of at least one user data block included in the user data area; the user data area is a partition in the file system for storing user data; the file system further includes an index area and a dynamic data area; the index area includes at least one first index block, the dynamic data area includes at least one dynamic index block, the dynamic index block includes one or more index shards; the index shard is used to record the index information of the user data; the first index block is used to record the location information of the index shard; determine the user data to be read in the target user data block according to the target user identifier; determine the first index block corresponding to the target user data block according to the target user identifier; query the first index block corresponding to the target user data block according to the target user identifier to determine the location information of the first index shard that stores the index information of the user data to be read; the first index shard is one of the one or more index shards included in the dynamic index block, and the location information of the first index shard is recorded in the first index block corresponding to the target user data block; based on the location information of the first index shard, perform a read operation on the index information of the user data to be read and the user data to be read.

[0150] It should be noted that Figure 16 and Figure 17 the division of modules in [reference] is illustrative and is only a logical function division. In actual implementation, there may be other division methods. For example, two or more functions can also be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of a software function module.

[0151] In an exemplary embodiment, the embodiment of the present application further provides a computer-readable storage medium, including computer-executable instructions, which when running on an electronic device, cause the electronic device to execute any one of the methods provided in the above embodiments.

[0152] In an exemplary embodiment, the embodiment of the present application further provides a computer program product including computer-executable instructions, which when running on an electronic device, cause the electronic device to execute any one of the methods provided in the above embodiments.

[0153] In an exemplary embodiment, the embodiment of the present application further provides a chip, including: a processor and an interface. The processor is coupled to a memory through the interface. When the processor executes a computer program in the memory or the electronic device executes instructions, any one of the methods provided in the above embodiments is executed.

[0154] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using a software program, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer-executable instructions. When the computer-executable instructions are loaded and executed on a computer, the processes or functions according to the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer-executable instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer-executable instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center in a wired manner (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or a wireless manner (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that the computer can access or a data storage device such as a server or data center that includes one or more integrated media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0155] Although the present application has been described in connection with various embodiments, those skilled in the art will appreciate and realize other variations of the disclosed embodiments by viewing the drawings, the disclosure, and the appended claims in the course of practicing the claimed application. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. A single processor or other unit may implement several functions recited in the claims. Certain measures are recited in mutually different dependent claims, but this does not indicate that these measures cannot be combined to produce favorable results.

[0156] Although the present application has been described in connection with specific features and their embodiments, it will be apparent that various modifications and combinations can be made without departing from the spirit and scope of the application. Accordingly, the specification and drawings are merely exemplary illustrations of the application defined by the appended claims and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of the application. Clearly, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalent technologies, the present application is also intended to include these changes and modifications.

[0157] As described above, the above are only specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A data writing method, characterized in that, The method comprises: Obtaining user data and index information of the user data; Determine a target user data block according to a user identifier of the user data; the target user data block is one of at least one user data block included in a user data area; the user data area is a partition in a file system for storing the user data; the file system further includes an index area and a dynamic data area; the index area includes at least one first index block, the dynamic data area includes at least one dynamic index block, and the dynamic index block includes one or more index shards; the index shards are used to record index information of the user data; the first index block is used to record location information of the index shards; The user data is written into the target user data block, and the index information of the user data is written into the first index shard in the dynamic index block; the first index shard is one of the one or more index shards included in the dynamic index block, and the location information of the first index shard is recorded in the first index block corresponding to the target user data block.

2. The method according to claim 1, wherein The determining the target user data block according to the user identifier of the user data comprises: querying, according to the user identifier of the user data, whether there is an incomplete user data block corresponding to the user identifier; If yes, taking the incomplete user data block as the target user data block; If not, a target user data block is allocated to the user corresponding to the user identifier.

3. The method according to claim 1, wherein The method further comprises: Determine the index shard corresponding to the user data block to be deleted according to the location information of the index shard recorded in the first index block; Marking the location information of the index shard corresponding to the to-be-deleted user data block recorded in the first index block as deleted; For a dynamic index block, according to the deletion mark of the index fragment in the dynamic index block, the location information in the first index block is deleted, and the disk space corresponding to the location information is released.

4. The method according to claim 1, wherein The file system further includes a second index block; the second index block is used to record location information of index slices that record index information of the user data.

5. The method according to any one of claims 1 to 4, characterized in that, The file system also includes a boot block and a backup boot block; the boot block and the backup boot block are both used to record the size of the user data block, the size of the dynamic index block, the size of the first index block and the size of the index shard.

6. The method according to claim 5, wherein The file system further includes a second index block; the second index block is used to record the location information of the index slice that records the index information of the user data, and the method further includes: Acquire formatting parameters; the formatting parameters include the size of the boot block, the size of the index block, and the upper limit number of data blocks in a single block group; the index block includes the first index block and the second index block; the data block is any one of the index block, the dynamic index block, and the user data block; Get the capacity of the memory; Calculating the number of data blocks, the size of a single block group, and the number of block groups according to the formatting parameters and the capacity of the memory; Determine the initial information of the index block according to the number of data blocks, the size of the block group, and the number of block groups; the initial information of the index block includes the unique identifier and location information of the index block; Write the initial information of the index block into the storage spaces corresponding to the startup block and the backup startup block in the memory respectively to complete the formatting of the file system.

7. A data reading method, characterized in that, The method includes: Obtain the target user identifier corresponding to the user data to be read; Determine the target user data block according to the target user identifier; the target user data block is one of at least one user data block included in the user data area; the user data area is a partition in the file system for storing user data; the file system further includes an index area and a dynamic data area; the index area includes at least one first index block, the dynamic data area includes at least one dynamic index block, the dynamic index block includes one or more index shards; the index shard is used to record the index information of the user data; the first index block is used to record the location information of the index shard; Determine the user data to be read in the target user data block according to the target user identifier; Determine the first index block corresponding to the target user data block according to the target user identifier; Query the first index block corresponding to the target user data block according to the target user identifier to determine the location information of the first index shard storing the index information of the user data to be read; the first index shard is one of one or more index shards included in the dynamic index block, and the location information of the first index shard is recorded in the first index block corresponding to the target user data block; Based on the location information of the first index shard, perform a read operation on the index information of the user data to be read and the user data to be read.

8. A data writing device, characterized in that The device includes: An obtaining module, configured to obtain user data and the index information of the user data; A processing module, configured to determine a target user data block according to the user identifier of the user data; the target user data block is one of at least one user data block included in the user data area; the user data area is a partition in the file system for storing the user data; the file system further includes an index area and a dynamic data area; the index area includes at least one first index block, the dynamic data area includes at least one dynamic index block, the dynamic index block includes one or more index shards; the index shard is used to record the index information of the user data; the first index block is used to record the location information of the index shard; write the user data into the target user data block, and write the index information of the user data into the first index shard in the dynamic index block; the first index shard is one of one or more index shards included in the dynamic index block, and the location information of the first index shard is recorded in the first index block corresponding to the target user data block.

9. The device according to claim 8, characterized in that The processing module is specifically configured to query whether there is an unfilled user data block corresponding to the user identifier of the user data; if so, use the unfilled user data block as the target user data block; if not, allocate a target user data block for the user corresponding to the user identifier. The processing module is further configured to determine the index shard corresponding to the user data block to be deleted according to the position information of the index shards recorded in the first index block. Mark the position information of the index shard corresponding to the user data block to be deleted recorded in the first index block as deleted; for a dynamic index block, delete the position information in the first index block according to the deletion marks of the index shards in the dynamic index block, and release the disk space corresponding to the position information. The file system further includes a second index block; the second index block is used to record the position information of the index shards that record the index information of the user data. The file system further includes a boot block and a backup boot block; both the boot block and the backup boot block are used to record the size of the user data block, the size of the dynamic index block, the size of the first index block, and the size of the index shard. The obtaining module is further configured to obtain formatting parameters; the formatting parameters include the size of the boot block, the size of the index block, and the upper limit number of data blocks in a single block group; the index block includes the first index block and the second index block; the data block is any one of the index block, the dynamic index block, and the user data block; obtain the capacity of the memory; the processing module is further configured to calculate the number of data blocks, the size of a single block group, and the number of block groups according to the formatting parameters and the capacity of the memory. Determine the initial information of the index block according to the number of data blocks, the size of the block group, and the number of block groups. The initial information of the index block includes the unique identifier and position information of the index block. Write the initial information of the index block into the storage spaces corresponding to the boot block and the backup boot block in the memory respectively to complete the formatting of the file system.

10. A data reading device, characterized in that, The device includes: An obtaining module, configured to obtain a target user identifier corresponding to the user data to be read. A processing module, configured to determine a target user data block according to the target user identifier; the target user data block is one of at least one user data block included in a user data area; the user data area is a partition in a file system for storing user data; the file system further includes an index area and a dynamic data area; the index area includes at least one first index block, the dynamic data area includes at least one dynamic index block, the dynamic index block includes one or more index shards; the index shard is used to record index information of the user data; the first index block is used to record location information of the index shard; according to the target user identifier, determine the user data to be read in the target user data block; according to the target user identifier, determine the first index block corresponding to the target user data block; according to the target user identifier, query the first index block corresponding to the target user data block, and determine the location information of the first index shard storing the index information of the user data to be read; the first index shard is one of one or more index shards included in the dynamic index block, and the location information of the first index shard is recorded in the first index block corresponding to the target user data block; based on the location information of the first index shard, perform a read operation on the index information of the user data to be read and the user data to be read.

11. An electronic device, characterized in that, The electronic device includes: a processor and a memory; The memory stores instructions executable by the processor; When the processor is configured to execute the instructions, the electronic device implements the method according to any one of claims 1-7.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes: computer software instructions; When the computer software instructions run in an electronic device, the electronic device implements the method according to any one of claims 1-7.

Images