Cross-file data access method and system based on data aggregation and embedded device

Abstract

The invention provides a cross-file data access method and system based on data aggregation and embedded equipment, and the method comprises the steps: collecting data access time sequence information of a preset time length, and sending the data access time sequence information to a server, so that the server carries out clustering analysis on the data access time sequence information, generating a data access mode file; receiving a data access mode file transmitted by the server, and creating a corresponding data aggregation storage block in the preset storage area according to the data access mode file; and in response to a data access request of an upper-layer application, judging whether the corresponding to-be-read data content has a matched data access mode, if so, reading the to-be-read data content from the corresponding data aggregation storage block, otherwise, reading the to-be-read data content from the corresponding source file. By designing a storage and index mechanism for aggregating small data blocks, the disk I / O operation frequency and the read amplification problem are effectively reduced, and the cross-file data access performance is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of embedded device data storage, and in particular to a cross-file data access method and system based on data aggregation and an embedded device. Background Art

[0002] As the performance of embedded devices improves and their functions become more complex, more and more sensors are integrated and used, and various sensor data are scattered and recorded in different files. This not only increases the difficulty of data management in embedded file systems, but also causes frequent reading and access of sensor data under complex data processing requirements. Figure 1 The small amount of data shown in the figure is read across files. This data access pattern will cause serious read amplification problems and significantly reduce the throughput of the file system.

[0003] However, existing research on embedded file systems mainly focuses on how to efficiently utilize existing resources and reduce memory overhead in an environment with limited resources such as processing power and storage space, and how to improve the access performance of small files in the file system. It does not involve optimization research on scenarios where files are read with smaller amounts of data, making it difficult to truly improve the data access performance of embedded devices. Summary of the invention

[0004] The purpose of the present invention is to provide a cross-file data access method based on data aggregation. For embedded device scenarios with limited memory, a storage and indexing mechanism for aggregated small data blocks is designed based on the data-level optimization level. Data can be read quickly while occupying less memory, effectively reducing the number of disk I / O operations and read amplification problems, speeding up data access efficiency, and thereby effectively improving cross-file data access performance.

[0005] In order to achieve the above objectives, it is necessary to provide a cross-file data access method, system, computer device and storage medium based on data aggregation in response to the above technical problems.

[0006] In a first aspect, an embodiment of the present invention provides a cross-file data access method based on data aggregation, the method comprising the following steps:

[0007] Collecting data access timing information of a preset duration, and sending the data access timing information to a server, so that the server performs cluster analysis on the data access timing information and generates a corresponding data access pattern file; the data access pattern file includes at least one data access pattern;

[0008] Receiving a data access mode file transmitted by the server, and creating a corresponding data aggregation storage block in a preset storage area according to the data access mode file;

[0009] In response to the data access request of the upper-layer application, determine whether there is a matching data access pattern for the corresponding data content to be read. If so, read the data content to be read from the corresponding data aggregation storage block. Otherwise, read the data content to be read from the corresponding source file.

[0010] Furthermore, the method further comprises:

[0011] In response to a data write request, it is determined whether the corresponding write update data includes the data in the data aggregate storage block. If so, the data in the data aggregate storage block and the corresponding source file are updated simultaneously. Otherwise, the data in the corresponding source file is updated.

[0012] Further, the data access timing information includes a plurality of file reading information; the file reading information includes a file path, a reading offset, a reading data amount and a reading file size;

[0013] The server performs cluster analysis on the data access timing information to generate a corresponding data access pattern file, comprising:

[0014] The file path of each file reading information in the data access timing information is converted into a numerical identifier, and the read offset, the read data amount and the read file size of each file reading information are normalized to obtain a corresponding access feature vector;

[0015] Performing time series clustering analysis on all access feature vectors corresponding to the data access time series information to obtain at least one access feature clustering cluster;

[0016] According to each access feature clustering cluster, a corresponding data access pattern is generated, and according to all data access patterns, the data access pattern file is generated; the data access pattern includes a data access sequence composed of a plurality of file data information.

[0017] Further, the file data information includes a file path to which the data belongs, an offset direction, an offset, and a data size; the data aggregation storage block includes an aggregation storage block;

[0018] The step of creating a corresponding data aggregation storage block in a preset storage area according to the data access mode file includes:

[0019] According to the device logical paging size, each file data information of each data access sequence in the data access mode file is converted to obtain corresponding logical paging data information, and the data access sequence is updated according to each logical paging data information; the logical paging data information includes the file path to which the data belongs and the corresponding logical page number range;

[0020] Extract all logical paging data corresponding to each data access sequence and aggregate them to generate corresponding aggregate storage blocks;

[0021] A data index relationship between each logical paging data corresponding to the data access sequence and a storage location in a corresponding aggregate storage block is generated by a preset hash algorithm.

[0022] Further, the data access mode file also includes a cache file data sequence;

[0023] The step of clustering the access features to generate the corresponding data access patterns, and generating the data access pattern file according to all the data access patterns comprises:

[0024] Cluster the clusters according to each access feature and generate the corresponding initial data access pattern;

[0025] Acquire duplicate file data information in each initial data access mode, and generate the cache file data sequence according to the duplicate file data information;

[0026] Removing each duplicate file data information from the corresponding initial data access pattern to obtain the corresponding data access pattern;

[0027] The data access pattern file is generated according to all data access patterns and the cache file data sequence.

[0028] Furthermore, the data aggregation storage block includes an aggregation storage block and a cache data block; the step of creating a corresponding data aggregation storage block in a preset storage area according to the data access mode file includes:

[0029] According to the device logical paging size, each file data information of each data access sequence in the data access mode file is converted to obtain corresponding logical paging data information, and the data access sequence is updated according to each logical paging data information; the logical paging data information includes the file path to which the data belongs and the corresponding logical page number range;

[0030] Extracting all logical paging data corresponding to each data access sequence and aggregating them to generate a corresponding aggregate storage block, and generating a data index relationship between each logical paging data corresponding to the data access sequence and a storage position in the corresponding aggregate storage block;

[0031] Convert each duplicate file data information in the cache file data sequence to obtain corresponding duplicate data paging information;

[0032] All duplicate data paging information corresponding to the cache file data sequence is extracted and aggregated to generate corresponding cache data blocks, and a data index relationship between each duplicate data paging information and a storage position in the corresponding cache data block is generated.

[0033] Furthermore, the data content to be read includes a file path to be read, an offset to be read, and a size of the data to be read;

[0034] The step of determining whether the corresponding data content to be read has a matching data access mode comprises:

[0035] Determine whether each data access mode includes the file path to be read, and if so, obtain the corresponding logical paging range to be read according to the offset to be read and the size of the data to be read;

[0036] Determine whether all logical pages to be read within the range of logical pages to be read are stored in the data aggregation storage block. If so, determine that there is a matching data access pattern for the data content to be read; otherwise, determine that there is no matching data access pattern for the data content to be read.

[0037] In a second aspect, an embodiment of the present invention provides a cross-file data access system based on data aggregation, the system comprising:

[0038] A pattern analysis module, used for collecting data access timing information of a preset duration, and sending the data access timing information to a server, so that the server performs cluster analysis on the data access timing information and generates a corresponding data access pattern file; the data access pattern file includes at least one data access pattern;

[0039] A data aggregation module, used to receive a data access mode file transmitted by the server, and create a corresponding data aggregation storage block in a preset storage area according to the data access mode file;

[0040] The accelerated reading module is used to respond to the data access request of the upper-level application and determine whether there is a matching data access pattern for the corresponding data content to be read. If so, the data content to be read is read from the corresponding data aggregation storage block. Otherwise, the data content to be read is read from the corresponding source file.

[0041] Furthermore, the system further comprises:

[0042] The data update module is used to respond to a data write request and determine whether the corresponding write update data includes the data in the data aggregation storage block. If so, the data in the data aggregation storage block and the corresponding source file are updated at the same time. Otherwise, the data in the corresponding source file is updated.

[0043] In a third aspect, an embodiment of the present invention further provides an embedded device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0044] The above-mentioned application provides a cross-file data access method, system and embedded device based on data aggregation. Through the method, the embedded device collects data access timing information of a preset duration, sends the data access timing information to the server, and the server performs cluster analysis on the data access timing information. After generating a corresponding data access pattern file, the data access pattern file transmitted by the server is received, and according to the data access pattern file, a corresponding data aggregation storage block is created in a preset storage area, and in response to a data access request of an upper-level application, it is determined whether there is a matching data access pattern for the corresponding data content to be read. If so, the data content to be read is read from the corresponding data aggregation storage block. Otherwise, the data content to be read is read from the corresponding source file. Technical solution. Compared with the existing technology, this cross-file data access method based on data aggregation is aimed at the scenario where the upper-level application of embedded devices performs cross-file small data volume granularity reading on the underlying file system. By obtaining the rules of frequently accessed small data volumes and performing data-level optimization, a storage and indexing mechanism for aggregated small data blocks is designed. It can quickly read data while occupying less memory, effectively reduce the number of disk I / O operations and the read amplification problem during cross-file reading, speed up data access efficiency, and thus effectively improve cross-file data access performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a schematic diagram of a read amplification scenario of a small data volume cross-file read request of an existing embedded file system;

[0046] Figure 2 It is a flowchart of a cross-file data access method based on data aggregation in an embodiment of the present invention;

[0047] Figure 3 is a schematic diagram of filling an aggregate storage block in an embodiment of the present invention;

[0048] Figure 4 is a schematic diagram of a read acceleration process in an embodiment of the present invention;

[0049] Figure 5 is another flow chart of a cross-file data access method based on data aggregation in an embodiment of the present invention;

[0050] Figure 6 It is a schematic diagram of the overall architecture of a cross-file data access method based on data aggregation in an embodiment of the present invention;

[0051] Figure 7 It is a structural diagram of a cross-file data access system based on data aggregation in an embodiment of the present invention;

[0052] Figure 8 It is another structural schematic diagram of the cross-file data access system based on data aggregation in an embodiment of the present invention. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical scheme and beneficial effects of the present application clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. Obviously, the embodiments described below are part of the embodiments of the present invention and are only used to illustrate the present invention, but are not used to limit the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] In the existing cross-file small-granularity reading scenario of embedded devices, a small amount of data (usually ranging from a few bytes to more than ten bytes) is read across files each time, such as Figure 1 As shown, a certain data access operation is to read 6 bytes from file A, move the file pointer to continue reading 8 bytes, and then read 4 bytes from file B. Since the bottom layer uses blocks as the minimum read and write unit, the read amplification problem will be very serious, and the amplification rate will reach 512*3 / (6+8+4), and it will be more serious when pre-reading is turned on. The cross-file data access method based on data aggregation provided by the present invention can be understood as considering that the existing technology cannot optimize the data-level regularity access of this underlying file system, and it is difficult to effectively improve the application status of data access efficiency. A cross-file small number reading method is proposed by finding out the frequently accessed small data and aggregating them in a storage data block. The corresponding storage data block can be directly indexed for fast reading during subsequent access. It can reduce the read amplification problem during cross-file reading and improve the reading efficiency of small data volumes. The following embodiments will describe the cross-file data access method based on data aggregation of the present invention in detail.

[0055] In one embodiment, Figure 2 As shown, a cross-file data access method based on data aggregation is provided, comprising the following steps:

[0056] S11. Collect data access timing information of a preset duration, and send the data access timing information to a server, so that the server performs cluster analysis on the data access timing information and generates a corresponding data access pattern file; the data access pattern file includes at least one data access pattern.

[0057] The preset duration in this embodiment can be set according to the actual application requirements. In principle, in order to more accurately capture the user's data access rules, the longer the collection time, the more advantageous it is. However, considering the situation that the disk space consumption problem on the embedded device cannot be collected for a long time, this embodiment preferably can divide the preset duration into multiple time periods, that is, collect data access timing information of the preset duration by collecting in multiple time periods, and upload the corresponding collection results to the server for summary use after each collection for a period of time, and then delete the collected data stored locally on the embedded device. In actual applications, data access situations in different time periods can be collected every day and finally summarized on the server. It can also be collected in the morning of the first day and uploaded to the server after deleting the locally stored data, and the afternoon data access information is collected again on the second day and uploaded to the server to delete the locally stored data. The server summarizes the data collected for several weeks for access pattern recognition. It should be noted that the collection of data access timing information in this embodiment can be achieved by deploying the ftrace tool on the embedded device to monitor the file read operation of the file system. The corresponding multi-period collection process can be understood as similar to a sampling survey. As long as the randomness of the sampling is guaranteed, the collection rules can be guaranteed to be available.

[0058] In order to facilitate the subsequent server to identify the data access rules of the upper-layer application on the embedded device based on the acquired data access timing information, this embodiment preferably sets the data access timing information to include multiple file read information, and each file read information includes the file path, read offset, read data amount and read file size. Among them, the information of reading file size is crucial for understanding and predicting frequent operations at the end of the file; for example, if an application frequently adds data to the end of the file, and also often reads these newly added parts, it is not enough to capture this dynamic change by only using a fixed offset relative to the beginning of the file. Therefore, real-time tracking of the current size of the file is crucial to capturing this change. It should be noted that after uploading the collected detailed file read information to the server to complete the corresponding access pattern analysis and identification, the data access timing information stored by the server can also be deleted to save the storage space of the server.

[0059] In order to extract statistically significant data access patterns from the disorganized file reading information, this embodiment preferably uses a time series clustering algorithm to perform cluster analysis on the data access time series information, so as to better analyze and capture various regular file operations, such as access to a specific offset and a specific size of a specific file. Specifically, the server performs cluster analysis on the data access time series information and generates a corresponding data access pattern file, including:

[0060] The file path of each file read information in the data access timing information is converted into a numerical identifier, and the read offset, read data amount and read file size of each file read information are normalized to obtain a corresponding access feature vector; wherein, the detailed acquisition process of the access feature vector corresponding to each file read information includes: first converting the file path (file name) into numerical data; then standardizing or normalizing the read offset, read data amount and read file size; then, converting each data feature obtained by preprocessing into a vector form suitable for processing by a machine learning algorithm, that is, obtaining a corresponding access feature vector. The above preprocessing enables the subsequent timing clustering algorithm to more accurately identify effective data access patterns from various file read operations.

[0061] Perform a time series clustering analysis on all access feature vectors corresponding to the data access time series information to obtain at least one access feature clustering cluster; wherein, the process of obtaining the access feature clustering cluster by time series clustering analysis can be understood as identifying various access patterns from continuous access feature vectors through a time series clustering algorithm, and treating data points that do not belong to any obvious pattern as noise; in this way, each data access pattern can be clustered into a segment, and these segments represent typical behavior patterns of the device when operating files, and these behavior patterns may include periodic updates for specific files, frequent data reading, etc. It should be noted that the specific method of using the time series clustering algorithm can be implemented with reference to the existing technology, and will not be described in detail here.

[0062] According to each access feature clustering cluster, a corresponding data access mode is generated, and according to all data access modes, the data access mode file is generated; the data access mode includes a data access sequence composed of multiple file data information; wherein the data structure corresponding to the file data information is {string path, bool is_start:true, intoffset, bool size:14}, path is the file path to which the file data (small data) belongs, offset represents the data offset, size represents the data size, and the variable is_start identifies whether the small data is small data with respect to the corresponding offset at the beginning of the file or the end of the file. Each data access mode can be understood as a plurality of file data sequences including an access order. For example, in a certain data access mode, the file data (small data) accessed in sequence are A, B, and C, and the corresponding data access mode (data access sequence) can be expressed as:

[0063] A->B->C, where

[0064] A is like {

[0065] path: / a

[0066] int offset:10 / / Offset is 10 bytes

[0067] bool start:true / / File start offset or file end offset

[0068] bool size:14 / / Size is 14 bytes

[0069] }

[0070] B is like {

[0071] path: / a / / A and B are from different offsets of the same file, and they are considered to be different small data

[0072] int offset:19 / / Offset is 19 bytes

[0073] bool start:true / / File start offset or file end offset

[0074] bool size:12 / / Size is 12 bytes

[0075] }

[0076] C is like {

[0077] path: / c

[0078] int offset:10 / / Offset is 10 bytes

[0079] bool start:true / / File start offset or file end offset

[0080] bool size:14 / / Size is 14 bytes

[0081] }

[0082] The various data access patterns obtained in the above manner can be used to guide data management on embedded devices. By exporting these small data read access patterns as data access pattern files and transmitting them to embedded devices, a basis is provided for the creation of subsequent data aggregation storage blocks.

[0083] S12. Receive the data access mode file transmitted by the server, and create a corresponding data aggregation storage block in a preset storage area according to the data access mode file; the file data information includes the file path to which the data belongs, the offset direction, the offset and the data size; the data aggregation storage block includes an aggregation storage block; wherein, the aggregation storage block can be understood as a memory block used to aggregate and store the data involved in each data access mode, and the size of each aggregation storage block is the same, and they are all aligned to the minimum read and write unit of the underlying storage device. In principle, one aggregation storage block corresponds to one data access mode, but considering that there may be waste of space at the end of the aggregation storage block in actual applications, this embodiment can preferably first splice the data corresponding to each data access mode, and then slice it according to an integer multiple size of the underlying physical block (the minimum unit of each read and write at the bottom layer is a block, which is usually 512 bytes for SD cards with SDIO (Secure Digital Input and Output) interface, and alignment can be used to avoid reading unnecessary content), and then store each slice to effectively save space and improve efficiency. It should be noted that when slicing and aggregating the data corresponding to the data access pattern, try not to store a data access pattern across slices. If it crosses slices, subsequent reads that hit the data access pattern will cause the corresponding slices to be frequently swapped in and out, reducing access performance. In addition, even in the extreme scenario where the data access mode is stored across slices, the aggregate data storage mode can also bring very considerable performance improvement: for example, a small data is usually 32Byte in tens of bytes, and a slice is 512Byte, which can store 512 / 32=16 small data, and assuming that a data access mode is 18 small data lengths, where the first small data is stored in slice 1, the middle 16 are stored in slice 2, and the last small data is stored in slice 3, then the amount of data read after hitting the data access mode is 3 slices (hitting slice 1 storing the first small data, hitting slice 2 storing the middle 16 small data, hitting slice 3 of the 18th small data, which involves three slice swaps), but compared with the original 18 times read amplification, the amount of data read at this time will be reduced to 3 / 18=1 / 6. That is, even if a data access mode is stored across slices, the read amplification problem can be effectively reduced. It should be noted that the slice storing the data corresponding to the data access mode in this example can be understood as an aggregate storage block in the data aggregation storage block.

[0084] In this embodiment, the preset storage area for data aggregation storage block storage varies depending on the type of embedded file system actually applied. For example, for the FAT (File Allocation Table) file system, the reserved area provided by it can be directly used to store data aggregation storage blocks and corresponding aggregation block index tables. In this file system, the data aggregation storage blocks stored in the reserved area can directly obtain the required access data in the data aggregation storage blocks across the file system layer, thereby improving the reading and writing efficiency. For file systems that do not provide reserved areas, such as LittleFS, the data aggregation storage blocks can be stored as files in the root directory of the file system.

[0085] Considering that there may be multiple data access modes in actual applications with completely repeated small data or partially repeated small data segments, which causes a waste of storage space in the data aggregation storage block, in order to ensure the reasonable use of the data aggregation storage block as much as possible, this embodiment preferably adopts logical paging to manage data; specifically, the step of creating a corresponding data aggregation storage block in a preset storage area according to the data access mode file includes:

[0086] According to the device logical paging size, each file data information of each data access sequence in the data access mode file is converted to obtain corresponding logical paging data information, and the data access sequence is updated according to each logical paging data information; the logical paging data information includes the file path to which the data belongs and the corresponding logical page number range, wherein the acquisition process of the logical page number range includes: if the device logical paging size is 16 bytes, the A file data information in a certain data access mode is a small data with a length of 14 bytes at offset 10 of file 1, then it can be determined that in order to cover this 14-byte data, the first logical page (0->16 bytes) and the second logical page (16-32 bytes) need to be aggregated, that is, the logical page number range of the A file data information is obtained; the A file data information in the corresponding data access sequence needs to be replaced with a1->a2, where the lowercase letters represent the files, and the numbers represent the logical page numbers of the files:

[0087] a1:{

[0088] path: / a / / unchanged

[0089] index:1 / / logical page number

[0090] }

[0091] a2:{

[0092] path: / a / / unchanged

[0093] index:2 / / Logical page number

[0094] }

[0095] If the data access sequence in the above example is A->B->C, B is the second and third logical pages in / a.txt, and C is the first and second logical pages in / c, then replace B in the data access sequence with a2->a3, and similarly, replace C with c1->c2; the corresponding data access sequence will be updated to: a1->a2->a2->a3->c1->c2.

[0096] Extract all logical paging data corresponding to each data access sequence and aggregate them to generate corresponding aggregate storage blocks; the specific method of filling the aggregate storage blocks is as follows: Figure 3 As shown, one aggregate storage block can actually store data corresponding to different data access modes to improve the memory utilization of the aggregate storage block.

[0097] By using a preset hash algorithm, a data index relationship between each logical paging data corresponding to the data access sequence and the storage location in the corresponding aggregate storage block is generated; wherein the data index relationship can be stored in a preset aggregate block index table, which can be based on the unordered_map <string,unorderedmap<int,pair<int,int> >>The index_table format definition is obtained, where unordered_map is an associative container implemented by a hash table in C++, and pair is a data pair in C++, which is represented by: original file path->logical page number->(aggregate storage block, offset within the block). That is, the data in the source file is indexed to the storage location of the data in the aggregate block through a hash table; for example: the index key is an 11-byte data at an offset of 16 bytes from the file / a / b / c.txt, and the index value is the position of the offset of 0 bytes from aggregate block No. 1. The hash index in the data index relationship can quickly locate the storage location of the source data in the aggregate storage block, thereby improving the response speed of data access. It should be noted that the hash algorithm can be selected according to the resource limitations of the embedded device and the data scale in the data access mode, and a simple and efficient index algorithm should be used as much as possible.

[0098] In practical applications, each aggregate storage block in the data aggregate storage block can be filled according to the following algorithm:

[0099] Logical page size: PAGE_SIZE

[0100]

[0101]

[0102] In this embodiment, considering that the logical paging representation of the collected file access information will introduce additional invalid data, affect the cluster analysis results, and cannot guarantee the reliability of data access pattern recognition, the collected file access data is not initially represented by logical paging, but after the data access pattern is analyzed, it is converted in the form of logical paging. The above example shows that after the embedded device obtains the data access pattern analyzed by the server, it is processed into the form of logical paging. In actual applications, if the server can obtain the logical paging size of the corresponding embedded device, the server can also complete the conversion operation and then send the converted data access pattern to the embedded device for use, so as to save the computing overhead of the embedded device.

[0103] It should be noted that in order to avoid excessive memory overhead, when enabling data aggregation storage blocks, all aggregation storage blocks cannot be read into the memory at the same time. Instead, when the upper-layer application accesses data in the file system in the record data access mode, the aggregation storage blocks corresponding to the data access mode are read into the memory. Subsequent read requests will hit the aggregation storage blocks in the memory, thereby reducing disk I / O. At the same time, in order to avoid the aggregation storage blocks occupying too much memory, only one aggregation block is allowed to be loaded into the memory. When the data access mode is switched, the aggregation storage blocks are swapped in and out to ensure that the memory space occupied by the aggregation storage blocks is within a controllable range. However, for more complex data access scenarios, when multiple upper-layer applications access data in the file system simultaneously in multiple data access modes, this means that these data access modes are mixed together, and this mixing will cause frequent switching of aggregate storage blocks, resulting in a read amplification problem; for example, the data access modes stored in aggregate storage block No. 1 and aggregate storage block No. 2 are [A, B, C] and [D, E, F]. When these two data access modes are performed simultaneously, the following access sequence may appear: [A, D, B, E, C, F]. If only one aggregate storage block is still allowed to be loaded into the memory, aggregate storage block No. 1 and aggregate storage block No. 2 will be frequently swapped in and out, resulting in the same read amplification problem as before; therefore, in order to achieve a balance between performance and memory overhead, for this scenario, multiple aggregate storage blocks can be allowed to be cached in the memory at the same time, that is, it can cope with the situation where multiple data access modes are performed simultaneously. The specific number of aggregate storage blocks cached in the memory can be determined by the operating system and is not limited here.

[0104] In addition, in order to avoid the situation where duplicate data in multiple data access modes occupy multiple storage locations and to improve the utilization and storage efficiency of data aggregation storage blocks, this embodiment preferably introduces logical paging management in data aggregation storage blocks, and further extracts and stores completely duplicate data in multiple data access modes separately. That is, in data access pattern identification, it is necessary to further analyze the duplicate data involved in each data access pattern, and the corresponding data access pattern file also includes a cache file data sequence. Specifically, the steps of clustering clusters according to each access feature, generating a corresponding data access pattern, and generating the data access pattern file according to all data access patterns include:

[0105] According to the clustering of each access feature, the corresponding initial data access pattern is generated; wherein, the initial data access pattern is equivalent to the data access pattern obtained without considering the scenario of completely repeated data between different data access patterns. The specific acquisition process is described above and will not be repeated here.

[0106] Duplicate file data information in each initial data access mode is obtained, and the cache file data sequence is generated according to the duplicate file data information; wherein the cache file data sequence can be understood as a sequence composed of completely identical file data information between multiple data access modes that exist simultaneously; for example, there are initial data access mode 1: A->B->C->D->G, and initial data access mode 2: F->B->E->C simultaneously, then the corresponding duplicate file data information is B and C, and according to the access order of the duplicate file data information, the cache file data sequence can be obtained as B->C, which is used for the subsequent creation of cache data blocks.

[0107] Each duplicate file data information is removed from the corresponding initial data access mode to obtain the corresponding data access mode; wherein the data access mode can be understood as the data access sequence obtained by removing the cache file data sequence in the initial data access mode; for example, in the above example, B and C in the initial data access mode 1 of A->B->C->D->G and the initial data access mode 2 of F->B->E->C are removed, but it should be noted that the access order of the original data is not changed, that is, the above two data access modes will be updated to A->D->G and F->E.

[0108] The data access pattern file is generated according to all data access patterns and the cache file data sequence. That is, the data access pattern file that simultaneously introduces logical paging and duplicate file data separate cache mechanism is composed of a cache file data sequence that stores duplicate file data information between all data access patterns and each data access pattern that removes duplicate file data information. In practical applications, the configuration of the data access pattern is completed through the cooperation of the two, so as to simultaneously avoid the waste of memory resources in the scenario of duplicate small data segments or completely duplicate small data due to the data access pattern, and further improve the rationality of the use of data aggregation storage blocks.

[0109] In the case of introducing both logical paging and duplicate file data separate caching mechanisms, after the server receives and sends a data access pattern file that includes both a data access pattern and a cache file data sequence, it can create a corresponding data aggregation storage block according to the data access pattern and the cache file data sequence. In this embodiment, preferably, the data aggregation storage block includes an aggregation storage block and a cache data block; the step of creating a corresponding data aggregation storage block in a preset storage area according to the data access pattern file includes:

[0110] According to the device logical paging size, each file data information of each data access sequence in the data access mode file is converted to obtain corresponding logical paging data information, and the data access sequence is updated according to each logical paging data information; the logical paging data information includes the file path to which the data belongs and the corresponding logical page number range; wherein, the acquisition of the logical page number range and the updating of the data access sequence can all refer to the above-mentioned related description which only introduces the logical paging mechanism, and will not be repeated here.

[0111] All logical paging data corresponding to each data access sequence are extracted and aggregated to generate corresponding aggregate storage blocks, and data index relationships between each logical paging data corresponding to the data access sequence and storage positions in the corresponding aggregate storage blocks are generated; wherein, the filling of the aggregate storage blocks and the generation and storage methods of the data index relationships can also be implemented by referring to the above-mentioned related descriptions of only introducing the logical paging mechanism, and will not be repeated here.

[0112] The duplicate file data information in the cache file data sequence is converted to obtain corresponding duplicate data paging information; wherein the duplicate data paging information also includes the file path and logical page number range to which the corresponding data belongs. The specific acquisition process can also refer to the corresponding description above, which will not be repeated here.

[0113] All duplicate data paging information corresponding to the cache file data sequence is extracted and aggregated to generate corresponding cache data blocks, and a data index relationship between each duplicate data paging information and a storage position in the corresponding cache data block is generated; the specific cache data block filling method and the generation and storage method of the data index relationship can all be implemented by referring to the relevant description of the above-mentioned aggregate storage block, which will not be described in detail here.

[0114] The storage index mechanism provided by this embodiment, which simultaneously introduces the aggregate storage block and the cache data block, can effectively manage the completely repeated small data and the repeated data segments between small data in multiple data access modes, thereby improving the performance and reliability of the overall system. At the same time, this mechanism ensures that the data aggregate storage block will not have the same data stored repeatedly, which not only avoids the waste of memory resources, but also reduces the complexity of data management, and can effectively improve the efficiency of data access.

[0115] S13, in response to the data access request of the upper-layer application, determine whether there is a matching data access pattern for the corresponding data content to be read, if so, read the data content to be read from the corresponding data aggregation storage block, otherwise, read the data content to be read from the corresponding source file; wherein, the data content to be read includes the file path to be read, the offset to be read and the size of the data to be read; in order to ensure the high efficiency of data reading and avoid the read amplification problem of a small number of cross-file accesses, this embodiment preferably first determines whether the data content to be read can hit the data in the data aggregation storage block, and if it hits, directly performs efficient index reading according to the corresponding aggregation block index table; specifically, the step of determining whether there is a matching data access pattern for the corresponding data content to be read includes:

[0116] Determine whether each data access mode includes the file path to be read. If so, obtain the corresponding logical paging range to be read according to the offset to be read and the size of the data to be read; wherein the matching check of the file path to be read can be based on the unordered_map designed by the aforementioned aggregate block index table. <string,unordered_map<int,pair<int,int> >>index_table, through index_table.contains(path) method to determine whether index_table contains path path, it will not be described in detail here.

[0117] Determine whether all logical pages to be read within the range of logical pages to be read are stored in the data aggregation storage block. If so, determine whether there is a matching data access pattern for the data content to be read. Otherwise, determine whether there is no matching data access pattern for the data content to be read. The matching check of the range of logical pages to be read can be understood as determining whether all logical pages within the range of the starting page offset / PAGE_SIZE and the ending page (offset+size+PAGE_SIZE) / PAGE_SIZE in the range of logical pages to be read have been stored in the data aggregation storage block. If they have been stored in the aggregation storage block or the data cache block, determine whether there is a matching data access pattern. The following read acceleration algorithm can be used to read directly from the matching aggregation storage block or the data cache block. Otherwise, the existing file system data reading method reads the required data from the source file. The corresponding read acceleration process is as follows: Figure 4 It should be noted that, since the data stored in each aggregate storage block and data cache block are different, there will be no repeated hits first, and it is only necessary to load the unique storage block after the hit.

[0118]

[0119]

[0120] The embodiment of the present application provides a technical solution in which an embedded device collects data access timing information of a preset duration, sends the data access timing information to a server, and the server performs cluster analysis on the data access timing information to generate a corresponding data access pattern file, then receives the data access pattern file transmitted by the server, creates a corresponding data aggregation storage block in a preset storage area according to the data access pattern file, and judges whether a matching data access pattern exists for the corresponding data content to be read in response to a data access request of an upper-level application. If so, the data content to be read is read from the corresponding data aggregation storage block, and vice versa, the data content to be read is read from the corresponding source file. The technical solution is aimed at the scenario in which the upper-level application of an embedded device performs a cross-file small data volume granularity reading of the underlying file system, and obtains the data-level optimization level design by obtaining the rules of frequent access to small data volumes, so as to obtain a storage and indexing mechanism for aggregated small data blocks, which can quickly read data while occupying less memory, effectively reduce the number of disk I / O operations and the read amplification problem during cross-file reading, speed up data access efficiency, and thereby effectively improve cross-file data access performance.

[0121] In addition, in order to ensure the consistency of data updates (data write consistency) based on the storage and indexing mechanism of the above-mentioned aggregated small data blocks, and thus ensure that the upper-layer application can obtain real-time and accurate data information, this embodiment preferably checks whether the data to be updated is the data in the data aggregation storage block each time a data update operation is performed, so as to ensure that the data in the data aggregation storage block is consistent with the data in the corresponding source file; specifically, Figure 5 As shown, in one embodiment, the method further includes:

[0122] S14, in response to the data write request, determine whether the corresponding write update data includes the data in the data aggregation storage block, if so, update the data in the data aggregation storage block and the corresponding source file at the same time, otherwise, update the data in the corresponding source file. In principle, the data update method of the data aggregation storage block and the source file only needs to ensure the consistency of the updates of the data aggregation storage block and the source file when the data need to be updated at the same time, which can be implemented with reference to the relevant existing technology and is not specifically limited here.

[0123] The data update method provided by this embodiment is based on the storage index mechanism that simultaneously introduces aggregate storage blocks and cache data blocks. Since the file is logically divided into pages of fixed size and these pages are mapped to specific data blocks, each small data can be quickly located by page number, reducing the complexity of duplicate data judgment and extraction to O(1), which not only speeds up data retrieval, but also simplifies the data update process, that is, it can greatly reduce the complexity of duplicate data judgment and processing algorithms during data update and simplify data access. For example: 16-byte paging, the data of 20 bytes at the position offset 14 bytes of file A is a small data that needs to be recorded, then the logical pages of file A that only need to be stored in the aggregate storage block are pages 0 to 2 (starting page: End page: ), that is, the data of the 0th logical page, 1st logical page, and 2nd logical page of the file, totaling 48 bytes, needs to be stored in the aggregate storage block. The corresponding aggregate block index table records that the A file stores pages 0, 1, and 2. <string,unordered_map<int,pair<int,int> >>index_table can quickly locate whether the data to be accessed is recorded in the data aggregation storage block by judging whether the path and logical page are hit in turn. The time complexity is O(1)*O(1). However, if the data segmentation method is used to manage similar small data, the data structure can be changed to unordered_map <string,vector<pair<int,int> >>>(file source path->data interval in the file, where vector is an array in C++), it takes O(1)*O(n) time to determine whether the data is in the interval (n is the number of small data recorded in the file). Therefore, this embodiment uses logical paging to optimize data access time by exchanging space for time, while storing a small amount of redundant data before and after the valid data to improve the processing performance of data duplication judgment when data is written and updated. In addition, by combining the use of cache data blocks and aggregate storage blocks, it can be ensured that there are no duplicate small data blocks in the entire data aggregate storage block. It is only necessary to maintain the consistency of the data aggregate storage block with the small data in the source file. This refined storage index mechanism can be effectively applied to cross-file small data access scenarios. While effectively solving the read amplification problem, it can also effectively avoid the write amplification problem.

[0124] The present application embodiment Figure 6As shown, the embedded device collects data access timing information of a preset time length, sends the data access timing information to the server, and the server performs cluster analysis on the data access timing information to generate a corresponding data access pattern file, receives the data access pattern file transmitted by the server, creates a corresponding data aggregation storage block in a preset storage area according to the data access pattern file, and responds to the data access request of the upper-layer application to determine whether there is a matching data access pattern for the corresponding data content to be read. If so, read the data content to be read from the corresponding data aggregation storage block. Otherwise, read the data content to be read from the corresponding source file, and respond to the data write request to determine whether the corresponding write update data contains the data in the data aggregation storage block. If so, then simultaneously A technical solution for updating the data aggregation storage block and the data in the corresponding source file, and vice versa, updating the data in the corresponding source file. Aiming at the scenario where the upper-level application of the embedded device performs cross-file small data volume granularity reading on the underlying file system, a storage and indexing mechanism for aggregated small data blocks is designed at the data-level optimization level by obtaining the rules of frequently accessed small data volumes. While avoiding the risk of introducing system uncertainty and instability by introducing a large number of changes to the file system interface through the introduction of an embedded database, it can quickly read data based on less memory usage requirements, effectively reduce the number of disk I / O operations and major read and write problems during cross-file reading, speed up data access efficiency and data update speed, and thus effectively improve cross-file data access performance, providing reliable technical support for improving the application performance of embedded devices.

[0125] It should be noted that although the steps in the above flowchart are shown in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders.

[0126] In one embodiment, Figure 7 As shown, a cross-file data access system based on data aggregation is provided, and the system includes:

[0127] Pattern analysis module 1, used for collecting data access timing information of a preset duration, and sending the data access timing information to a server, so that the server performs cluster analysis on the data access timing information and generates a corresponding data access pattern file;

[0128] Data aggregation module 2, used for receiving the data access mode file transmitted by the server, and creating a corresponding data aggregation storage block in a preset storage area according to the data access mode file;

[0129] The accelerated reading module 3 is used to respond to the data access request of the upper-level application and determine whether there is a matching data access pattern for the corresponding data content to be read. If so, the data content to be read is read from the corresponding data aggregation storage block. Otherwise, the data content to be read is read from the corresponding source file.

[0130] In one embodiment, Figure 8 As shown, a cross-file data access system based on data aggregation is provided, and the system also includes:

[0131] The data update module 4 is used to respond to a data write request and determine whether the corresponding write update data includes the data in the data aggregation storage block. If so, the data in the data aggregation storage block and the corresponding source file are updated at the same time. Otherwise, the data in the corresponding source file is updated.

[0132] For the specific definition of the cross-file data access system based on data aggregation, please refer to the definition of the cross-file data access method based on data aggregation above, and the corresponding technical effects can also be obtained equivalently, which will not be repeated here. Each module in the above-mentioned cross-file data access system based on data aggregation can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0133] In one embodiment, an embedded device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the steps of the above method are implemented when the processor executes the computer program.

[0134] In summary, an embodiment of the present invention provides a cross-file data access method, system and embedded device based on data aggregation. The cross-file data access method based on data aggregation realizes that the embedded device collects data access timing information of a preset time length, sends the data access timing information to the server, and the server performs cluster analysis on the data access timing information to generate a corresponding data access pattern file, receives the data access pattern file transmitted by the server, creates a corresponding data aggregation storage block in a preset storage area according to the data access pattern file, and responds to the data access request of the upper-level application to determine whether there is a matching data access pattern for the corresponding data content to be read. If so, the data content to be read is read from the corresponding data aggregation storage block, otherwise, the data content to be read is read from the corresponding source file, and responds to the data write request to determine the corresponding write The invention discloses a technical scheme for determining whether the updated data includes the data in the data aggregation storage block. If so, the data aggregation storage block and the data in the corresponding source file are updated at the same time. Otherwise, the data in the corresponding source file is updated. The method targets the scenario in which the upper-layer application of an embedded device performs a small-volume cross-file granularity reading of the underlying file system. By obtaining the regularity of frequently accessing small data volumes, the data-level optimization layer is designed to obtain a storage and indexing mechanism for aggregated small data blocks. The method avoids the risk of introducing system uncertainty and instability and massively changing the file system interface by introducing an embedded database. The method can read data quickly based on less memory usage requirements, effectively reduce the number of disk I / O operations and the large reading and writing problems during cross-file reading, speed up data access efficiency and data update speed, and thus effectively improve the cross-file data access performance, providing reliable technical support for improving the application performance of embedded devices.

[0135] Each embodiment in this specification is described in a progressive manner, and the same or similar parts of each embodiment can be directly referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. It should be noted that the technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above-mentioned embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0136] The above-mentioned embodiments only express several preferred implementation modes of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in the technical field, several improvements and substitutions can be made without departing from the technical principles of the present invention, and these improvements and substitutions should also be regarded as the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be based on the protection scope of the claims.

Claims

1. A cross-file data access method based on data aggregation, characterized in that: The method comprises the following steps: Collecting data access timing information of a preset duration, and sending the data access timing information to a server, so that the server performs cluster analysis on the data access timing information and generates a corresponding data access pattern file; the data access pattern file includes at least one data access pattern; Receiving a data access mode file transmitted by the server, and creating a corresponding data aggregation storage block in a preset storage area according to the data access mode file; In response to the data access request of the upper-layer application, determine whether there is a matching data access pattern for the corresponding data content to be read. If so, read the data content to be read from the corresponding data aggregation storage block. Otherwise, read the data content to be read from the corresponding source file.

2. The cross-file data access method based on data aggregation according to claim 1, characterized in that: The method further comprises: In response to a data write request, it is determined whether the corresponding write update data includes the data in the data aggregate storage block. If so, the data in the data aggregate storage block and the corresponding source file are updated simultaneously. Otherwise, the data in the corresponding source file is updated.

3. The cross-file data access method based on data aggregation according to any one of claims 1 or 2, characterized in that: The data access timing information includes a plurality of file reading information; the file reading information includes a file path, a reading offset, a reading data amount and a reading file size; The server performs cluster analysis on the data access timing information to generate a corresponding data access pattern file, comprising: The file path of each file reading information in the data access timing information is converted into a numerical identifier, and the read offset, the read data amount and the read file size of each file reading information are normalized to obtain a corresponding access feature vector; Performing time series clustering analysis on all access feature vectors corresponding to the data access time series information to obtain at least one access feature clustering cluster; According to each access feature clustering cluster, a corresponding data access pattern is generated, and according to all data access patterns, the data access pattern file is generated; the data access pattern includes a data access sequence composed of a plurality of file data information.

4. The cross-file data access method based on data aggregation according to claim 3, characterized in that: The file data information includes the file path to which the data belongs, the offset direction, the offset and the data size; the data aggregation storage block includes an aggregation storage block; The step of creating a corresponding data aggregation storage block in a preset storage area according to the data access mode file includes: According to the device logical paging size, each file data information of each data access sequence in the data access mode file is converted to obtain corresponding logical paging data information, and the data access sequence is updated according to each logical paging data information; the logical paging data information includes the file path to which the data belongs and the corresponding logical page number range; Extract all logical paging data corresponding to each data access sequence and aggregate them to generate corresponding aggregate storage blocks; A data index relationship between each logical paging data corresponding to the data access sequence and a storage location in a corresponding aggregate storage block is generated by a preset hash algorithm.

5. The cross-file data access method based on data aggregation according to claim 3, characterized in that: The data access mode file also includes a cache file data sequence; The step of clustering the access features to generate the corresponding data access patterns, and generating the data access pattern file according to all the data access patterns comprises: Cluster the clusters according to each access feature and generate the corresponding initial data access pattern; Acquire duplicate file data information in each initial data access mode, and generate the cache file data sequence according to the duplicate file data information; Removing each duplicate file data information from the corresponding initial data access pattern to obtain the corresponding data access pattern; The data access pattern file is generated according to all data access patterns and the cache file data sequence.

6. The cross-file data access method based on data aggregation according to claim 5, characterized in that: The data aggregation storage block includes an aggregation storage block and a cache data block; The step of creating a corresponding data aggregation storage block in a preset storage area according to the data access mode file includes: According to the device logical paging size, each file data information of each data access sequence in the data access mode file is converted to obtain corresponding logical paging data information, and the data access sequence is updated according to each logical paging data information; the logical paging data information includes the file path to which the data belongs and the corresponding logical page number range; Extracting all logical paging data corresponding to each data access sequence and aggregating them to generate a corresponding aggregate storage block, and generating a data index relationship between each logical paging data corresponding to the data access sequence and a storage position in the corresponding aggregate storage block; Convert each duplicate file data information in the cache file data sequence to obtain corresponding duplicate data paging information; All duplicate data paging information corresponding to the cache file data sequence is extracted and aggregated to generate corresponding cache data blocks, and a data index relationship between each duplicate data paging information and a storage position in the corresponding cache data block is generated.

7. The cross-file data access method based on data aggregation according to any one of claims 4 or 6, characterized in that: The data to be read includes the file path to be read, the offset to be read and the size of the data to be read; The step of determining whether the corresponding data content to be read has a matching data access mode comprises: Determine whether each data access mode includes the file path to be read, and if so, obtain the corresponding logical paging range to be read according to the offset to be read and the size of the data to be read; Determine whether all logical pages to be read within the range of logical pages to be read are stored in the data aggregation storage block. If so, determine that there is a matching data access pattern for the data content to be read; otherwise, determine that there is no matching data access pattern for the data content to be read.

8. A cross-file data access system based on data aggregation, characterized in that: The system comprises: a pattern analysis module, configured to collect data access timing information of a preset duration, and send the data access timing information to a server, so that the server performs cluster analysis on the data access timing information and generates a corresponding data access pattern file; the data access pattern file includes at least one data access pattern; A data aggregation module, used to receive a data access mode file transmitted by the server, and create a corresponding data aggregation storage block in a preset storage area according to the data access mode file; The accelerated reading module is used to respond to the data access request of the upper-level application and determine whether there is a matching data access pattern for the corresponding data content to be read. If so, the data content to be read is read from the corresponding data aggregation storage block. Otherwise, the data content to be read is read from the corresponding source file.

9. The cross-file data access system based on data aggregation as shown in claim 8, characterized in that: The system further comprises: The data update module is used to respond to a data write request and determine whether the corresponding write update data includes the data in the data aggregation storage block. If so, the data in the data aggregation storage block and the corresponding source file are updated at the same time. Otherwise, the data in the corresponding source file is updated.

10. An embedded device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

Images