Data storage method and device, electronic equipment and storage medium

Abstract

The invention discloses a data storage method and device, electronic equipment and a storage medium, and the method comprises the steps: compressing obtained tracking data based on a compression service to obtain a first compressed data block, and storing the first compressed data block into a data file based on a fragment storage service; traversing a first compressed data block stored in the data file, and determining memory index information based on a check bit in identification information of tracking data in the first compressed data block, the offset of the first compressed data block in the data file and the identification information of the data file; and storing the memory index information into the memory index. According to the scheme, the effects of saving data storage space and quickly responding to complex query requirements can be achieved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data storage method, device, electronic equipment and storage medium. Background Art

[0002] With the development of big data and cloud computing technologies, the amount of tracking data generated and processed by the system has exploded. In the prior art, when storing tracking data, a skip list data structure is used to store the generated tracking data, and the query is essentially based on the binary search method. This storage method results in low data storage utilization, high data duplication, low query efficiency, and inability to quickly respond to complex query requirements. At the same time, as the amount of data increases, this storage method is difficult to expand effectively and has poor scalability, resulting in a decrease in system performance. Summary of the invention

[0003] The present invention provides a data storage method, device, electronic device and storage medium, which can save data storage space and quickly respond to complex query requirements.

[0004] In a first aspect, the present invention provides a data storage method, comprising:

[0005] Compressing the acquired tracking data based on the compression service to obtain a first compressed data block, and storing the first compressed data block into a data file based on the shard storage service;

[0006] Traversing the first compressed data block stored in the data file, and determining the memory index information based on the check bit in the identification information of the tracking data in the first compressed data block, the offset of the first compressed data block in the data file, and the identification information of the data file;

[0007] The memory index information is stored in a memory index, and a preset form of the memory index is a multi-level hash index.

[0008] In a second aspect, the present invention further provides a data storage device, comprising:

[0009] A data processing module, configured to compress the acquired tracking data based on a compression service to obtain a first compressed data block, and store the first compressed data block into a data file based on a shard storage service;

[0010] a memory index information determination module, configured to traverse the first compressed data block stored in the data file, and determine the memory index information based on a check bit in the identification information of the tracking data in the first compressed data block, an offset of the first compressed data block in the data file, and the identification information of the data file;

[0011] A memory index building module is used to store the memory index information into a memory index, and the preset form of the memory index is a multi-stage hash index.

[0012] In a third aspect, an embodiment of the present invention further provides an electronic device, including:

[0013] one or more processors;

[0014] a storage device for storing one or more programs,

[0015] When the one or more programs are executed by the one or more processors, the one or more processors implement the data storage method provided in any embodiment of the present invention.

[0016] In a fourth aspect, an embodiment of the present invention further provides a storage medium comprising computer executable instructions, wherein the computer executable instructions, when executed by a computer processor, are used to execute a data storage method as provided in any embodiment of the present invention.

[0017] The technical solution provided by the present invention compresses the acquired tracking data based on the compression service to obtain the first compressed data block, and stores the first compressed data block in the data file based on the shard storage service; traverses the first compressed data block stored in the data file, and determines the memory index information based on the check bit in the identification information of the tracking data in the first compressed data block, the offset of the first compressed data block in the data file, and the identification information of the data file; and stores the memory index information in the memory index. By adopting the solution of the present invention, storage space can be saved by compressing the tracking data, and a fast response to data queries can be achieved by constructing the memory index based on the memory index information; at the same time, the data storage method of the present invention can significantly improve the scalability of data storage and indexing, and can effectively cope with the increase of massive data.

[0018] The above invention content is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and other features, advantages and aspects of the embodiments of the present invention will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0020] Figure 1 This is a flow chart of a data storage method provided by an embodiment of the present invention;

[0021] Figure 2 This is a flow chart of another data storage method provided by an embodiment of the present invention;

[0022] Figure 3 It is a schematic diagram of storage logic between a memory index and a log file provided by an embodiment of the present invention;

[0023] Figure 4 It is a schematic diagram of a sharding area provided by an embodiment of the present invention;

[0024] Figure 5 It is a structural schematic diagram of a data storage device provided by an embodiment of the present invention;

[0025] Figure 6 It is a structural schematic diagram of an electronic device for implementing a data storage method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0026] Embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are only for exemplary purposes and are not intended to limit the scope of protection of the present invention.

[0027] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0028] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0029] It should be noted that the concepts such as "first" and "second" mentioned in the present invention are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0030] It should be noted that the modifications of "one" and "plurality" mentioned in the present invention are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0031] The names of the messages or information exchanged between multiple devices in the embodiments of the present invention are only used for illustrative purposes, and are not used to limit the scope of these messages or information.

[0032] Figure 1 A flow chart of a data storage method provided in an embodiment of the present invention is applicable to the case of storing tracking data. The method can be executed by a data storage device, which can be implemented in the form of software and / or hardware and is generally integrated in any electronic device with network communication function, which can be a mobile terminal, a PC or a server, etc.

[0033] like Figure 1 As shown, the data storage method of the embodiment of the present invention may include the following process:

[0034] S110 . Compress the acquired tracking data based on the compression service to obtain a first compressed data block, and store the first compressed data block into a data file based on the shard storage service.

[0035] Among them, tracking data refers to data that can be used to record information generated by software or systems during operation. The generation of tracking data is time-series, and tracking data will continue to be generated over time. For the acquired tracking data, specific identification information can be set and used to uniquely identify the tracking data. Among them, the identification information of the tracking data includes a check digit, timestamp information, and a universally unique identifier (Uniformly Unique Identifier, UU ID). The identification information of the tracking data may also include application identification information, and the application identification information is the ID information of the specific application that generates the tracking data. Among them, the check digit information is calculated based on a specific algorithm for the tracking data, and the check digit information is appended to the identification information of the tracking data.

[0036] In an embodiment of the present application, data or files can be compressed based on a compression service, thereby reducing the required storage space. After acquiring the tracking data, the acquired tracking data can be compressed based on the compression service to obtain a first compressed data block. Specifically, the tracking data can be compressed according to a preset compression algorithm, and the preset compression algorithm can select a compression algorithm with good compression performance according to actual needs. In addition to the compressed data block, the first compressed data block formed also includes the compression algorithm identification information used. Based on the compression algorithm identification information, the compression algorithm used by the compressed data block can be determined, so that the corresponding decompression algorithm can be determined in the subsequent decompression process.

[0037] After obtaining the first compressed data block, the first compressed data block can be stored in a data file based on the shard storage service. Based on the shard storage service, the data set to be stored can be divided into multiple parts, and then the data is stored in the relevant physical locations of the corresponding shards. A data file is a file stored in a disk, and at least one compressed data block can be stored in the data file. For example, the data file can be a Data file.

[0038] S120, traverse the first compressed data block stored in the data file, and determine the memory index information based on the check bit in the identification information of the tracking data in the first compressed data block, the offset of the first compressed data block in the data file, and the identification information of the data file.

[0039] The offset of the first compressed data block in the data file refers to the offset of the storage position of the first compressed data block in the data file. The identification information of the data file is information that uniquely represents the data file, for example, the identification information of the data file may be data file ID information.

[0040] It should be noted that, since the first compressed data block may contain multiple tracking data, for each tracking data in the first compressed data block, a number of memory index information equal to the number of tracking data contained in the first compressed data block will be determined. Exemplarily, if the first compressed data block contains two tracking data T1 and T2, then for this first compressed data block, two memory index information will be determined, the first memory index information is determined based on the check bit in the identification information of the tracking data T1, the offset of the first compressed data block in the data file, and the identification information of the data file, and the second memory index information is determined based on the check bit in the identification information of the tracking data T2, the offset of the first compressed data block in the data file, and the identification information of the data file. In other words, the number of memory index information determined is equal to the number of tracking data contained in the first compressed data block. For the multiple memory index information generated for the same compressed data block, only the check bit field in the tracking data identification information in the memory index information is different, but the two fields about the offset of the first compressed data block in the data file and the identification information of the data file are the same.

[0041] S130. Store the memory index information into the memory index, where the preset form of the memory index is a multi-level hash index.

[0042] After determining the memory index information, the memory index information can be used to construct a corresponding memory index, thereby facilitating subsequent related data query operations. Specifically, the preset form of the memory index can be a multi-stage hash index. Among them, the multi-stage hash index is logically represented as multiple levels, in a ladder shape, but in actual physics it is still a storage structure of a continuous address. The first level of the multi-stage hash index can accommodate the largest number of elements. As the number of levels increases, the number of elements that can be accommodated in each level decreases. The length of each level in the multi-stage hash index is pre-set.

[0043] The technical solution provided by the present invention compresses the acquired tracking data based on the compression service to obtain the first compressed data block, and stores the first compressed data block in the data file based on the shard storage service; traverses the first compressed data block stored in the data file, and determines the memory index information based on the check bit in the identification information of the tracking data in the first compressed data block, the offset of the first compressed data block in the data file, and the identification information of the data file; and stores the memory index information in the memory index. By adopting the solution of the present invention, storage space can be saved by compressing the tracking data, and a fast response to data queries can be achieved by constructing the memory index based on the memory index information; at the same time, the data storage method of the present invention can significantly improve the scalability of data storage and indexing, and can effectively cope with the increase of massive data.

[0044] Figure 2 This is a flow chart of another data storage method provided by an embodiment of the present invention. The technical solution of this embodiment further optimizes the process after the memory index information is stored in the memory index in the above embodiment on the basis of the technical solution of the above embodiment. This embodiment can be combined with various optional solutions in one or more of the above embodiments. Figure 2 As shown, the data storage method of the embodiment of the present invention may include the following process:

[0045] S210: compress the acquired tracking data based on the compression service to obtain a first compressed data block, and store the first compressed data block into a data file based on the shard storage service.

[0046] As an optional but non-limiting implementation method, the obtained tracking data is compressed based on the compression service, including: when the amount of obtained tracking data reaches a preset amount, the obtained tracking data is compressed based on the compression service; or, when the time for obtaining the tracking data reaches a preset duration, the obtained tracking data is compressed based on the compression service.

[0047] Since the generation of tracking data is relatively frequent and intensive, multiple tracking data may be generated in the same second, and the size of the tracking data is not completely consistent. For the tracking data obtained, you can choose to compress multiple tracking data into a compressed data block. Specifically, when the number of tracking data obtained reaches a preset number, the obtained tracking data can be compressed based on the compression service. Alternatively, when the time to obtain the tracking data reaches a preset duration, the obtained tracking data can be compressed based on the compression service. For example, it can be set to compress the 20 tracking data obtained into a compressed data block when the number of tracking data obtained reaches 20; it can also be set to compress the tracking data obtained within 5 minutes into a compressed data block when 5 minutes of tracking data are obtained.

[0048] As an optional but non-limiting implementation, storing the first compressed data block into a data file based on the shard storage service includes:

[0049] If the generation time of the tracking data in the first compressed data block is within the set time period of the target shard area, the first compressed data block is stored in the data file corresponding to the target shard area; wherein the target shard area is the currently open shard area, and the shard area is divided based on the time sequence and the preset time length.

[0050] Specifically, when using a shard storage service, multiple shard areas can be divided based on the time sequence and the preset time length. For example, taking the preset time length of 1 hour as an example, a shard area can be divided every 1 hour according to the time sequence, so for the time period from 3 to 6 o'clock, three shard areas will be divided. The set time period corresponding to the first shard area is 3 to 4 o'clock, the set time period corresponding to the second shard area is 4 to 5 o'clock, and the set time period corresponding to the third shard area is 5 to 6 o'clock. And so on, new shard areas can be continuously divided as time goes by. Among them, the preset time length can be set based on actual needs, and can be set to 1 hour, 2 hours, etc.

[0051] Since there is a certain time sequence between each shard area, and each shard area has its own set time limit, then for the first compressed data block, the target shard area where the compressed data block is to be stored can be determined based on the generation time of the tracking data in the compressed data block. Specifically, based on the fact that the generation time of the tracking data in the first compressed data block is within the set time period of the target shard area, the first compressed data block can be stored in the data file corresponding to the target shard area. Exemplarily, if the generation time of the tracking data in the first compressed data block is 3:15, it can be determined that the generation time of these tracking data belongs to the shard area with a set time period of 3 to 4 o'clock, then the shard area with a set time period of 3 to 4 o'clock is the target shard area, and the first compressed data block can be stored in the data file corresponding to the target shard area.

[0052] S220, traverse the first compressed data block stored in the data file, and determine the memory index information based on the check bit in the identification information of the tracking data in the first compressed data block, the offset of the first compressed data block in the data file, and the identification information of the data file.

[0053] S230, storing the memory index information into the memory index, the preset form of the memory index is a multi-level hash index.

[0054] As an optional but non-limiting implementation, storing the memory index information into the memory index may include the following steps A1-A5:

[0055] Step A1: Calculate a first hash value corresponding to the tracking data in the first compressed data block.

[0056] Specifically, the tracing data in the first compressed data block may be calculated by means of a hash function to generate a first hash value corresponding to the tracing data.

[0057] It should be noted that when generating a hash value, a corresponding hash value is calculated for each tracking data in the compressed data block, and then the index information corresponding to each tracking data is stored in the memory index.

[0058] Step A2: The remainder is obtained by performing a modulo calculation on the first hash value and the current order length in the memory index.

[0059] The current level is the level currently being calculated and stored. Generally, the first level in the multi-level hash index is used as the current level by default, but as the calculation process progresses, other subsequent levels can also be used as the current level. After calculating the first hash value, the remainder can be obtained by calculating the remainder of the first hash value and the length of the current level in the memory index.

[0060] Step A3: Search the storage unit at the position corresponding to the remainder in the current order of the memory index.

[0061] After the remainder is obtained, the storage unit at the position corresponding to the remainder can be searched in the current order of the memory index. For example, if the remainder obtained is 5, it indicates that the storage unit at the fifth position of the current order needs to be searched. Specifically, it can start from the first storage unit of the current order and search all the way to the fifth storage unit of the current order.

[0062] Step A4: If the storage unit does not store data, the memory index information is stored in the storage unit.

[0063] After the storage unit is found, if there is no data stored in the storage unit, it means that there is a vacant position at the storage unit location, and the memory index information can be stored in the storage unit.

[0064] Step A5: If the storage unit has stored data, the memory index information is transferred to the next stage after the current stage for calculation and storage.

[0065] If there is no data stored in the storage unit, it means that data content has been inserted at the location of the storage unit, and the memory index information needs to be transferred to the next level after the current level, and the calculation and storage will continue. Specifically, after transferring to the next level, the remainder calculation can continue to be performed based on the first hash value and the length of the next level in the memory index, and the new remainder obtained can be used to search in this level at this time, and continue to determine whether data has been stored at the corresponding position at this time. The detailed calculation and storage principle is similar to the above process, and so on, until the memory index information is successfully stored in the memory index.

[0066] S240. Find the corresponding shard area based on the time in the identification information of the data to be queried.

[0067] After the tracking data is stored, a data query operation can also be performed. Specifically, after the data to be queried is determined, the shard area where the data to be queried is located can be determined based on the time in the identification information of the data to be queried. The time in the identification information of the data to be queried can be determined based on the timestamp information in the identification information of the data to be queried, and then the shard area corresponding to the time period can be determined.

[0068] S250: Calculate a second hash value corresponding to the data to be queried.

[0069] Specifically, the second hash value corresponding to the data to be queried may be calculated based on a hash function.

[0070] S260: Search in the index file based on the second hash value to find out the second index information that is the same as the check bit in the identification information of the data to be queried.

[0071] After determining the second hash value, a search can be performed in the index file based on the second hash value. Specifically, the search can be performed starting from the first level of the index file, by calculating the remainder of the second hash value and the length of the first level of the index file, and searching for index information at a position corresponding to the remainder in the first level. Then, the check bit information in the determined index information is compared with the check bit information in the identification information of the data to be searched to see if they are the same.

[0072] If the comparison result of the check digit is the same, the index information determined at this time is the second index information to be searched; if the comparison result of the check digit is not the same, then continue to transfer to the second level of the index file for query, recalculate the second hash value and the second level length of the index file, and find the index information at the corresponding position in the second level based on the newly determined remainder, and then continue a new round of check digit comparison. And so on, until the second index information with the same check digit as the identification information of the data to be queried is found in the index file.

[0073] S270. Obtain a corresponding second compressed data block based on the second index information, and decompress the data block.

[0074] Since the index information records not only the check bit information but also the offset of the compressed data block in the data file and the identification information of the data file, the corresponding second compressed data block can be obtained based on the second index information and the second compressed data block can be decompressed.

[0075] Specifically, during decompression, the corresponding decompression algorithm may be determined based on the compression algorithm identification information in the compressed data block, and the corresponding compressed data block may be decompressed to obtain each tracking data in the second compressed data block.

[0076] S280: traverse and compare the identification information of each tracking data obtained after decompression with the identification information of the data to be queried.

[0077] For each tracking data obtained after decompression, identification information of each tracking data may be traversed and compared with identification information of the data to be queried to determine the data to be queried.

[0078] S290: The tracking data having the same identification information as the identification information of the data to be queried is taken as the target tracking data, and returned.

[0079] The tracking data with consistent identification information found is the target tracking data, and the target tracking data is the data to be queried, and the target tracking data can be returned.

[0080] As an optional but non-limiting implementation, after storing the memory index information into the memory index, the data storage method of the embodiment of the present invention further includes:

[0081] The updated content of the memory index, the order of the updated content in the memory index, and the offset are stored in a log file; wherein the log file is used to restore the memory index after the memory index is lost; the updated content is the updated memory index information.

[0082] When the memory index is updated, the update details of the memory index can be recorded in time, so that the memory index can be restored after the memory index is lost. Specifically, the updated content of the memory index is the updated memory index information, and the updated content of the memory index, the order of the updated content in the memory index, and the offset can be stored in a log file. For example, the log file can be a wal file (Write-Ahead Logging). Figure 3 As shown, Figure 3 A schematic diagram of storage logic between a memory index and a log file provided in an embodiment of the present invention.

[0083] As an optional but non-limiting implementation, the data storage method of the embodiment of the present invention may further include the following steps B1-B2:

[0084] Step B1: a preset number of shard areas whose set time periods are closest to the current time are taken as currently open shard areas, and the shard areas before the currently open shard areas are closed.

[0085] For the multiple shard areas formed by division, if all the shard areas are set to an open state, it may be detrimental to the management and use of the shard areas. You can choose to set some shard areas to an open state and some shard areas to a closed state. Specifically, a preset number of shard areas whose set time periods of the shard areas are closest to the current time can be used as the currently open shard areas, and the shard areas located before the currently open shard areas can be closed.

[0086] The current moment is the moment of data storage. Since the tracking data is continuously generated in chronological order, the current moment can be used as a key dividing point when storing the tracking data, and the preset number of shards that are closest to the set time period of the shards and the current moment can be used as the currently open shards. Then, the newly generated tracking data can be written into the currently open shards.

[0087] Specifically, the preset number of shard areas closest to the current moment can be determined by first determining the shard area to which the current moment belongs, and then determining the preset number of adjacent shard areas forward or backward based on the shard area to which the current moment belongs. Exemplarily, if the current time is 4:20 and the number of slice areas to be opened is pre-set to three, it can be determined that the current time belongs to the slice area from 4 to 5 o'clock, and the three adjacent slice areas can be determined forward or backward based on the determined slice area. Specifically, the three slice areas determined can be: the slice area from 2 to 3 o'clock, the slice area from 3 to 4 o'clock, and the slice area from 4 to 5 o'clock; the three slice areas determined can also be: the slice area from 3 to 4 o'clock, the slice area from 4 to 5 o'clock, and the slice area from 5 to 6 o'clock; the three slice areas determined can also be: the slice area from 4 to 5 o'clock, the slice area from 5 to 6 o'clock, and the slice area from 6 to 7 o'clock.

[0088] It should be noted that the preset number of currently open shard areas can be determined based on actual needs. In addition to three as in the above example, it can also be two, etc.

[0089] The currently open shard area can ensure the continuous writing of the newly generated tracking data, and the other shard areas before the currently open shard area can be closed.

[0090] Step B2: After the shard area is closed, the memory index file corresponding to the closed shard area is transferred from the memory to the disk.

[0091] After the shards are closed, the memory index files corresponding to the closed shards can be transferred from the memory to the disk. After the memory index files are stored in the disk, the memory index files will become index files. The index files are stored in the disk and will not be lost due to power failures.

[0092] By setting a preset number of shards that are closest to the current time in multiple shards to an open state, closing the shards before the currently open shards, and writing the memory index files corresponding to the closed shards to the disk, it is easy to manage and use each shard partition.

[0093] It should be noted that for each shard, each shard has its own corresponding index file, data file, and log file. The difference is that for the currently open shard, its index file is a memory index file, which is specifically built in the memory; while for the closed shard, its corresponding index file is no longer in the memory, but has been written to the disk index file. Figure 4 As shown, Figure 4 A schematic diagram of a sharding area provided in an embodiment of the present invention.

[0094] As an optional but non-limiting implementation, searching in the memory index file based on the second hash value includes:

[0095] If the shard area corresponding to the data to be queried is in a closed state, the index file is obtained from the disk; if the shard area corresponding to the data to be queried is in an open state, the index file is obtained from the memory.

[0096] The shards and sections corresponding to the data to be queried may be in an open state or a closed state. If the shards and sections corresponding to the data to be queried are in a closed state, the index file of the corresponding shards and sections is obtained from the disk, and the obtained index file is used as the index file to perform subsequent query solutions. If the shards and sections corresponding to the data to be queried are in an open state, the corresponding memory index file is obtained from the memory, and the obtained memory index file is used as the index file to perform subsequent query solutions.

[0097] The technical solution provided by the present invention compresses the acquired tracking data based on the compression service to obtain the first compressed data block, and stores the first compressed data block into the data file based on the shard storage service; traverses the first compressed data block stored in the data file, and determines the memory index information based on the check bit in the identification information of the tracking data in the first compressed data block, the offset of the first compressed data block in the data file, and the identification information of the data file; and stores the memory index information into the memory index. When performing data queries, it is also possible to respond quickly based on the index file, with high query efficiency, and can quickly respond to complex query requirements. By adopting the solution of the present invention, storage space can be saved by compressing the tracking data, and a memory index can be constructed based on the memory index information to achieve a quick response to data queries; at the same time, the data storage method of the present invention can significantly improve the scalability of data storage and indexing, and can effectively cope with the increase of massive data.

[0098] Figure 5 This is a schematic diagram of the structure of a data storage device provided by an embodiment of the present invention. The embodiment of the present invention is applicable to the case of storing tracking data. The data storage device can be implemented in the form of software and / or hardware, and is generally integrated on any electronic device with network communication function, which can be a mobile terminal, PC or server, etc. Figure 5 As shown, the data storage device of the embodiment of the present invention may include a data processing module 510, a memory index information determination module 520 and a memory index construction module 530. Among them:

[0099] The data processing module 510 is used to compress the acquired tracking data based on the compression service to obtain a first compressed data block, and store the first compressed data block into a data file based on the shard storage service;

[0100] A memory index information determination module 520, configured to traverse the first compressed data block stored in the data file, and determine the memory index information based on the check bit in the identification information of the tracking data in the first compressed data block, the offset of the first compressed data block in the data file, and the identification information of the data file;

[0101] The memory index building module 530 is used to store the memory index information into the memory index, and the preset form of the memory index is a multi-level hash index.

[0102] As an optional but non-limiting implementation, storing the memory index information into the memory index includes:

[0103] Calculating a first hash value corresponding to the tracking data in the first compressed data block;

[0104] A remainder is obtained by performing a modulo calculation on the first hash value and the current order length in the memory index;

[0105] Searching for a storage unit at a position corresponding to the remainder in the current order of the memory index;

[0106] If the storage unit does not store data, storing the memory index information in the storage unit;

[0107] If the storage unit has stored data, the memory index information is transferred to the next stage after the current stage for calculation and storage.

[0108] As an optional but non-limiting implementation, storing the first compressed data block into a data file based on the shard storage service includes:

[0109] If the generation time of the tracking data in the first compressed data block is within the set time period of the target sharding area, the first compressed data block is stored in the data file corresponding to the target sharding area; wherein the target sharding area is a currently open sharding area, and the sharding area is divided based on the time sequence and the preset time length;

[0110] Accordingly, the method further includes:

[0111] The preset number of shards whose set time periods are closest to the current time are used as currently open shards, and the shards before the currently open shards are closed;

[0112] After the shard is closed, the memory index file corresponding to the closed shard is transferred from the memory to the disk.

[0113] As an optional but non-limiting implementation, after storing the memory index information into the memory index, the method further includes:

[0114] Storing the updated content of the memory index, the order of the updated content in the memory index, and the offset in a log file;

[0115] The log file is used to restore the memory index after the memory index is lost; and the updated content is the updated memory index information.

[0116] As an optional but non-limiting implementation, after storing the memory index information into the memory index, the method further includes:

[0117] Find the corresponding shard area based on the time in the identification information of the data to be queried;

[0118] Calculate a second hash value corresponding to the data to be queried;

[0119] Searching in the index file based on the second Hash value to find out the second index information that is the same as the check bit in the identification information of the data to be queried;

[0120] Acquire a corresponding second compressed data block based on the second index information, and decompress the data block;

[0121] Traverse and compare the identification information of each tracking data obtained after decompression with the identification information of the data to be queried;

[0122] The tracking data having the same identification information as the identification information of the data to be queried is taken as the target tracking data and returned.

[0123] As an optional but non-limiting implementation, the searching in the index file based on the second Hash value includes:

[0124] If the shard area corresponding to the data to be queried is in a closed state, obtaining the index file from the disk;

[0125] If the shard area corresponding to the data to be queried is in an open state, the index file is obtained from the memory.

[0126] As an optional but non-limiting implementation, compressing the acquired tracking data based on the compression service includes:

[0127] When the amount of the acquired tracking data reaches a preset amount, the acquired tracking data is compressed based on the compression service; or,

[0128] When the time to obtain the tracking data reaches a preset time, the obtained tracking data is compressed based on the compression service.

[0129] The technical solution provided by the present invention compresses the acquired tracking data based on the compression service to obtain the first compressed data block, and stores the first compressed data block in the data file based on the shard storage service; traverses the first compressed data block stored in the data file, and determines the memory index information based on the check bit in the identification information of the tracking data in the first compressed data block, the offset of the first compressed data block in the data file, and the identification information of the data file; and stores the memory index information in the memory index. By adopting the solution of the present invention, storage space can be saved by compressing the tracking data, and a fast response to data queries can be achieved by constructing the memory index based on the memory index information; at the same time, the data storage method of the present invention can significantly improve the scalability of data storage and indexing, and can effectively cope with the increase of massive data.

[0130] The data storage device provided in the embodiment of the present invention can be used to execute the data storage method, and has the corresponding functional modules and beneficial effects of executing the data storage method.

[0131] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present invention.

[0132] Figure 6 A schematic diagram of the structure of an electronic device for implementing a data storage method provided by an embodiment of the present invention. Figure 6 , which shows a schematic diagram of the structure of an electronic device 610 suitable for implementing an embodiment of the present invention. The terminal device in the embodiment of the present invention may include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0133] like Figure 6As shown, the electronic device 610 includes at least one processor 611, and a memory connected to the at least one processor 611 in communication, such as a read-only memory (ROM) 612, a random access memory (RAM) 613, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 611 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 612 or the computer program loaded from the storage unit 618 to the random access memory (RAM) 613. In the (RAM) 613, various programs and data required for the operation of the electronic device 610 can also be stored. The processor 611, (ROM) 612 and (RAM) 613 are connected to each other via a bus 614. An input / output (I / O) interface 615 is also connected to the bus 614.

[0134] A number of components in the electronic device 610 are connected to the I / O interface 615, including: an input unit 616, such as a keyboard, a mouse, etc.; an output unit 617, such as various types of displays, speakers, etc.; a storage unit 618, such as a disk, an optical disk, etc.; and a communication unit 619, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 619 allows the electronic device 610 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0135] The processor 611 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 611 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 611 executes the data storage method provided in any embodiment of the present invention.

[0136] In particular, according to an embodiment of the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the data storage method shown in the flowchart. When the computer program is executed by a processing device, the above functions defined in the data storage method of the embodiment of the present invention are executed.

[0137] The names of the messages or information exchanged between multiple devices in the embodiments of the present invention are only used for illustrative purposes, and are not used to limit the scope of these messages or information.

[0138] The electronic device provided in the embodiment of the present invention and the data storage method provided in the above embodiment belong to the same inventive concept. The technical details not fully described in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0139] An embodiment of the present invention provides a computer storage medium on which a computer program is stored. When the program is executed by a processor, the data storage method provided by the above embodiment is implemented.

[0140] It should be noted that the computer-readable medium of the present invention can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0141] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0142] Computer program code for performing the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0143] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0144] The units involved in the embodiments of the present invention may be implemented by software or hardware, wherein the name of a unit does not limit the unit itself in some cases.

[0145] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0146] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0147] The above description is only a preferred embodiment of the present invention and an explanation of the technical principles used. Those skilled in the art should understand that the disclosure scope involved in the present invention is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present invention (but not limited to) by each other.

[0148] In addition, although each operation is described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the present invention. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0149] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A data storage method, characterized in that: The method comprises: Compressing the acquired tracking data based on the compression service to obtain a first compressed data block, and storing the first compressed data block into a data file based on the shard storage service; Traversing the first compressed data block stored in the data file, and determining the memory index information based on the check bit in the identification information of the tracking data in the first compressed data block, the offset of the first compressed data block in the data file, and the identification information of the data file; The memory index information is stored in a memory index, and a preset form of the memory index is a multi-level hash index.

2. The method according to claim 1, characterized in that The storing the memory index information into the memory index includes: Calculating a first hash value corresponding to the tracking data in the first compressed data block; A remainder is obtained by performing a modulo calculation on the first hash value and the current order length in the memory index; Searching for a storage unit at a position corresponding to the remainder in the current order of the memory index; If the storage unit does not store data, storing the memory index information in the storage unit; If the storage unit has stored data, the memory index information is transferred to the next stage after the current stage for calculation and storage.

3. The method according to claim 1, characterized in that The storing the first compressed data block into a data file based on the shard storage service includes: If the generation time of the tracking data in the first compressed data block is within the set time period of the target sharding area, the first compressed data block is stored in the data file corresponding to the target sharding area; wherein the target sharding area is a currently open sharding area, and the sharding area is divided based on the time sequence and the preset time length; Accordingly, the method further includes: The preset number of shards whose set time periods are closest to the current time are used as currently open shards, and the shards before the currently open shards are closed; After the shard is closed, the memory index file corresponding to the closed shard is transferred from the memory to the disk.

4. The method according to claim 1, characterized in that: After storing the memory index information into the memory index, the method further includes: Storing the updated content of the memory index, the order of the updated content in the memory index, and the offset in a log file; The log file is used to restore the memory index after the memory index is lost; and the updated content is the updated memory index information.

5. The method according to claim 1, characterized in that: After storing the memory index information into the memory index, the method further includes: Find the corresponding shard area based on the time in the identification information of the data to be queried; Calculate a second hash value corresponding to the data to be queried; Searching in the index file based on the second Hash value to find out the second index information that is the same as the check bit in the identification information of the data to be queried; Acquire a corresponding second compressed data block based on the second index information, and decompress the data block; Traverse and compare the identification information of each tracking data obtained after decompression with the identification information of the data to be queried; The tracking data having the same identification information as the identification information of the data to be queried is taken as the target tracking data and returned.

6. The method according to claim 5, characterized in that The searching in the index file based on the second Hash value includes: If the shard area corresponding to the data to be queried is in a closed state, obtaining the index file from the disk; If the shard area corresponding to the data to be queried is in an open state, the index file is obtained from the memory.

7. The method according to claim 1, characterized in that The compressing the acquired tracking data based on the compression service includes: When the amount of the acquired tracking data reaches a preset amount, the acquired tracking data is compressed based on the compression service; or, When the time to obtain the tracking data reaches a preset time, the obtained tracking data is compressed based on the compression service.

8. A data storage device, characterized in that: The device comprises: A data processing module, configured to compress the acquired tracking data based on a compression service to obtain a first compressed data block, and store the first compressed data block into a data file based on a shard storage service; a memory index information determination module, configured to traverse the first compressed data block stored in the data file, and determine the memory index information based on a check bit in the identification information of the tracking data in the first compressed data block, an offset of the first compressed data block in the data file, and the identification information of the data file; A memory index building module is used to store the memory index information into a memory index, and the preset form of the memory index is a multi-stage hash index.

9. An electronic device, characterized in that: The electronic device comprises: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the data storage method according to any one of claims 1 to 7.

10. A storage medium containing computer executable instructions, characterized in that: The computer executable instructions are used to perform the data storage method according to any one of claims 1 to 7 when executed by a computer processor.

Images