Data query method, data storage method, device, computing device and medium

Abstract

The present disclosure provides a data query method, including: receiving a query request, the query request at least including attribute information of target message data; based on the query request, determining target index data from multiple index data, wherein the target index data includes a target file path associated with the attribute information of the target message data, and each index data in the multiple index data includes the attribute information of historical message data and the file path of the file where the historical message data is located; based on the target index data, determining a target file from at least one file, wherein the file path of the target file is the target file path, and at least one file is used to store historical message data; obtaining the target message data from the target file. The present disclosure also provides a data storage method, apparatus, computing device, computer-readable storage medium, and computer program product.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and more specifically, to a data query method, a data storage method, a data query device, a data storage device, a computing device, and a computer-readable storage medium. Background Art

[0002] Faced with the large amount of message data generated in the network, related technologies usually use big data technologies such as Kafka or Spark to store message data and provide support for subsequent queries. However, the message data storage method of related technologies usually processes the message data and then stores it in the database, resulting in the inability to obtain the original data of the message during subsequent queries, making it difficult to meet the forensic needs of message data. In addition, when querying the stored message data, the related technologies have a large computing resource overhead and a slow query speed. Summary of the invention

[0003] In view of this, the present disclosure provides an optimized data query method, a data storage method, a data query device, a data storage device, a computing device, and a computer-readable storage medium.

[0004] One aspect of the present disclosure provides a data query method, including: receiving a query request, the query request including at least attribute information of target message data; based on the query request, determining target index data from multiple index data, wherein the target index data includes a target file path associated with the attribute information of the target message data, each index data in the multiple index data includes the attribute information of historical message data and the file path of the file where the historical message data is located; based on the target index data, determining a target file from at least one file, wherein the file path of the target file is the target file path, the at least one file is used to store the historical message data, and obtaining the target message data from the target file.

[0005] According to an embodiment of the present disclosure, the above-mentioned multiple index data are stored in multiple first databases, and the database identifier and index data identifier of each of the first databases are associated and stored in the second database, and the index data identifier represents the index data stored in the first database. Wherein, the step of determining the target index data from the multiple index data based on the query request includes: determining the index data identifier indicated by the query request from the second database based on the query request, determining at least one database identifier associated with the indicated index data identifier from the second database based on the index data identifier indicated by the query request, determining at least one first database corresponding to the at least one database identifier based on the at least one database identifier, and determining the index data whose attribute information matches the attribute information of the target message data from the index data stored in the at least one first database as the target index data.

[0006] According to an embodiment of the present disclosure, the above-mentioned multiple index data are stored in the multiple first databases according to the timestamps of the historical message data, the index data identifier includes the timestamp range of the index data stored in the first database, and the query request also includes a target time range, and the timestamp of the target message data is within the target time range. Wherein, based on the query request, determining the index data identifier indicated by the query request from the second database includes: based on the target time range in the query request, determining the index data identifier indicated by the query request from the second database, and the timestamp range of the index data identifier indicated by the query request includes the target time range.

[0007] According to an embodiment of the present disclosure, the target file includes multiple historical message data. Wherein, obtaining the target message data from the target file includes: based on the attribute information of the target message data, determining at least one historical message data from the multiple historical message data as the target message data, wherein the attribute information of the at least one historical message data matches the attribute information of the target message data.

[0008] According to an embodiment of the present disclosure, the above-mentioned at least one file is a file in a distributed file system; the at least one file corresponds one-to-one to at least one preset time range, and for each file in the at least one file, the message generation time of each historical message data stored in the file is within the preset time range corresponding to the file.

[0009] According to an embodiment of the present disclosure, for each file in the at least one file, the multiple historical message data stored in the file are compressed into multiple sub-files; for each sub-file, the multiple historical message data in the sub-file are compressed in sequence. The multiple historical message data are compressed in sequence, including: compressing at least one received historical message data to obtain a preliminary compressed sub-file, and compressing at least one newly received historical message data into the preliminary compressed sub-file. The file path also includes the file name of the sub-file.

[0010] According to an embodiment of the present disclosure, the above-mentioned attribute information includes at least one of the following: source IP address, destination IP address, source port, destination port, and data transmission protocol.

[0011] According to an embodiment of the present disclosure, the first database includes a bitmap database, and the second database includes a bitmap database.

[0012] Another aspect of the present disclosure provides a data storage method, including: obtaining historical message data to be stored, parsing each historical message data in the historical message data to be stored, obtaining attribute information of each historical message data, storing the historical message data to be stored in at least one file in a distributed file system, and recording the file path of each file where the historical message data is located, for each historical message data, determining the attribute information of the historical message data and the file path of the file where the historical message data is located as index information, and storing the index information in association with a bitmap database.

[0013] According to an embodiment of the present disclosure, the above-mentioned at least one file corresponds one-to-one to at least one preset time range; storing the historical message data to be stored in at least one file in the distributed file system includes, for each of the historical message data: determining the message generation time of the historical message data, and based on the message generation time and the at least one preset time range, storing the historical message data in one of the at least one files, wherein the message generation time is within the preset time range corresponding to the stored file.

[0014] According to an embodiment of the present disclosure, for each file in the at least one file, the multiple historical message data stored in the file are compressed into multiple sub-files; for each sub-file: at least one received historical message data is compressed to obtain a preliminary compressed sub-file, and at least one newly received historical message data is compressed into the preliminary compressed sub-file.

[0015] Another aspect of the present disclosure provides a data query device, including: a receiving module, a first determination module, a second determination module and a first acquisition module. The receiving module is used to receive a query request, and the query request at least includes attribute information of the target message data. The first determination module is used to determine the target index data from multiple index data based on the query request, wherein the target index data includes a target file path associated with the attribute information of the target message data, and each index data in the multiple index data includes the attribute information of the historical message data and the file path of the file where the historical message data is located. The second determination module is used to determine the target file from at least one file based on the target index data, wherein the file path of the target file is the target file path, and the at least one file is used to store the historical message data. The first acquisition module is used to obtain the target message data from the target file.

[0016] Another aspect of the present disclosure provides a data storage device, including: a second acquisition module, a parsing module, a first storage module, a third determination module and a second storage module. Among them, the second acquisition module is used to obtain historical message data to be stored. The parsing module is used to parse each historical message data in the historical message data to be stored to obtain attribute information of each historical message data. The first storage module is used to store the historical message data to be stored in at least one file in a distributed file system, and record the file path of each file where the historical message data is located. The third determination module is used to determine the attribute information of the historical message data and the file path of the file where the historical message data is located as index information for each historical message data. The second storage module is used to associate the index information and store it in a bitmap database.

[0017] Another aspect of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, which are used to implement the above method when executed.

[0018] Another aspect of the present disclosure provides a computer program product, which includes computer executable instructions, and when the instructions are executed, are used to implement the method as described above.

[0019] According to the embodiments of the present disclosure, the use of the above-mentioned data query method and data storage method can at least partially solve the problems that the message data storage method of the related technology is difficult to meet the forensic needs of message data, the query computing resource overhead is large, and the query speed is slow. Therefore, it is possible to obtain unprocessed original message data when querying forensics, improve the query speed, and reduce the resource consumption of data query. Technical effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The above and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0021] Figure 1 The system architecture of the data query method and the data storage method according to the embodiment of the present disclosure is schematically shown;

[0022] Figure 2 The principle diagram of the data query method and the data storage method according to the embodiment of the present disclosure is schematically shown;

[0023] Figure 3 A flowchart of a data query method according to an embodiment of the present disclosure is schematically shown;

[0024] Figure 4 A schematic diagram showing a principle diagram of index data storage according to an embodiment of the present disclosure is shown;

[0025] Figure 5 A flowchart for determining target index data according to an embodiment of the present disclosure is schematically shown;

[0026] Figure 6 A flowchart of a data storage method according to an embodiment of the present disclosure is schematically shown;

[0027] Figure 7 A block diagram of a data query device according to an embodiment of the present disclosure is schematically shown;

[0028] Figure 8 A block diagram schematically shows a data storage device according to an embodiment of the present disclosure; and

[0029] Fig. 9 A block diagram of a computer system suitable for data query and data storage according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0030] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0031] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "include", "comprising", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0032] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.

[0033] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0034] An embodiment of the present disclosure provides a data query method, including: receiving a query request, the query request at least including attribute information of target message data; based on the query request, determining the target index data from multiple index data, wherein the target index data includes a target file path associated with the attribute information of the target message data, each index data in the multiple index data includes the attribute information of historical message data and the file path of the file where the historical message data is located; based on the target index data, determining the target file from at least one file, wherein the file path of the target file is the target file path, at least one file is used to store historical message data, and obtaining the target message data from the target file.

[0035] An embodiment of the present disclosure also provides a data storage method for storing historical message data, the method comprising: obtaining multiple historical message data to be stored, parsing each of the multiple historical message data to be stored, obtaining attribute information of each historical message data, storing the multiple historical message data to be stored in at least one file in a distributed file system, and for each historical message data, associating the attribute information of the historical message data with the file path of the file where the historical message data is located and storing it in a bitmap database to obtain index data for each historical message data.

[0036] Figure 1 The system architecture of the data query method and data storage method according to the embodiment of the present disclosure is schematically shown. It should be noted that: Figure 1 What is shown is merely an example of a system architecture to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0037] like Figure 1As shown, the system architecture 100 according to this embodiment may include forwarding devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the forwarding devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0038] The forwarding devices 101, 102, 103 interact with the server 105 through the network 104 to receive or send messages, etc. The forwarding devices 101, 102, 103 may include but are not limited to routers, switches, gateways, etc.

[0039] The server 105 may be a server providing various services, such as providing storage function (only for example) for message data from forwarding devices 101, 102, 103. The server 105 may analyze and process received query requests, and obtain target message data for the query request.

[0040] It should be noted that the data query method and data storage method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the data query device and data storage device provided in the embodiment of the present disclosure can generally be set in the server 105. The data query method and data storage method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the forwarding devices 101, 102, 103 and / or the server 105. Accordingly, the data query device and data storage device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the forwarding devices 101, 102, 103 and / or the server 105.

[0041] It should be understood that Figure 1 The number of forwarding devices, networks and servers in the embodiment is only for illustration. Any number of forwarding devices, networks and servers may be provided according to implementation requirements.

[0042] Figure 2 The principle diagram of the data query method and the data storage method according to the embodiment of the present disclosure is schematically shown.

[0043] like Figure 2 As shown, the multiple historical message data 210 to be stored are obtained from devices such as routers and switches. For example, the multiple historical message data 210 include historical message data 211, historical message data 212, historical message data 213, and historical message data 214.

[0044] Each historical message data is parsed to obtain the attribute information of each historical message data. For example, the attribute information of the historical message data 211 is "attribute A", the attribute information of the historical message data 212 is "attribute B", the attribute information of the historical message data 213 is "attribute C", and the attribute information of the historical message data 214 is "attribute D". The attribute information may include the quaternion of the message, and the quaternion includes the source IP address, the destination IP address, the source port, and the destination port.

[0045] Next, the plurality of historical message data 210 are stored in a plurality of files 221 and 222, and the plurality of files 221 and 222 can be stored in a distributed file system. For example, the historical message data 211 and the historical message data 212 are compressed and stored in the file 221, and the historical message data 213 and the historical message data 214 are compressed and stored in the file 222. Among them, the file path of the file 221 is, for example, "path a", and the file path of the file 222 is, for example, "path b", and the corresponding file can be found through the file path.

[0046] The attribute information of each historical message data is associated with the file path of the file stored therein, and an index file 230 is obtained. The index file 230 is stored in a bitmap database, for example. The index file 230 includes a plurality of index data corresponding to the plurality of historical message data. The plurality of index data includes, for example, the associated data of "attribute A" and "path a", the associated data of "attribute B" and "path a", the associated data of "attribute C" and "path b", and the associated data of "attribute D" and "path b".

[0047] After storing the historical message data in the multiple files 221, 222 and generating the index file 230, the target message data 260 can be obtained from the multiple historical message data based on the received query request 240. Specifically, the query request 240 includes, for example, the required attribute information of the target message data, for example, the attribute information included in the query request 240 is "attribute A".

[0048] Then, based on the "attribute A" in the query request 240, target index data 250 is determined from the multiple index data in the index file 230. The target index data 250 is, for example, the association data of "attribute A" and "path a". Then, based on the "path a" in the target index data 250, the file 221 with the file path "path a" is determined from the multiple files 221 and 222 as the target file, and then the file 221 is decompressed, and then the historical message data 211 with the attribute data "attribute A" is obtained from the decompressed file 221 based on the "attribute A" in the query request 240 as the target message data 260.

[0049] The following combination Figure 2 The data query method and data storage method of the embodiments of the present disclosure are described with reference to the schematic diagram.

[0050] Figure 3 The flowchart of the data query method according to the embodiment of the present disclosure is schematically shown.

[0051] like Figure 3 As shown, the method may include, for example, the following operations S310 to S340.

[0052] In operation S310, a query request is received, the query request at least including attribute information of target message data, wherein the query request is used to query target message data from a plurality of stored historical message data, for example.

[0053] In operation S320 , target index data is determined from a plurality of index data based on the query request.

[0054] In an embodiment of the present disclosure, multiple index data correspond to multiple historical message data, for example, one-to-one, that is, each index data includes attribute information of the corresponding historical message data and the file path of the file where the historical message data is located. Among them, multiple index data are stored in a bitmap database, for example, which has an advantage in query efficiency.

[0055] Based on the attribute information in the query request, index data having the attribute information is determined from a plurality of index data as target index data, and the target index data includes a target file path associated with the attribute information.

[0056] In operation S330, a target file is determined from at least one file based on the target index data, wherein the file path of the target file is the target file path, and the at least one file is used to store historical message data.

[0057] In an embodiment of the present disclosure, at least one file is stored in a distributed file system, for example, each file includes a plurality of compressed and stored historical message data, and each file has a file path. Based on the target file path in the target index data, a file whose file path is consistent with the target file path is determined from the at least one file as the target file.

[0058] In operation S340, target message data is obtained from the target file.

[0059] In an embodiment of the present disclosure, the target file includes multiple historical message data. Based on the attribute information of the target message data in the query request, at least one historical message data can be determined as the target message data from the multiple historical message data stored in the target file, and the attribute information of the at least one historical message data matches the attribute information of the target message data.

[0060] For example, after determining the target file, since the target file stores multiple historical message data, based on the attribute information in the query request, historical message data whose attribute information is consistent with the attribute information in the query request can be obtained from the target file as the required target message data.

[0061] In the embodiments of the present disclosure, the originality of the historical message data is ensured by storing the historical message data in a distributed file system, so that the unprocessed original message data can be obtained from the distributed file system when querying and collecting evidence for the message in the future. In addition, after the historical message data is stored in the distributed file system, the file path and the attribute information of the message are stored in a bitmap database as index data in association, so that it is convenient to find the index data from the bitmap database based on the attribute information to obtain the target file path of the target message data storage, and obtain the unprocessed target message data from the file based on the target file path, thereby improving the query speed and reducing the resource consumption of data query.

[0062] Figure 4 The schematic diagram shows a principle diagram of index data storage according to an embodiment of the present disclosure.

[0063] like Figure 4 As shown, a plurality of historical message data 410 includes historical message data 411-418 as an example. Historical message data 411 and 412 are stored in file 421, historical message data 413 and 414 are stored in file 422, historical message data 415 and 416 are stored in file 423, and historical message data 417 and 418 are stored in file 424. The file path of file 421 is "path a", the file path of file 422 is "path b", the file path of file 423 is "path c", and the file path of file 424 is "path d".

[0064] Multiple index data corresponding to multiple historical message data are stored in multiple first databases. For example, multiple index data corresponding to historical message data 411-414 are stored in the first database 431, and multiple index data corresponding to historical message data 415-418 are stored in the first database 432. The database identifier of the first database 431 is, for example, "first database P", and the database identifier of the first database 432 is, for example, "first database Q". The first database 431 includes a bitmap database, and the second database 432 includes a bitmap database.

[0065] The attribute information of each historical message data includes, for example, any one or more of source IP address, destination IP address, source port, destination port, and data transmission protocol. The attribute information of each historical message data is associated with the file path of the file stored therein and stored in the first database.

[0066] In addition, each historical message data has a timestamp, which, for example, represents the time when the message was generated. For ease of understanding, taking the timestamp of the historical message data 411 as "20200101" as an example, it means that the historical message data 411 was generated on January 1, 2020. However, the timestamp of each historical message data can also be accurately represented to a certain moment, for example, it can be represented as "20200101163020", indicating that the historical message data was generated at 16:30:20 on January 1, 2020. The embodiment of the present disclosure does not specifically limit the representation form of the timestamp.

[0067] Among them, multiple index data are stored in multiple first databases according to the timestamp of the historical message data. Each index data includes, for example, the timestamp, attribute information, and file path of the historical message data. For example, each first database can store 4 index data, then the index data corresponding to the multiple historical message data 411-418 are stored in the first database 431 in sequence from small (early) to large (late) according to the timestamp, so that the first database 431 stores multiple index data corresponding to the historical message data 411-414. When the first database 431 is full of 4 index data, the index data corresponding to the remaining historical message data 415-418 are stored in the second database 432 in sequence according to the timestamp.

[0068] Next, the database identifier and index data identifier of each first database are associated and stored in the second database 440. The index data identifier represents the index data stored in the first database, for example, the index data identifier includes the timestamp range of the index data stored in the first database. The timestamp range is represented by a minimum timestamp and a maximum timestamp, for example.

[0069] Taking the first database 431 as an example, the minimum timestamp of the index data in the first database 431 is "20200101", and the maximum timestamp is "20200104". The minimum timestamp "20200101", the maximum timestamp "20200104" and the database identifier "first database P" of the first database 431 are associated and stored in the second database 440. The process for the first database 432 is the same or similar, and will not be repeated here.

[0070] The following combination Figure 4 Schematic diagram and Figure 5 The flowchart of the present invention is used to describe the process of determining the target index data in an embodiment of the present disclosure.

[0071] Figure 5 The flowchart of determining target index data according to an embodiment of the present disclosure is schematically shown.

[0072] like Figure 5 As shown, the above operation S320 regarding determining target index data from a plurality of index data based on the query request includes the following operations S521 to S524.

[0073] In operation S521, based on the query request, an index data identifier indicated by the query request is determined from a second database.

[0074] In an embodiment of the present disclosure, the query request also includes a target time range, and the timestamp of the target message data is within the target time range. For example, when the target message data needs to be queried, the target time range for generating the target message data can be specified, for example, the target time range is from January 2, 2020 to January 3, 2020, and the time when the target message data is generated is within the target time range.

[0075] Then, based on the target time range in the query request, the index data identifier indicated by the query request is determined from the second database, and the indicated index data identifier is, for example, a minimum timestamp of "20200101" and a maximum timestamp of "20200104". It can be seen that the timestamp range of the index data identifier indicated by the query request (January 1, 2020 to January 4, 2020) includes the target time range (January 2, 2020 to January 3, 2020).

[0076] In operation S522, based on the index data identifier indicated by the query request, at least one database identifier associated with the indicated index data identifier is determined from the second database.

[0077] For example, the database identifier associated with the index data identifiers "20200101" and "20200104" indicated by the query request is "first database P".

[0078] In operation S523, based on the at least one database identifier, at least one first database corresponding to the at least one database identifier is determined. For example, the first database 431 corresponding to the database identifier "first database P" is determined.

[0079] In operation S524, index data whose attribute information matches the attribute information of the target message data is determined from the index data stored in the at least one first database as the target index data.

[0080] In one example, when the attribute information included in the query request is "destination port 31", the index data including "destination port 31" is determined from the first database 431 as the target index data, and the target index data includes, for example, the file path "path a". Next, the target message data is obtained from the file 421 corresponding to "path a", for example, one or more historical message data with the destination port "31" is obtained from the file 421 as the target message data.

[0081] In another example, when the attribute information included in the query request is "Telnet protocol", multiple index data including "Telnet protocol" are determined from the first database 431 as target index data, and the target index data includes, for example, file paths "path a" and "path b". Next, the target message data is obtained from the file 421 corresponding to "path a" and the file 422 corresponding to "path b", for example, one or more historical message data whose data transmission protocol is "Telnet protocol" is obtained from the file 421, and one or more historical message data whose data transmission protocol is "Telnet protocol" is obtained from the file 422, and the obtained historical message data is used as the target message data.

[0082] In another example, when the attribute information included in the query request is "destination port 31 or 32" and "Telnet protocol", the first index data including "destination port 31 or 32" and "Telnet protocol" is determined from the first database 431 as the target index data, and the target index data includes, for example, the file path "path a". Next, the target message data is obtained from the file 421 corresponding to "path a", for example, one or more historical message data with the destination port being "31 or 32" and the data transmission protocol being "Telnet protocol" is obtained from the file 421 as the target message data.

[0083] In another example, when the target time range in the query request is from January 4, 2020 to January 5, 2020, the index data identifiers indicated by the query request are, for example, "20200101" and "20200104" and "20200105" and "20200108". The first database corresponding to the index data identifier includes, for example, the first database 431 and the first database 432. Next, based on the attribute information in the query request, the target index data including the attribute information is determined from the first database 431 and the first database 432, and the target message data is obtained from the corresponding file based on the file path included in the target index data.

[0084] It can be understood that the embodiments of the present disclosure store index data of historical message data in multiple first databases and establish a second database for indexing multiple first databases. This makes it easy to first determine the corresponding first database from the second database when querying target message data, and then determine the file path where the target message data is stored from the determined first database without traversing all the first databases, thereby improving query efficiency and reducing the computing resources consumed by the query.

[0085] In an embodiment of the present disclosure, at least one file is a file in a distributed file system, and at least one file corresponds one-to-one to at least one preset time range. For each file in the at least one file, the message generation time of each historical message data stored in the file is within the preset time range corresponding to the file.

[0086] For example, at least one file includes file 1, file 2, file 3, etc., and the message generation time of the historical message data stored in each file is, for example, within 1 hour. For example, the preset time range corresponding to file 1 is 00:00:00 on January 1, 2020 to 0:59:59 on January 1, 2020. The preset time range corresponding to file 2 is, for example, 01:00:00 on January 1, 2020 to 1:59:59 on January 1, 2020. The preset time range corresponding to file 3 is, for example, 02:00:00 on January 1, 2020 to 2:59:59 on January 1, 2020. Taking the historical message data stored in file 1 as an example, the message generation time of each historical message data stored in file 1 is, for example, within the time range of 00:00:00 on January 1, 2020 to 0:59:59 on January 1, 2020.

[0087] For each file in at least one file, the multiple historical message data stored in the file are compressed into multiple sub-files. Taking file 1 as an example, the multiple historical message data stored in file 1 are compressed into sub-file 11, sub-file 12, sub-file 13, etc. For example, taking the example of sequentially generating 3000 historical message data within the time range of 00:00:00 on January 1, 2020 to 0:59:59 on January 1, 2020, each sub-file can store 1000 historical message data. For example, in the process of sequentially generating 3000 historical message data, the generated historical message data is sequentially stored in sub-file 11, and after sub-file 11 is full of 1000 historical message data, the historical message data generated thereafter is sequentially stored in sub-file 12 until sub-file 12 is full of 1000, and the subsequently generated messages are sequentially stored in sub-file 13.

[0088] For each sub-file, the multiple historical message data in the sub-file are compressed in sequence. Taking the sub-file 11 as an example, the 1000 historical message data stored in the sub-file 11 are compressed and stored in sequence.

[0089] In one embodiment, for the sub-file 11, the received historical message data can be compressed at preset intervals, and the preset interval can be 1 minute. For example, 200 historical message data received within 1 minute are compressed to obtain a preliminary compressed sub-file, and then 300 historical message data newly received in the subsequent 1 minute are compressed into the preliminary compressed sub-file. Among them, the 300 newly received historical message data can be compressed into the preliminary compressed sub-file by using a streaming compression technology, and the streaming compression technology has the function of continuing to compress new files into a compressed file. Therefore, for the newly received historical message data in each subsequent 1 minute, the newly received historical message data can be continuously compressed into the previous compressed sub-file until the sub-file 11 is full of 1000 historical message data, and finally the 1000 historical message data stored in the sub-file 11 are compressed into one file.

[0090] In another embodiment, for the sub-file 11, a compression process can be performed once with a preset number of historical file data each time, and the preset number is, for example, 200. For example, the 200 received historical message data are compressed to obtain a preliminary compressed sub-file, and then the subsequent 200 newly received historical message data are compressed into the preliminary compressed sub-file. Among them, the 200 newly received historical message data can be compressed into the preliminary compressed sub-file by a streaming compression technology. For each subsequent 200 newly received historical message data, the newly received historical message data can continue to be compressed into the previous compressed sub-file until the sub-file 11 is full of 1000 historical message data, and finally the 1000 historical message data stored in the sub-file 11 are compressed into one file.

[0091] In the embodiment of the present disclosure, the historical message data is compressed multiple times by the streaming compression technology to obtain a sub-file, which can reduce the peak consumption of storage space. For example, after waiting to receive 1000 historical message data, the 1000 historical message data are compressed into the sub-file 11 at one time. In the process of waiting to receive the full 1000 historical message data, the received message data will occupy a large storage space because it is not compressed. By compressing the historical message data multiple times by the streaming compression technology, the storage space occupied by the historical message data can be reduced.

[0092] In the embodiment of the present disclosure, since each file has multiple sub-files, each sub-file also has a file name. The file path for each file may also include the file name of the sub-file, so that when acquiring target message data, it can be acquired from the corresponding sub-file in the target file, thereby improving the data acquisition speed.

[0093] Figure 6 The flowchart of the data storage method according to the embodiment of the present disclosure is schematically shown.

[0094] like Figure 6 As shown, the method may include, for example, the following operations S610 to S650.

[0095] In operation S610, historical message data to be stored is obtained.

[0096] In operation S620, each historical message data in the historical message data to be stored is parsed to obtain attribute information of each historical message data.

[0097] The attribute information includes any one or more of the source IP address, the destination IP address, the source port, the destination port, and the data transmission protocol.

[0098] In operation S630, the historical message data to be stored is stored in at least one file in the distributed file system, and the file path of the file where each historical message data is located is recorded.

[0099] In operation S640, for each historical message data, attribute information of the historical message data and a file path of the file where the historical message data is located are determined as index information.

[0100] In operation S650, the index information is associated and stored in the bitmap database.

[0101] In an embodiment of the present disclosure, at least one file corresponds to at least one preset time range. Storing the historical message data to be stored in at least one file in the distributed file system includes: for each historical message data, determining the message generation time of the historical message data, based on the message generation time and at least one preset time range, storing the historical message data in one of the at least one file, wherein the message generation time is within the preset time range corresponding to the stored file.

[0102] In an embodiment of the present disclosure, for each file in at least one file, the multiple historical message data stored in the file are compressed into multiple sub-files. For each sub-file: at least one received historical message data is compressed to obtain a preliminary compressed sub-file, and at least one newly received historical message data is compressed into the preliminary compressed sub-file.

[0103] In the embodiments of the present disclosure, the originality of the historical message data is ensured by storing the historical message data in a distributed file system, so that the unprocessed original message data can be obtained from the distributed file system when querying and collecting evidence for the message in the future. In addition, after the historical message data is stored in the distributed file system, the file path and the attribute information of the message are stored in a bitmap database as index data in association, so that it is convenient to find the index data from the bitmap database based on the attribute information to obtain the target file path of the target message data storage, and obtain the unprocessed target message data from the file based on the target file path, thereby improving the query speed and reducing the resource consumption of data query.

[0104] Figure 7 A block diagram of a data query device according to an embodiment of the present disclosure is schematically shown.

[0105] like Figure 7 As shown, the data query device 700 may include: a receiving module 710 , a first determining module 720 , a second determining module 730 and a first acquiring module 740 .

[0106] The receiving module 710 may be used to receive a query request, where the query request includes at least attribute information of the target message data. Figure 3 The operation S310 described above will not be described in detail here.

[0107] The first determination module 720 may be used to determine target index data from multiple index data based on the query request, wherein the target index data includes a target file path associated with the attribute information of the target message data, and each index data in the multiple index data includes the attribute information of the historical message data and the file path of the file where the historical message data is located. According to the embodiment of the present disclosure, the first determination module 720 may, for example, execute the above reference Figure 3 The operation S320 described above will not be described in detail here.

[0108] The second determination module 730 may be used to determine a target file from at least one file based on the target index data, wherein the file path of the target file is the target file path, and the at least one file is used to store historical message data. According to an embodiment of the present disclosure, the second determination module 730 may, for example, execute the above reference Figure 3 The operation S330 described above will not be described in detail here.

[0109] The first acquisition module 740 may be used to acquire target message data from the target file. According to the embodiment of the present disclosure, the first acquisition module 740 may, for example, execute the above reference Figure 3 The operation S340 described above will not be described in detail here.

[0110] Figure 8 A block diagram of a data storage device according to an embodiment of the present disclosure is schematically shown.

[0111] like Figure 8 As shown, the data storage device 800 may include: a second acquisition module 810 , a parsing module 820 , a first storage module 830 , a third determination module 840 and a second storage module 850 .

[0112] The second acquisition module 810 may be used to acquire the historical message data to be stored. According to the embodiment of the present disclosure, the second acquisition module 810 may, for example, execute the above reference Figure 6 The operation S610 described above will not be described in detail here.

[0113] The parsing module 820 may be used to parse each historical message data in the historical message data to be stored, and obtain the attribute information of each historical message data. According to the embodiment of the present disclosure, the parsing module 820 may, for example, execute the above reference Figure 6 The operation S620 described above will not be described in detail here.

[0114] The first storage module 830 can be used to store multiple historical message data to be stored in at least one file in the distributed file system, and record the file path of the file where each historical message data is located. According to an embodiment of the present disclosure, the first storage module 830 can, for example, execute the above reference Figure 6 The operation S630 described above will not be described in detail here.

[0115] The third determination module 840 may be used to determine, for each historical message data, the attribute information of the historical message data and the file path of the file where the historical message data is located as index information. According to an embodiment of the present disclosure, the third determination module 840 may, for example, execute the above reference Figure 6 The operation S640 described above will not be described in detail here.

[0116] The second storage module 850 may be used to associate and store the index information to the bitmap database. According to an embodiment of the present disclosure, the second storage module 850 may, for example, execute the above reference Figure 6 The operation S650 described above will not be described in detail here.

[0117] According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits, or at least part of the functions of any one of them can be implemented in one module. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be split into multiple modules for implementation. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be at least partially implemented as hardware circuits, such as field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), systems on chips, systems on substrates, systems on packages, application specific integrated circuits (ASICs), or can be implemented by hardware or firmware in any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, according to the embodiments of the present invention, one or more of the modules, submodules, units, and subunits can be at least partially implemented as computer program modules, and when the computer program modules are run, the corresponding functions can be performed.

[0118] Fig. 9 A block diagram of a computer system suitable for data query and data storage according to an embodiment of the present disclosure is schematically shown. Fig. 9 The computer system shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0119] like Fig. 9 As shown, the computer system 900 according to the embodiment of the present disclosure includes a processor 901, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 902 or the program loaded from the storage part 908 to the random access memory (RAM) 903. The processor 901 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 901 may also include an onboard memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for performing different actions of the method flow according to the embodiment of the present disclosure.

[0120] In RAM 903, various programs and data required for the operation of system 900 are stored. Processor 901, ROM 902 and RAM 903 are connected to each other through bus 904. Processor 901 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 902 and / or RAM 903. It should be noted that the program can also be stored in one or more memories other than ROM 902 and RAM 903. Processor 901 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.

[0121] According to an embodiment of the present disclosure, the system 900 may further include an input / output (I / O) interface 905, which is also connected to the bus 904. The system 900 may further include one or more of the following components connected to the I / O interface 905: an input section 906 including a keyboard, a mouse, etc.; an output section 907 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, a modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the I / O interface 905 as needed. A removable medium 911, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 910 as needed, so that a computer program read therefrom is installed into the storage section 908 as needed.

[0122] According to an embodiment of the present disclosure, the method flow according to an embodiment of the present disclosure can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 909, and / or installed from the removable medium 911. When the computer program is executed by the processor 901, the above-mentioned functions defined in the system of the embodiment of the present disclosure are executed. According to an embodiment of the present disclosure, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.

[0123] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0124] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, such as, but not limited to, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0125] For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the ROM 902 and / or the RAM 903 described above and / or one or more memories other than the ROM 902 and the RAM 903 .

[0126] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0127] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. The scope of the present disclosure is defined by the attached claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A data query method, comprising: receiving a query request, wherein the query request at least includes attribute information of target message data; Based on the query request, determine target index data from a plurality of index data, wherein the target index data includes a target file path associated with the attribute information of the target message data, and each index data in the plurality of index data includes the attribute information of the historical message data and a file path of a file where the historical message data is located; Based on the target index data, determining a target file from at least one file, wherein a file path of the target file is the target file path, and the at least one file is used to store the historical message data; and Acquire the target message data from the target file; Among them, the multiple index data are stored in multiple first databases, and the database identifier and index data identifier of each of the first databases are associated and stored in the second database, and the index data identifier represents the index data stored in the first database; the multiple index data are stored in the multiple first databases according to the timestamp of the historical message data; the index data identifier includes the timestamp range of the index data stored in the first database.

2. The method according to claim 1, wherein: The determining target index data from a plurality of index data based on the query request includes: Based on the query request, determining the index data identifier indicated by the query request from the second database; Based on the index data identifier indicated by the query request, determining at least one database identifier associated with the indicated index data identifier from the second database; Based on the at least one database identifier, determining at least one first database corresponding to the at least one database identifier; and Index data whose attribute information matches the attribute information of the target message data is determined from the index data stored in the at least one first database as the target index data.

3. The method according to claim 2, wherein: The query request also includes a target time range, and the timestamp of the target message data is within the target time range; Wherein, based on the query request, determining from the second database the index data identifier indicated by the query request includes: Based on the target time range in the query request, the index data identifier indicated by the query request is determined from the second database, and the timestamp range of the index data identifier indicated by the query request includes the target time range.

4. The method according to claim 1, wherein: The target file includes a plurality of historical message data; Wherein, obtaining the target message data from the target file includes: Based on the attribute information of the target message data, at least one historical message data is determined from the plurality of historical message data as the target message data, wherein the attribute information of the at least one historical message data matches the attribute information of the target message data.

5. The method according to claim 1, wherein: The at least one file is a file in a distributed file system; the at least one file corresponds one-to-one to at least one preset time range, and for each file in the at least one file, the message generation time of each historical message data stored in the file is within the preset time range corresponding to the file.

6. The method according to claim 1, wherein: For each file in the at least one file, the plurality of historical message data stored in the file are compressed into a plurality of sub-files; For each sub-file, multiple historical message data in the sub-file are compressed in sequence; The multiple historical message data are compressed in sequence, including: compressing at least one received historical message data to obtain a preliminary compressed subfile, and compressing at least one newly received historical message data into the preliminary compressed subfile; The file path also includes the file name of the sub-file.

7. The method according to any one of claims 1 to 6, wherein: The attribute information includes at least one of the following: Source IP address, destination IP address, source port, destination port, data transmission protocol.

8. The method according to claim 2 or 3, wherein: The first database includes a bitmap database, and the second database includes a bitmap database.

9. A data storage method, comprising: Obtain historical message data to be stored; Parsing each historical message data in the historical message data to be stored to obtain attribute information of each historical message data; Storing the historical message data to be stored in at least one file in a distributed file system, and recording the file path of each file where the historical message data is located; For each historical message data, determining the attribute information of the historical message data and the file path of the file where the historical message data is located as index data; as well as The index data are stored in association with each other, wherein the multiple index data are stored in multiple first databases, and the database identifier and the index data identifier of each first database are stored in association with each other in the second database, and the index data identifier represents the index data stored in the first database; A plurality of index data are stored in the plurality of first databases according to the timestamps of the historical message data; the index data identifier includes a timestamp range of the index data stored in the first database.

10. The method according to claim 9, wherein: The at least one file corresponds to at least one preset time range one by one; the storing the historical message data to be stored in at least one file in the distributed file system includes, for each of the historical message data: Determine the message generation time of the historical message data; as well as Based on the message generation time and the at least one preset time range, the historical message data is stored in one of the at least one file, wherein the message generation time is within the preset time range corresponding to the stored file.

11. The method according to claim 9, wherein: For each file in the at least one file, the multiple historical message data stored in the file are compressed into multiple sub-files; for each sub-file: Compressing at least one received historical message data to obtain a preliminary compressed sub-file; as well as The at least one newly received historical message data is compressed into the preliminary compressed sub-file.

12. A data query device, comprising: A receiving module, used for receiving a query request, wherein the query request at least includes attribute information of target message data; A first determination module is used to determine target index data from multiple index data based on the query request, wherein the target index data includes a target file path associated with the attribute information of the target message data, and each index data in the multiple index data includes the attribute information of the historical message data and the file path of the file where the historical message data is located; A second determination module, configured to determine a target file from at least one file based on the target index data, wherein a file path of the target file is the target file path, and the at least one file is used to store the historical message data; and A first acquisition module, used for acquiring the target message data from the target file; Among them, the multiple index data are stored in multiple first databases, and the database identifier and index data identifier of each of the first databases are associated and stored in the second database, and the index data identifier represents the index data stored in the first database; the multiple index data are stored in the multiple first databases according to the timestamp of the historical message data; the index data identifier includes the timestamp range of the index data stored in the first database.

13. A data storage device, comprising: A second acquisition module is used to acquire historical message data to be stored; A parsing module, used for parsing each historical message data in the historical message data to be stored to obtain attribute information of each historical message data; A first storage module, used for storing the historical message data to be stored in at least one file in a distributed file system, and recording the file path of each file where the historical message data is located; A third determination module is used to determine, for each piece of historical message data, attribute information of the historical message data and a file path of a file where the historical message data is located as index data; as well as A second storage module stores the index data in association with each other, wherein the multiple index data are stored in multiple first databases, and the database identifier and index data identifier of each first database are stored in association with each other in the second database, and the index data identifier represents the index data stored in the first database; A plurality of index data are stored in the plurality of first databases according to the timestamps of the historical message data; the index data identifier includes a timestamp range of the index data stored in the first database.

14. A computing device comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors execute the method according to any one of claims 1 to 11.

15. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the method according to any one of claims 1 to 11.

16. A computer program product, comprising computer executable instructions, wherein the instructions are used to implement the method according to any one of claims 1 to 11 when executed.

Images