Data acquisition method and system, computer program product and electronic equipment

Abstract

The invention discloses a data acquisition method and system, a computer program product and electronic equipment. The method comprises the steps that a data collection request initiated by a target object is responded, the data collection request at least carries a data identifier of target data to be collected and a data source compressed file, and the data source compressed file at least comprises a compressed structured data file block, an unstructured data file block and a metadata block; decompressing a data source compressed file in the target data acquisition request to obtain a plurality of file blocks and metadata blocks; target data is determined to be read from the multiple file blocks according to the matching result of the data identification in the target data collection request and the metadata storing all the file blocks in the metadata block, and the data type of the target data comprises structured data and / or unstructured data. The technical problem that correlation collection cannot be performed on structured and non-structured mixed data in real time in the prior art is solved.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and more specifically, to a data collection method and system, a computer program product, and an electronic device. Background Art

[0002] With the continuous development of data science and information technology, the role of structured data and unstructured data in data management and analysis is becoming increasingly important. Among them, structured data is easy to manage and analyze, but may not cover all information, while unstructured data contains more details and background information, but is more complicated to process and analyze. Therefore, combining structured and unstructured data for collection can provide a more comprehensive description of the characteristics and specific circumstances of things for subsequent data value mining and analysis, and improve the accuracy and efficiency of analysis and decision-making.

[0003] However, when dealing with large-scale structured and unstructured mixed data, traditional solutions such as multiple mining algorithms, feature matrix, embedded development and high-precision instrument collection, pre-processing and mixed data enhancement processing by training large models, dynamic hierarchical distributed data collection algorithms and domain topology order, based on GPT (Generative Pre-trained Transformer) or other external systems, etc., can solve the problem of mixed data collection to a certain extent, but often expose their own significant shortcomings in practical applications. For example: the implementation process of using mining algorithms or feature matrices is relatively complicated, and often requires writing complex logic to process data, which increases the overall complexity and the possibility of error, and is accompanied by performance problems; embedded development and high-precision instrument collection only focus on high-precision collection of a single data type, and fail to combine the information advantages of structured and unstructured data; training large models for processing scenarios with high real-time requirements, the model update cycle is long and cannot meet immediate needs. The method of using algorithms and domain topology order is poor in flexibility and business adaptability; the method based on GPT or other external systems has problems in security, real-time, and deployment and application complexity.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present application provide a data collection method and system, a computer program product, and an electronic device to at least solve the technical problem that the related technology cannot collect structured and unstructured mixed data in real time.

[0006] According to one aspect of an embodiment of the present application, a data collection method is provided, comprising: responding to a data collection request initiated by a target object, wherein the data collection request carries at least a data identifier of target data to be collected and a data source compressed file, wherein the data source compressed file comprises at least: a plurality of compressed file blocks and a metadata block for storing metadata of each file block, wherein the plurality of file blocks comprises a structured data file block for storing at least one structured data and an unstructured data file block for storing at least one unstructured data; decompressing the data source compressed file in the target data collection request to obtain a plurality of file blocks and a metadata block; and determining to read the target data from the plurality of file blocks based on a matching result between the data identifier in the target data collection request and the metadata storing each file block in the metadata block, wherein the data type of the target data comprises: structured data and / or unstructured data.

[0007] Optionally, the data source compressed file also includes: a file header for describing basic information of the data source compressed file, an index block for describing the offset of each data block, and a file tail for describing a pointer to the index block, wherein the file header includes at least: a first field for storing the file format of the data source compressed file, a second field for storing the file compression algorithm used by the data source compressed file, a third field for storing the number of data blocks in the data source compressed file, and a fourth field for storing the offset of the file tail of the data source compressed file; the structured data file block includes at least: a fifth field for storing the structured data file block's own identifier, a sixth field for storing each structured data, and a seventh field for verifying the integrity of all structured data in the structured data file block, wherein the sixth field includes at least a target sub-word for storing the offset of each unstructured data corresponding to the structured data. segment; the unstructured data file block at least includes: an eighth field for storing the unstructured data file block's own identification, a ninth field for storing the data name of each unstructured data, a tenth field for storing each unstructured data itself, and an eleventh field for verifying the integrity of each unstructured data, wherein the ninth field is associated with the target subfield in the sixth field; the metadata block at least includes: a twelfth field for storing the metadata block's own identification, a thirteenth field for storing the length of each metadata, a fourteenth field for storing the data content of the metadata, and a fifteenth field for verifying the integrity of the metadata, wherein the data content of the metadata includes at least one of the following: data source, creation event, owner information, and data organization structure; the file tail at least includes: a sixteenth field for storing the offset of the index block, and a seventeenth field for verifying the integrity of the data source compressed file.

[0008] Optionally, the data source compressed file in the target data acquisition request is decompressed to obtain multiple file blocks and metadata blocks, including: reading the first field in the file header in the data source compressed file, and verifying the file header based on the first field; when the file header passes the verification, reading the second field in the file header, and determining the corresponding file decompression algorithm based on the second field, and decompressing the multiple file blocks and metadata blocks compressed by the data source compressed file according to the file decompression algorithm.

[0009] Optionally, before determining to read the target data from multiple file blocks, the method also includes: reading the fourth field in the file header of the data source compressed file, and determining the file tail of the data source compressed file based on the fourth field; determining the index block of the data source compressed file based on the sixteenth field in the file tail; locating the metadata block and each file block of the data source compressed file based on the offset of the metadata block recorded in the index block and the offset of each file block.

[0010] Optionally, based on the matching result between the data identifier in the target data acquisition request and the metadata of each file block stored in the metadata block, the target data to be read is determined from multiple file blocks, including: matching the metadata of each file block stored in the metadata block based on the data identifier in the target data acquisition request, and determining the target metadata that matches successfully; obtaining the target metadata from the structured data file block to determine the corresponding target structured data, and reading the target sixth field of the target structured data, and judging whether the target subfield of the target sixth field is empty; when the target subfield of the target sixth field is not empty, reading the corresponding target unstructured data from the unstructured data file block based on the target subfield of the target sixth field, and taking the target unstructured data and the target structured data as the target data to be collected.

[0011] Optionally, the method further includes: when the target subfield of the target sixth field is empty, directly using the target structured data as the target data to be collected.

[0012] Optionally, after determining to read the target data from the multiple file blocks, the method further includes: encrypting the structured data and the unstructured data using a preset encryption algorithm, and storing the encryption result obtained by the encryption.

[0013] According to another aspect of an embodiment of the present application, a data acquisition system is also provided, including: an acquisition module, used to respond to a data acquisition request initiated by a target object, wherein the data acquisition request carries at least a data identifier of the target data to be collected and a data source compressed file, wherein the data source compressed file includes at least: a plurality of compressed file blocks and a metadata block for storing metadata of each file block, wherein the plurality of file blocks include a structured data file block for storing at least one structured data and an unstructured data file block for storing at least one unstructured data; a decompression module, used to decompress the data source compressed file in the target data acquisition request to obtain a plurality of file blocks and a metadata block; and a collection module, used to determine to read the target data from the plurality of file blocks based on a matching result between the data identifier in the target data acquisition request and the metadata storing each file block in the metadata block, wherein the data type of the target data includes: structured data and / or unstructured data.

[0014] According to another aspect of an embodiment of the present application, a computer program product is further provided, the computer program product comprising: a computer program, wherein the above-mentioned data acquisition method is implemented when the computer program is executed by a processor.

[0015] According to another aspect of an embodiment of the present application, an electronic device is further provided. The electronic device includes: a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to execute the above-mentioned data acquisition method through the computer program.

[0016] In the embodiment of the present application, the data acquisition system responds to the data acquisition request of the target object, and decompresses the data source compressed file in the request to obtain multiple file blocks and metadata blocks in the data source compressed file; by matching the data identifier of the data to be collected in the request with the information in the metadata block, the target data can be accurately located and read regardless of its data type. This mechanism overcomes the limitations of traditional data acquisition methods in processing real-time, large-scale mixed data, such as high processing complexity, poor real-time performance, high storage costs, and insufficient data security, and achieves the technical effect of efficiently associating and collecting structured and unstructured mixed data in real-time scenarios, achieving the purpose of simplifying the data acquisition process, improving data processing performance, and reducing storage costs, thereby solving the technical problem that related technologies cannot perform associated collection of structured and unstructured mixed data in real time. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0018] Figure 1 is a flow chart of an optional data collection method according to an embodiment of the present application;

[0019] Figure 2 is a schematic diagram of an optional file header format according to an embodiment of the present application;

[0020] Figure 3 is a schematic diagram of a format of an optional structured data file block according to an embodiment of the present application;

[0021] Figure 4 is a schematic diagram of a format of an optional unstructured data file block according to an embodiment of the present application;

[0022] Figure 5 is a schematic diagram of a format of an optional metadata block according to an embodiment of the present application;

[0023] Figure 6 is a schematic diagram of a format of an optional data source compression file according to an embodiment of the present application;

[0024] Figure 7 is a schematic structural diagram of an optional data acquisition system according to an embodiment of the present application;

[0025] Figure 8 It is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0027] It should be noted that the terms "first", "second", etc. in the specification, claims and drawings of the present application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] In order to better understand the embodiments of the present application, some nouns or terms that appear in the description of the embodiments of the present application are first translated and explained as follows:

[0029] Structured data: refers to data that can be represented and stored using relational data blocks and is in two-dimensional form. The general characteristics are: data is in rows, a row of data represents the information of an entity, and the attributes of each row of data are the same.

[0030] Unstructured data: refers to information that does not have a predefined data model or is not organized in a predefined way. That is, data that does not follow a specific data model or format, which can include text, images, audio, video, social media posts, emails, web content, log files, etc. This data is not convenient to store and manage using traditional database tables and fields. Unlike structured data (such as tabular data in a relational database), the format and content of unstructured data are more flexible and diverse.

[0031] Metadata: Data about data or information about data or information. It can be understood as the smallest unit of data, used to represent data to describe its elements or attributes (name, size, data type, etc.), or its structure (length, field, data column), or its related data (where it is located, how to contact, owner).

[0032] Example 1

[0033] According to an embodiment of the present application, a data collection method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0034] Figure 1 is a flow chart of a data collection method provided according to an embodiment of the present application, such as Figure 1 As shown, the method includes the following steps S102-S106, wherein:

[0035] Step S102, responding to a data collection request initiated by a target object, wherein the data collection request carries at least a data identifier of the target data to be collected and a data source compressed file.

[0036] In the technical solution provided in the above step S102, when the target object needs to collect target data from the specified data source, the data collection system can first obtain a data collection request initiated by the target object through the interactive interface, and the data collection request carries at least a data identifier of the target data to be collected (used to determine which data is the target of the current request, so the data identifier can be the data source, data creation time, owner information, etc.), and a data source compressed file (i.e., a file composed of a data set in the specified data source), wherein the data source refers to the location where the data is generated or stored, including but not limited to various databases, file systems, network servers, Internet of Things devices, social media platforms, logging systems and other data generation points or transfer stations, and the data source compressed file refers to the compression processing of the data set in the process of file transmission in order to improve data security and storage efficiency, such as a compressed file generated by the original binary file after being processed by a file compression algorithm, which can save the same data information in a smaller storage space.

[0037] In addition, the data source compressed file includes at least multiple compressed file blocks and metadata blocks for storing feature information of the data to be collected, and the multiple file blocks include structured data file blocks for storing at least one structured data and unstructured data file blocks for storing at least one unstructured data, wherein the structured data file blocks store data that can be easily divided and managed, such as tabular data, for easy quick query and analysis; the unstructured data file blocks store data that does not follow a fixed format, such as text, images, audio, video, etc., which provide rich content information but are more complicated to process.

[0038] Step S104, decompressing the data source compressed file in the target data collection request to obtain a plurality of file blocks and metadata blocks.

[0039] In the technical solution provided in the above step S104, in order to obtain the target data to be collected from the data source compressed file, the multiple compressed data blocks in the data source compressed file can be decompressed first, so as to facilitate the subsequent reading of the stored data information from the structured data file blocks, unstructured data file blocks and metadata blocks.

[0040] Step S106, determining to read target data from multiple file blocks based on the matching result of the data identifier in the target data acquisition request and the metadata storing each file block in the metadata block.

[0041] In the technical solution provided in the above step S106, since the data identifier can reflect the key information of the target data to be collected, and the metadata block stores detailed metadata about the structured data file block and the unstructured data file block, the data collection system can match the data identifier in the data collection request with the metadata stored in the metadata block. Based on the matching results, the system can determine which structured data file blocks and unstructured data file blocks are related to the requested target data. Once the match is successful, the system can directly locate the specific structured data file blocks and unstructured data file blocks, thereby reading the target data, and the read target data can be structured data and / or unstructured data of each field in the structured data.

[0042] Based on the scheme defined in the above steps S102 to S106, it can be known that in the embodiment of the present application, the data acquisition system responds to the data acquisition request of the target object and decompresses the data source compressed file in the request to obtain multiple file blocks and metadata blocks in the data source compressed file; by matching the data identifier of the data to be collected in the request with the information in the metadata block, the target data can be accurately located and read regardless of its data type. This mechanism overcomes the limitations of traditional data acquisition methods in processing real-time, large-scale mixed data, such as high processing complexity, poor real-time performance, high storage cost and insufficient data security, and achieves the technical effect of efficiently associating and collecting structured and unstructured mixed data in real-time scenarios, achieving the purpose of simplifying the data acquisition process, improving data processing performance and reducing storage costs.

[0043] The following describes each step of the data collection method in conjunction with a specific implementation process.

[0044] As an optional implementation, the data source compressed file may be a compressed file of a binary file in a specific format, and the specific structure of the original uncompressed binary file in a specific format includes: a file header (File Header) for describing the basic information of the data source compressed file, a structured data file block (Data Block), an unstructured data file block (File Block), a metadata block (Meta Block), an index block (Index Block) for describing the offset of each data block, and a file trailer (File Trailer) for describing the pointer of the index block. The above structures together constitute a hybrid data storage unit for storing structured data and unstructured data. For each component of a binary file in a specific format, its structure is as follows:

[0045] (1) File Header: As the beginning of a binary file, it is mainly used to describe the basic information of the file and has a fixed size to facilitate identification and processing of file contents. The structure of the file header is as follows: Figure 2 As shown, it contains the following key elements:

[0046] Magic Number (first field): A file format (such as binary format) used to store compressed files of a data source, which enables the reading program to quickly determine the type and version of the file. Therefore, the existence of the Magic Number helps prevent the file from being misread or misoperated.

[0047] Compression (second field): specifies the file compression algorithm used to store the compressed files of the data source, such as the Huffman compression algorithm (that is, using the number of times characters appear in the file to build a Huffman tree to generate Huffman encoding), the LZ777 compression algorithm (that is, replacing repeated bytes in the original file with length and distance pairs to achieve the purpose of compression), the GZIP compression algorithm (a combination of the LZ777 compression algorithm and the Huffman compression algorithm), etc.

[0048] Number of Blocks (third field): used to describe the number of data blocks contained in the data source compressed file, that is, the number of structured data blocks, unstructured data blocks, and metadata blocks. It is usually a constant value.

[0049] File Trailer Offset (the fourth field): used to store the offset of the file tail of the data source compressed file, used for file integrity verification and direct positioning of index blocks. It is usually a constant value.

[0050] (2) Structured data file block (Data Block): stores structured data. Structured data generally contains multiple lines of data, each line of data represents a piece of information. These data are divided into multiple fields by specific delimiters. Among them, the structure of the structured data file block is as follows: Figure 3 As shown, it contains the following key elements:

[0051] Data Block Magic (fifth field): used to store the identifier of the structured data file block itself, which is usually a constant value.

[0052] Record (sixth field): used to store each structured data, and each Record represents a row in the structured data, and a row consists of multiple subfields, each subfield is separated by a specific separator. Since the structured data includes unstructured data corresponding to different fields, the sixth field includes at least a target subfield for storing the offset of each unstructured data corresponding to the structured data.

[0053] CheckSum (seventh field): This is the checksum field, which is used to verify the integrity of all structured data in the structured data file block to ensure that the data has not been tampered with or damaged during storage or transmission. This verification mechanism is crucial to the reliability and accuracy of the data.

[0054] (3) Unstructured data file block: used to store data that does not conform to a fixed format, such as text files, images, audio, video, etc. The processing of such data usually requires special parsing operations. Figure 4 As shown, it contains the following key elements:

[0055] File Block Magic (eighth field): used to store the identifier of the unstructured data file block itself, which is usually a constant value.

[0056] File Name (9th field): The data name used to store each unstructured data. This data name is usually associated with a specific subfield of a row of structured data in the structured data file block (that is, the 9th field is associated with the target subfield in the 6th field). This association allows users to quickly find the corresponding unstructured data through an identifier in the structured data.

[0057] File Data (the tenth field): stores each unstructured data itself (ie, stores the actual unstructured data content). The data can be any type of binary or text data, depending on the purpose and type of the file.

[0058] CheckSum (the eleventh field): This is the checksum field, which is used to verify the integrity of all unstructured data in the unstructured data file block to ensure that the data has not been tampered with or damaged during storage or transmission. This verification mechanism is crucial to the reliability and accuracy of the data.

[0059] (4) Metadata Block: It is used to store information metadata of each structured data in the structured data block and each unstructured data in the unstructured data block. It not only provides the background and context of the data, but also establishes associations between the data. Therefore, the information metadata includes but is not limited to: data source, creation time, data format, organizational structure of structured data, association relationship between unstructured data and structured data, access rights and other detailed information. The structure of the metadata block is as follows: Figure 5 As shown, it contains the following key elements:

[0060] Meta Block Magic (the twelfth field): stores the metadata block's own identifier, which is usually a constant value.

[0061] Meta Length (the 13th field): stores the length of each metadata. This field ensures that the parser can quickly skip the metadata block and directly access the subsequent file blocks.

[0062] Meta Data (the fourteenth field): stores the data content of metadata, which includes but is not limited to: data source, creation time, owner information, organizational structure of structured data, etc.

[0063] CheckSum (the fifteenth field): This is the checksum field, which is used to verify the integrity of all metadata in the metadata block to ensure that the data has not been tampered with or damaged during storage or transmission. This verification mechanism is crucial to the reliability and accuracy of the data.

[0064] (5) Index Block: describes the offset of each data block (structured data file block, unstructured data file block, metadata block), and is used to locate the position of each data block in the data source compressed file.

[0065] (6) File Trailer: A pointer describing the index block, which is used to locate the index block in the compressed file of the data source and verify the integrity of the entire current data. Therefore, the file trailer contains the following key elements:

[0066] Index Block Offset (the sixteenth field): the offset of the index block.

[0067] CheckSum (the seventeenth field): the checksum field, used to verify the integrity of the compressed file of the data source.

[0068] According to the various file components described above, the specific structure of the data source compressed file in a specific format provided in the embodiment of the present application can be referred to Figure 6An optional structure shown in FIG. Figure 2-6 The field value parameters listed are only used as examples.

[0069] After the data collection system obtains the above data source compressed file, it can read multiple file blocks and metadata blocks in the data source compressed file according to the following method, including:

[0070] Step 1: Read the first field in the file header of the data source compressed file, and verify the file header based on the first field;

[0071] Step 2: When the file header passes the verification, read the second field in the file header, and determine the corresponding file decompression algorithm based on the second field, and decompress the multiple file blocks and metadata blocks compressed by the data source compressed file according to the file decompression algorithm.

[0072] That is to say, first read the Magic Number field in the file header of the custom format in the data source compressed file, and verify the Magic Number field. If the file format is correct, read the multiple file blocks and metadata blocks compressed by the data source compressed file; then, determine the file decompression algorithm according to the Compression field in the file header, and decompress the multiple file blocks and metadata blocks compressed by the data source compressed file according to the corresponding file decompression algorithm, so as to obtain multiple uncompressed file blocks and metadata blocks. For example, if the file compression algorithm used by the data source compressed file is the GZIP compression algorithm, then the corresponding decompression algorithm is the GZIP decompression algorithm.

[0073] Furthermore, the data acquisition system can read the fourth field (i.e., the File Trailer Offset field) in the file header of the data source compressed file, and determine the file tail of the data source compressed file based on the fourth field; determine the index block of the data source compressed file based on the sixteenth field (i.e., the Index Block Offset field) in the file tail; and locate the metadata block and each file block of the data source compressed file based on the offset of the metadata block and the offset of each file block recorded in the index block.

[0074] It should be noted that before locating the metadata block and each file block of the data source compressed file, the data acquisition system can also verify the integrity of the file based on the seventeenth field (i.e., the CheckSum field) in the file tail. If the verification is correct, the next step is to locate the metadata block and each file block of the data source compressed file. Otherwise, the positioning is stopped directly, and feedback is given that the file has missing information.

[0075] Furthermore, after locating the metadata block and each file block in the compressed file of the data source, the data collection system may collect the target data according to the following steps, including:

[0076] First, the data identifier in the target data collection request is matched with the metadata of each file block stored in the metadata block, and the successfully matched target metadata is determined, so as to know the data source, creation time, owner information, organizational structure and field information of the target data to be collected.

[0077] Next, the structured data is read line by line, and the structured field information in the target metadata is mapped one by one with each field of each line of structured data to determine the successfully matched target structured data (that is, the target metadata is obtained from the structured data file block to determine the corresponding target structured data). Since the target subfield in the sixth field of the target structured data represents the data offset of its unstructured data, the data acquisition system can read the target sixth field of the target structured data and determine whether the target subfield of the target sixth field is empty;

[0078] When the target subfield of the sixth target field is not empty, the data collection system can first locate the position of the unstructured data block according to the index block, and then read the corresponding target unstructured data from the unstructured data file block according to the target subfield of the sixth target field. In this way, both structured target data and unstructured target data can be read, realizing the collection of structured and unstructured mixed data in real-time scenarios.

[0079] When the target subfield of the sixth target field is empty, it means that the target structured data has no associated unstructured data. In this case, the target structured data is directly used as the target data to be collected.

[0080] It should be noted that, in the above process of reading corresponding data from the file block, it is possible to first determine whether the type of the file block is correct through the Data Block Magic field or the File Block Magic field of the file block, and only perform subsequent data reading operations if it is correct.

[0081] Finally, after determining to read the target data from the multiple file blocks, the target data can be parsed and stored. Alternatively, the structured data and the unstructured data are encrypted using a preset encryption algorithm, and the encryption result obtained is stored, wherein the encryption algorithm can be an elliptic curve encryption algorithm, an encryption method based on the Java asymmetric encryption KeyPairGenerator class, etc. The storage location can be a storage medium or a preset database.

[0082] The well-structured data format provided by the embodiment of the present application can effectively integrate and manage structured data and unstructured data, which not only realizes the synchronous analysis of data, but also ensures the efficient use and management of data. The data source compressed file based on this data format allows users to obtain the associated collection of structured and unstructured mixed data in real time, greatly improving the efficiency of mixed data collection in large-scale scenarios.

[0083] Example 2

[0084] According to an embodiment of the present application, a data acquisition system for implementing the data acquisition method in embodiment 1 is also provided. Figure 7 As shown, the data acquisition system at least includes: an acquisition module 72, a decompression module 74 and a collection module 76, wherein:

[0085] The acquisition module 72 is used to respond to the data collection request initiated by the target object, wherein the data collection request carries at least the data identification of the target data to be collected and the data source compressed file, wherein the data source compressed file at least includes: a plurality of compressed file blocks and a metadata block for storing metadata of each file block, wherein the plurality of file blocks include a structured data file block for storing at least one structured data and an unstructured data file block for storing at least one unstructured data;

[0086] A decompression module 74, used to decompress the data source compressed file in the target data collection request to obtain multiple file blocks and metadata blocks;

[0087] The acquisition module 76 is used to determine the target data to be read from multiple file blocks based on the matching result between the data identifier in the target data acquisition request and the metadata storing each file block in the metadata block, wherein the data type of the target data includes: structured data and / or unstructured data.

[0088] It should be noted that each module in the data acquisition system in the embodiment of the present application corresponds one-to-one to each implementation step of the data acquisition method in Example 1. Since a detailed description has been given in Example 1, some details not reflected in this embodiment can be referred to Example 1 and will not be repeated here.

[0089] Example 3

[0090] According to an embodiment of the present application, a computer program product is also provided. The computer program product includes a computer program, wherein when the computer program is executed by a processor, the data collection method in Embodiment 1 is implemented.

[0091] According to an embodiment of the present application, a non-volatile storage medium is also provided, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the data acquisition method in Example 1 by running the computer program.

[0092] According to an embodiment of the present application, a processor is further provided, which is used to run a computer program, wherein the data collection method in Example 1 is executed when the computer program is running.

[0093] According to an embodiment of the present application, an electronic device is further provided, which includes: a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to execute the data acquisition method in Example 1 through the computer program.

[0094] Specifically, the computer program executes the following steps when it is running: responding to a data collection request initiated by a target object, wherein the data collection request carries at least a data identifier of the target data to be collected and a data source compressed file, wherein the data source compressed file includes at least: a plurality of compressed file blocks and a metadata block for storing metadata of each file block, wherein the plurality of file blocks include a structured data file block for storing at least one structured data and an unstructured data file block for storing at least one unstructured data; decompressing the data source compressed file in the target data collection request to obtain a plurality of file blocks and a metadata block; determining to read the target data from the plurality of file blocks based on a matching result between the data identifier in the target data collection request and the metadata storing each file block in the metadata block, wherein the data type of the target data includes: structured data and / or unstructured data.

[0095] As an optional implementation, the electronic device may be in the form of a mobile terminal, a computer terminal or a similar computing device. Figure 8 FIG. 1 shows a hardware structure block diagram of an electronic device for implementing a data acquisition method. Figure 8 As shown, the electronic device 80 may include one or more (802a, 802b, ..., 802n are used to illustrate) processors 802 (the processor 802 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 804 for storing data, and a transmission device 806 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 8 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 8 More or fewer components as shown, or with Figure 8 Different configurations shown.

[0096] It should be noted that the one or more processors 802 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the electronic device 80. As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0097] The memory 804 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the data acquisition method in the embodiment of the present application. The processor 802 executes various functional applications and data processing by running the software programs and modules stored in the memory 804, that is, realizing the vulnerability detection method of the above-mentioned application. The memory 804 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 804 may further include a memory remotely arranged relative to the processor 802, and these remote memories may be connected to the electronic device 80 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0098] The transmission device 806 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the electronic device 80. In one example, the transmission device 806 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 806 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0099] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the electronic device 80 .

[0100] The serial numbers of the above embodiments are only for description and do not represent the advantages or disadvantages of the embodiments.

[0101] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0102] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0103] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0104] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0105] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the relevant technology or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc. Various media that can store program codes.

[0106] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A data collection method, characterized in that: include: Responding to a data collection request initiated by a target object, wherein the data collection request carries at least a data identifier of the target data to be collected and a data source compressed file, wherein the data source compressed file includes at least: a plurality of compressed file blocks and a metadata block for storing metadata of each of the file blocks, wherein the plurality of file blocks include a structured data file block for storing at least one structured data and an unstructured data file block for storing at least one unstructured data; Decompressing the data source compressed file in the target data acquisition request to obtain the multiple file blocks and the metadata block; Based on the matching result between the data identifier in the target data acquisition request and the metadata storing each of the file blocks in the metadata block, the target data is determined to be read from the multiple file blocks, wherein the data type of the target data includes: structured data and / or unstructured data.

2. The method according to claim 1, characterized in that The data source compressed file also includes: a file header for describing basic information of the data source compressed file, an index block for describing the offset of each data block, and a file footer for describing a pointer to the index block, wherein: The file header at least includes: a first field for storing the file format of the data source compressed file, a second field for storing the file compression algorithm used by the data source compressed file, a third field for storing the number of data blocks in the data source compressed file, and a fourth field for storing the offset of the file end of the data source compressed file; The structured data file block at least includes: a fifth field for storing the structured data file block's own identification, a sixth field for storing each structured data, and a seventh field for verifying the integrity of all the structured data in the structured data file block, wherein the sixth field at least includes a target subfield for storing the offset of each unstructured data corresponding to the structured data; The unstructured data file block at least includes: an eighth field for storing the identification of the unstructured data file block itself, a ninth field for storing the data name of each unstructured data, a tenth field for storing each unstructured data itself, and an eleventh field for verifying the integrity of each unstructured data, wherein the ninth field is associated with the target subfield in the sixth field; The metadata block at least includes: a twelfth field for storing the metadata block's own identification, a thirteenth field for storing the length of each metadata, a fourteenth field for storing the data content of the metadata, and a fifteenth field for verifying the integrity of the metadata, wherein the data content of the metadata includes at least one of the following: data source, creation event, owner information, and data organization structure; The file tail includes at least: a sixteenth field for storing the offset of the index block, and a seventeenth field for verifying the integrity of the data source compressed file.

3. The method according to claim 2, characterized in that Decompressing the data source compressed file in the target data acquisition request to obtain the multiple file blocks and the metadata block includes: Reading a first field in a file header in the data source compressed file, and verifying the file header according to the first field; When the file header passes the verification, the second field in the file header is read, and the corresponding file decompression algorithm is determined based on the second field, and the multiple file blocks and metadata blocks compressed by the data source compressed file are decompressed according to the file decompression algorithm.

4. The method according to claim 2, characterized in that: Before determining to read the target data from the plurality of file blocks, the method further includes: Reading a fourth field in a file header in the data source compressed file, and determining a file tail of the data source compressed file according to the fourth field; Determine the index block of the data source compressed file according to the sixteenth field in the file tail; The metadata block and each file block of the data source compressed file are located according to the offset of the metadata block and the offset of each file block recorded in the index block.

5. The method according to claim 1, characterized in that Determining to read the target data from the plurality of file blocks according to a matching result between the data identifier in the target data acquisition request and the metadata storing each of the file blocks in the metadata block includes: Matching the metadata of each file block stored in the metadata block according to the data identifier in the target data collection request, and determining the successfully matched target metadata; Acquire the target metadata from the structured data file block to determine the corresponding target structured data, read the target sixth field of the target structured data, and determine whether the target subfield of the target sixth field is empty; When the target subfield of the target sixth field is not empty, the corresponding target unstructured data is read from the unstructured data file block according to the target subfield of the target sixth field, and the target unstructured data and the target structured data are used as the target data to be collected.

6. The method according to claim 5, characterized in that The method further comprises: When the target subfield of the sixth target field is empty, the target structured data is directly used as the target data to be collected.

7. The method according to claim 1, characterized in that After determining to read the target data from the plurality of file blocks, the method further includes: The structured data and the unstructured data are encrypted using a preset encryption algorithm, and the encryption results are stored.

8. A data acquisition system, characterized in that: include: An acquisition module is used to respond to a data collection request initiated by a target object, wherein the data collection request carries at least a data identifier of the target data to be collected and a data source compressed file, wherein the data source compressed file includes at least: a plurality of compressed file blocks and a metadata block for storing metadata of each of the file blocks, wherein the plurality of file blocks include a structured data file block for storing at least one structured data and an unstructured data file block for storing at least one unstructured data; A decompression module, used for decompressing the data source compressed file in the target data acquisition request to obtain the multiple file blocks and the metadata block; An acquisition module is used to determine to read the target data from the multiple file blocks based on a matching result between a data identifier in the target data acquisition request and the metadata of each file block stored in the metadata block, wherein the data type of the target data includes: structured data and / or unstructured data.

9. A computer program product, characterized in that include: A computer program, wherein when the computer program is executed by a processor, the data collection method according to any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: include: A memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the data collection method according to any one of claims 1 to 7 through the computer program.

Images