Storage and query method and device for unstructured data, medium and program product

Abstract

The invention provides an unstructured data storage and query method, electronic equipment, a readable storage medium and a computer program product, and the method comprises the steps: obtaining compressed data corresponding to unstructured data, carrying out the hierarchical storage of the compressed data, and storing a metadata set corresponding to the compressed data; receiving a user query request; the user query request comprises a query condition and user identity information; when the user identity information passes verification, detecting a target data set in the metadata set based on a query condition; decompressing and decrypting the target data set to obtain a query data set; recombining the query data set to obtain a query result; the query result is unstructured data. According to the method, the query capability and query efficiency of the unstructured data can be improved, and meanwhile, the security of data query can be effectively guaranteed.

Description

Technical Field

[0001] The present application relates to the technical field of data storage, and in particular to a storage query method for unstructured data, an electronic device, a readable storage medium, and a computer program product. Background Art

[0002] At present, the existing technology usually responds to query requests by directly traversing the stored data, but its query efficiency is relatively low when facing massive data. At the same time, this method cannot effectively meet the query requirements when facing encrypted and compressed data. In addition, although this method can achieve a certain query effect, its query security cannot be guaranteed. Summary of the invention

[0003] The purpose of this application is to provide a storage query method, electronic device, readable storage medium and computer program product for unstructured data, which can improve the query capability and query efficiency of unstructured data, and at the same time can effectively ensure the security of data query.

[0004] The first aspect of the present application provides a storage query method for unstructured data, comprising:

[0005] Obtaining compressed data corresponding to the unstructured data, and storing the compressed data in layers, while storing a metadata set corresponding to the compressed data;

[0006] Receiving a user query request; the user query request includes query conditions and user identity information;

[0007] When the user identity information is verified, detecting a target data set in the metadata set based on the query condition;

[0008] Decompressing and decrypting the target data set to obtain a query data set;

[0009] The query data set is reorganized to obtain a query result; the query result is unstructured data.

[0010] In the above implementation process, the method can obtain the compressed data corresponding to the unstructured data, and store the compressed data and the corresponding metadata set in layers, so that the data storage is more efficient, and then effectively supports the operation of fast retrieval and query. Then, based on the query conditions and user identity information, the pertinence and accuracy of the query are ensured. At the same time, the verification process of the user identity information can also provide security protection for subsequent data access control. Secondly, the method can ensure that only legitimate users can access the data, thereby effectively preventing illegal access and data leakage. After that, the method can also quickly locate the target data through the data index and description information contained in the metadata, thereby improving the query efficiency. In addition, the method can also ensure the readability and availability of the data through decompression and decryption processing. Among them, the data exists in an encrypted form during the transmission and storage process, which can effectively prevent the data from being stolen or tampered by unauthorized users. Finally, the method can also generate targeted, clear and accurate query results according to user needs, thereby ensuring the query effect of the data.

[0011] Furthermore, the obtaining of compressed data corresponding to the unstructured data, and storing the compressed data in layers, while storing a set of metadata corresponding to the compressed data, includes:

[0012] Obtain compressed data after layering, segmenting, checking for duplicates, encrypting and compressing unstructured data;

[0013] Based on the metadata information of the compressed data, the compressed data is stored in layers, and the storage location of the compressed data is recorded; wherein the metadata information includes encryption information, compression information and attribute information;

[0014] generating index information corresponding to the compressed data;

[0015] The compressed data, the metadata information, the storage location and the index information are integrated and stored to obtain a metadata set.

[0016] In the above implementation process, the method can combine metadata information to perform hierarchical storage and index generation for compressed data, thereby achieving efficient organization and encryption protection of data, while ensuring the rapid location and retrieval capabilities of data, and optimizing storage efficiency and query performance.

[0017] Further, the encryption information includes the encryption algorithm, encryption status and key information of the compressed data;

[0018] The compression information includes the compression algorithm, compression state, compression ratio and checksum of the compressed data;

[0019] The attribute information includes data sensitivity, access frequency, usage scenario and data type of the compressed data;

[0020] The storage location includes a storage hierarchy and a physical storage location of the compressed data;

[0021] The index information includes a unique hash value of the compressed data and an index identifier.

[0022] In the above implementation process, this method can achieve efficient compression storage and fast and accurate retrieval of data through detailed metadata information (including encryption information, compression information and attribute information, etc.) and precise storage location records, combined with a unique hash index, thereby improving the overall efficiency of data management.

[0023] Furthermore, storing the compressed data in layers based on the metadata information of the compressed data includes:

[0024] When the data sensitivity of the compressed data is highly sensitive, storing the compressed data in a high-sensitivity data layer located in a high-security storage medium; or

[0025] When the data sensitivity of the compressed data is medium sensitivity, storing the compressed data in a medium sensitivity data layer; or

[0026] When the data sensitivity of the compressed data is low sensitivity, storing the compressed data in a low sensitivity data layer located in a high-performance storage medium;

[0027] The high-security storage medium has higher security than the high-performance storage medium, and the high-performance storage medium has higher performance than the high-security storage medium.

[0028] In the above implementation process, the method can achieve an effective balance between data security and storage performance, thereby ensuring the safe storage of highly sensitive data and improving the access efficiency of less sensitive data.

[0029] Furthermore, the decompression and decryption of the target data set to obtain the query data set includes:

[0030] Acquire a metadata target set corresponding to the target data set;

[0031] Extracting the compression algorithm and encryption algorithm corresponding to the target data set from the metadata target set;

[0032] Decompressing the target data set based on a decompression algorithm corresponding to the compression algorithm to obtain a decompressed data set;

[0033] The decompressed data set is decrypted based on a decryption algorithm corresponding to the encryption algorithm to obtain a query data set.

[0034] In the above implementation process, the method can realize efficient decompression and decryption of the target data set by accurately extracting the compression and encryption algorithm information in the metadata, thereby ensuring the accurate restoration and secure access of the data, and helping to improve the response speed of data processing and query.

[0035] Furthermore, the method further comprises:

[0036] Calculating a first checksum of each decompressed data in the decompressed data set;

[0037] determining whether the first checksum is consistent with a second checksum stored in the metadata target set;

[0038] If so, the decryption algorithm corresponding to the encryption algorithm is executed to decrypt the decompressed data set to obtain the query data set.

[0039] In the above implementation process, the method can ensure the integrity and accuracy of the data through consistency verification of the checksum, thereby effectively preventing the data from being damaged or tampered with during the decompression process, enhancing the reliability of data processing, and providing a high-quality data foundation for subsequent decryption and query.

[0040] Furthermore, the method further comprises:

[0041] Get the user's query behavior record;

[0042] Based on the query behavior record, the metadata set is dynamically adjusted.

[0043] In the above implementation process, the method can dynamically optimize the metadata set by acquiring and analyzing user query behavior, thereby helping to improve the personalization and efficiency of data retrieval.

[0044] Furthermore, dynamically adjusting the metadata set based on the query behavior record includes:

[0045] When the query behavior record shows that the query frequency of the target data set is higher than a first preset frequency threshold, increasing the priority of the metadata corresponding to the target data set in the metadata set; or

[0046] When the query behavior record shows that the query frequency of the target data set is lower than a second preset frequency threshold, lowering the priority of the metadata corresponding to the target data set in the metadata set, or removing the metadata corresponding to the target data set from the metadata set; or

[0047] When the query behavior record shows that there is an abnormal query or a potential attack, the index complexity of the metadata set is increased.

[0048] In the above implementation process, the method can optimize data storage and retrieval strategies based on user query behavior, which is conducive to improving the access efficiency of high-frequency data, while reducing the storage cost of low-frequency data, and can also enhance the defense capability against abnormal queries, thereby improving the security, efficiency and flexibility of data management and access at the overall level.

[0049] A second aspect of the present application provides a storage query device for unstructured data, the storage query device for unstructured data comprising:

[0050] A storage unit, used to obtain compressed data corresponding to the unstructured data, store the compressed data in layers, and store a metadata set corresponding to the compressed data;

[0051] A receiving unit, configured to receive a user query request; the user query request includes a query condition and user identity information;

[0052] a detection unit, configured to detect a target data set in a metadata set based on the query condition when the user identity information is verified;

[0053] A processing unit, configured to decompress and decrypt the target data set to obtain a query data set;

[0054] The reorganization unit is used to reorganize the query data set to obtain a query result; the query result is unstructured data.

[0055] Furthermore, the storage unit comprises:

[0056] The first acquisition subunit is used to obtain compressed data obtained after layering, segmenting, checking for duplicates, encrypting and compressing the unstructured data;

[0057] A storage subunit, configured to store the compressed data in layers based on metadata information of the compressed data, and record a storage location of the compressed data; wherein the metadata information includes encryption information, compression information, and attribute information;

[0058] A generating subunit, configured to generate index information corresponding to the compressed data;

[0059] The storage subunit is used to integrate and store the compressed data, the metadata information, the storage location and the index information to obtain a metadata set.

[0060] Further, the encryption information includes the encryption algorithm, encryption status and key information of the compressed data;

[0061] The compression information includes the compression algorithm, compression state, compression ratio and checksum of the compressed data;

[0062] The attribute information includes data sensitivity, access frequency, usage scenario and data type of the compressed data;

[0063] The storage location includes a storage hierarchy and a physical storage location of the compressed data;

[0064] The index information includes a unique hash value of the compressed data and an index identifier.

[0065] Further, the storage subunit is specifically used to store the compressed data in a high-sensitivity data layer located in a high-security storage medium when the data sensitivity of the compressed data is highly sensitive; or

[0066] When the data sensitivity of the compressed data is medium sensitivity, storing the compressed data in a medium sensitivity data layer; or

[0067] When the data sensitivity of the compressed data is low sensitivity, storing the compressed data in a low sensitivity data layer located in a high-performance storage medium;

[0068] The high-security storage medium has higher security than the high-performance storage medium, and the high-performance storage medium has higher performance than the high-security storage medium.

[0069] Furthermore, the processing unit includes:

[0070] A second acquisition subunit is used to acquire a metadata target set corresponding to the target data set;

[0071] An extraction subunit, used to extract the compression algorithm and encryption algorithm corresponding to the target data set from the metadata target set;

[0072] A decompression subunit, configured to decompress the target data set based on a decompression algorithm corresponding to the compression algorithm to obtain a decompressed data set;

[0073] The decryption subunit is used to decrypt the decompressed data set based on the decryption algorithm corresponding to the encryption algorithm to obtain the query data set.

[0074] Furthermore, the processing unit further includes:

[0075] A calculation subunit, configured to calculate a first checksum of each decompressed data in the decompressed data set;

[0076] a judging subunit, configured to judge whether the first checksum is consistent with a second checksum stored in the metadata target set;

[0077] The decryption subunit is specifically configured to, when the judgment result of the judgment subunit is yes, decrypt the decompressed data set based on the decryption algorithm corresponding to the encryption algorithm to obtain the query data set.

[0078] Furthermore, the storage query device for unstructured data further includes:

[0079] An acquisition unit, used to acquire the query behavior record of the user;

[0080] An adjustment unit is used to dynamically adjust the metadata set based on the query behavior record.

[0081] Further, the adjustment unit is specifically configured to increase the priority of the metadata corresponding to the target data set in the metadata set when the query behavior record shows that the query frequency of the target data set is higher than a first preset frequency threshold; or

[0082] When the query behavior record shows that the query frequency of the target data set is lower than a second preset frequency threshold, lowering the priority of the metadata corresponding to the target data set in the metadata set, or removing the metadata corresponding to the target data set from the metadata set; or

[0083] When the query behavior record shows that there is an abnormal query or a potential attack, the index complexity of the metadata set is increased.

[0084] A third aspect of the present application provides an electronic device, comprising a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the storage query method for unstructured data described in any one of the first aspect of the present application.

[0085] A fourth aspect of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer program instructions. When the computer program instructions are read and executed by a processor, the storage query method for unstructured data described in any one of the first aspects of the present application is executed.

[0086] A fifth aspect of the present application provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it executes the storage query method for unstructured data described in any one of the first aspects of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0088] Figure 1 A flowchart of a storage query method for unstructured data provided in an embodiment of the present application;

[0089] Figure 2 A system architecture diagram of a secure encryption system for unstructured data provided in an embodiment of the present application;

[0090] Figure 3 A flowchart of another method for storing and querying unstructured data provided in an embodiment of the present application;

[0091] Figure 4 A schematic diagram of a multi-layer storage structure of a compressed data and metadata set provided in an embodiment of the present application;

[0092] Figure 5 A schematic diagram of a storage format of a metadata set provided in an embodiment of the present application;

[0093] Figure 6 A schematic diagram of the structure of a storage query device for unstructured data provided in an embodiment of the present application;

[0094] Figure 7 A schematic diagram of the structure of another storage query device for unstructured data provided in an embodiment of the present application. DETAILED DESCRIPTION

[0095] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0096] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0097] Example 1

[0098] Please see Figure 1 , Figure 1 A flowchart of a storage query method for unstructured data provided in this embodiment. The storage query method for unstructured data includes:

[0099] S101, receiving a user query request; the user query request includes query conditions and user identity information.

[0100] S102: When the user identity information is verified, a target data set is detected in the metadata set based on the query condition.

[0101] S103: Decompress and decrypt the target data set to obtain a query data set.

[0102] S104: reorganize the query data set to obtain a query result; the query result is unstructured data.

[0103] Please see Figure 2 , Figure 2 A system architecture diagram of a secure encryption system for unstructured data is shown. Among them, the secure encryption system for unstructured data includes six components, namely a data preprocessing system, a data dynamic encryption system, a data dynamic compression system, a data storage system, a data query system and a key management system. It can be seen that the above-mentioned data query system corresponds to the "storage query method based on dynamic encryption / compression" in the figure, and the above-mentioned data storage system corresponds to the "storage management" in the figure, that is, the data query system is used to execute the storage query method for unstructured data proposed in this application, and is specifically used to perform corresponding queries on unstructured data. At the same time, the basis for the query of this method is the stored unstructured data. Therefore, the present application also proposes a storage architecture and an index architecture for unstructured data to assist in the query.

[0104] In this embodiment, the execution subject of the method may be a computing device such as a computer or a server, and no limitation is made in this embodiment.

[0105] In this embodiment, the execution subject of the method may also be a smart device such as a smart phone, a tablet computer, etc., which is not limited in this embodiment.

[0106] It can be seen that implementing the storage query method for unstructured data described in this embodiment can improve the query capability and query efficiency of unstructured data, and can also effectively ensure the security of data query.

[0107] Example 2

[0108] Please see Figure 3 , Figure 3 A flowchart of a storage query method for unstructured data provided in this embodiment. The storage query method for unstructured data includes:

[0109] S201. Obtain compressed data after layering, segmenting, checking for duplicates, encrypting and compressing unstructured data.

[0110] S202. Store the compressed data in layers based on metadata information of the compressed data, and record the storage location of the compressed data; wherein the metadata information includes encryption information, compression information, and attribute information.

[0111] In this embodiment, tiered storage refers to storing compressed data in different storage media according to the sensitivity, access frequency and usage scenario of the data, wherein each layer adopts different encryption and compression processing strategies according to different data requirements (such as security and storage efficiency).

[0112] As an optional implementation, hierarchical storage of compressed data based on metadata information of the compressed data includes:

[0113] When the data sensitivity of the compressed data is highly sensitive, storing the compressed data in a high-sensitivity data layer located in a high-security storage medium; or

[0114] When the data sensitivity of the compressed data is medium sensitivity, the compressed data is stored in the medium sensitivity data layer; or

[0115] When the data sensitivity of the compressed data is low sensitivity, storing the compressed data in a low sensitivity data layer located in a high-performance storage medium;

[0116] Among them, the high-security storage medium has higher security than the high-performance storage medium, and the high-performance storage medium has higher performance than the high-security storage medium.

[0117] For example, the storage data layer includes:

[0118] (1) Highly sensitive data layer

[0119] ①Used to store highly sensitive data involving privacy, commercial secrets, etc.;

[0120] ② Use strong encryption (such as AES-256) and smaller data blocks for block storage;

[0121] ③Data is usually stored in storage devices with high security levels, with security being a priority.

[0122] (2) Medium-sensitivity data layer

[0123] ① Used to store regular business data, using standard encryption algorithms (such as AES-128) and medium-granularity blocks;

[0124] ② The data access frequency in this layer is moderate, and the system will dynamically adjust the storage strategy to balance security and storage efficiency.

[0125] (3) Low-sensitivity data layer

[0126] ①Used to store low-sensitivity data such as logs and statistics;

[0127] ② Use lightweight encryption algorithms and larger-granularity blocks, focusing on reducing storage costs and giving priority to compression.

[0128] By implementing this implementation method, the method can perform hierarchical storage according to the sensitivity of the data, and reflect the storage status of data blocks at different levels through indexes, so as to assist in rapid data search.

[0129] In this embodiment, the encryption information includes the encryption algorithm, encryption status and key information of the compressed data;

[0130] The compression information includes the compression algorithm, compression status, compression ratio and checksum of the compressed data;

[0131] The attribute information includes data sensitivity, access frequency, usage scenario, and data type of the compressed data;

[0132] The storage location includes the storage hierarchy and physical storage location of the compressed data;

[0133] The index information includes a unique hash value of the compressed data and an index identifier.

[0134] In this embodiment, the unique hash value is a unique identifier for each stored compressed data, and the method can determine whether the data block has been stored by the hash value; the index identifier (also known as the index ID) is a unique ID for each compressed data, which is used for identification during the storage and indexing process.

[0135] In this embodiment, the data type may be text, image, video, etc.

[0136] In this embodiment, the encryption algorithm is the encryption algorithm used for recording the data block (such as AES-256, AES-128, etc.);

[0137] The encryption status is whether the data is encrypted, which is used to determine whether decryption is required during query;

[0138] The key information is the key identifier or key storage location used for decryption.

[0139] In this embodiment, the compression algorithm is an algorithm used to compress data blocks (such as Gzip, Zstandard, etc.);

[0140] The compression ratio is the size and ratio of the data block before and after compression, which is used to help evaluate the compression effect;

[0141] The compression status indicates whether the data has been compressed.

[0142] In this embodiment, the storage level is the storage level to which the data belongs (high sensitivity, medium sensitivity, low sensitivity);

[0143] The storage physical location is the specific location of the data block in the physical storage medium.

[0144] In this embodiment, the method can also obtain deduplication reference information, and when the data is duplicate data, determine the original data it references based on the metadata set to avoid repeated storage of the duplicate data.

[0145] S203: Generate index information corresponding to the compressed data.

[0146] S204: Integrate and store the compressed data, metadata information, storage location and index information to obtain a metadata set.

[0147] Please see Figure 4 , Figure 4 A multi-layer storage structure of compressed data and metadata collection is shown. The multi-layer storage structure includes two core parts: hierarchical data storage area and metadata and index management. Specifically, the multi-layer data storage area can be divided into different storage data layers according to the sensitivity level of the data and the difference in access frequency, and the data can be stored in different storage media, thereby realizing flexible and personalized storage and meeting various requirements for data storage and query.

[0148] For metadata and index management, see Figure 5 The storage format of the metadata set is shown. Based on this, the metadata and index management is used to store the metadata set according to the above storage format so that it can realize the index function.

[0149] S205, receiving a user query request; the user query request includes query conditions and user identity information.

[0150] In this embodiment, the user query request is Q request , including query conditions and user identity information.

[0151] In this embodiment, the compressed encrypted data set CompressedData={c 1 ,c 2 ,...,c n} has been stored in the hierarchical data storage area.

[0152] In this embodiment, the metadata set M={m 1 ,m 2 ,...,m n} has also been stored. Among them, each metadata m i Contains compression and encryption information for a data block.

[0153] In this embodiment, the method can query the request Q request Extract the query condition Condition and user identity information UserInfo.

[0154] S206: When the user identity information is verified, the target data set is detected in the metadata set based on the query condition.

[0155] In this embodiment, the method can perform user identity verification, and when the user identity UserInfo verification fails, the query request is rejected and the algorithm ends.

[0156] S207: Obtain a metadata target set corresponding to the target data set.

[0157] In this embodiment, the method can retrieve the target data set TargetBlocks (the target data is the target data block, so the target data set is actually the target data block set) that meets the query condition in the metadata set M. The calculation formula is as follows:

[0158] TargetBlocks = {c i ∈CompressedData|Condition};

[0159] Among them, if TargetBlocks is not encrypted or compressed, the method can directly return the query result R; if TargetBlocks is retrieved, the process ends.

[0160] S208. Extract the compression algorithm and encryption algorithm corresponding to the target data set from the metadata target set.

[0161] In this embodiment, the method can be used to calculate the target data c in each target data set. i ∈TargetBlocks for decompression and decryption.

[0162] For example, this method can obtain the corresponding metadata m i , containing the following information:

[0163] (1) Compression algorithm: compressAlgo i ;

[0164] (2) Encryption algorithm: encryptAlgo i ;

[0165] (3) Sensitivity level: si;

[0166] (4) Data type: t i ;

[0167] (5) Checksum i .

[0168] S209: Decompress the target data set based on a decompression algorithm corresponding to the compression algorithm to obtain a decompressed data set.

[0169] For example, if compressAlgo i ≠None, decompression is required.

[0170] At this point, select the decompression algorithm: decompressAlgo i =compressAlgo i .decompress;

[0171] And through the formula c i ′=compressAlgo i .decompress(c i ) to perform the decompression operation.

[0172] S210: Calculate a first checksum of each decompressed data in the decompressed data set.

[0173] S211, determine whether the first checksum is consistent with the second checksum stored in the metadata target set, if so, execute step S212; if not, end this process.

[0174] For example, the method may calculate the checksum (ie, the first checksum) of the decompressed data after decompression:

[0175] checksum i ′=Hash(c i ),

[0176] And compare the checksum i ′ and metadata m i The original checksum in i (i.e. the second checksum), if the check passes, proceed to the next step.

[0177] S212. Decrypt the decompressed data set based on a decryption algorithm corresponding to the encryption algorithm to obtain a query data set.

[0178] For example, the specific steps of decrypting data in the method may include:

[0179] (1) According to the encryption algorithm encryptAlgo i , select the corresponding decryption algorithm decryptAlgo i ;

[0180] (2) If it is symmetric encryption, directly use the UserInfo private key information for decryption;

[0181] (3) If attribute-based encryption (ABE) is included: generating or obtaining decryption key components based on user attributes;

[0182] (4) Execute decryption: d i =decryptAlgo i (c i ,k i );

[0183] (5) The decrypted data d i Add to the query result set R.

[0184] In this embodiment, since the unstructured data is processed by layers and blocks, the objects of decompression and decryption in this method are all data blocks. Therefore, in order to restore the complete data, the method will decompress and decrypt multiple data blocks and finally reorganize them into a complete data.

[0185] S213. Reorganize the query data set to obtain a query result; the query result is unstructured data.

[0186] In this embodiment, the query result is R, and the decryption result is the plaintext data after decompression and decryption. Before the method performs the query, the data structure where the query result is located needs to be initialized, that is,

[0187]

[0188] In this embodiment, the method can assemble the query result set R according to the logical order and relationship of the original data to form a complete query result, and then return the assembled query result to the user.

[0189] S214: Obtain the user's query behavior record.

[0190] In this embodiment, the method can archive query records, record the user's query behavior, including the query keywords, query time and matching results. The system can optimize the index based on these records and dynamically adjust the encryption and compression strategies.

[0191] S215. Dynamically adjust the metadata set based on the query behavior record.

[0192] In this embodiment, the method can dynamically adjust the index according to the access frequency of the data, so that frequently accessed data blocks are preferentially entered into the index, thereby improving query efficiency.

[0193] As an optional implementation, dynamically adjusting the metadata set based on the query behavior record includes:

[0194] When the query behavior record shows that the query frequency of the target data set is higher than a first preset frequency threshold, increasing the priority of the metadata corresponding to the target data set in the metadata set; or

[0195] When the query behavior record shows that the query frequency of the target data set is lower than a second preset frequency threshold, lowering the priority of the metadata corresponding to the target data set in the metadata set, or removing the metadata corresponding to the target data set from the metadata set; or

[0196] When the query behavior record shows that there are abnormal queries or potential attacks, the index complexity of the metadata collection is increased.

[0197] In this embodiment, the method can dynamically optimize the index according to the query mode and access frequency. Frequently accessed data can be prioritized in the index to improve the response speed; and for infrequently accessed data, the index will be flexibly adjusted according to demand to reduce unnecessary retrieval overhead.

[0198] In this embodiment, when the method detects an abnormal query pattern or a potential attack, the index can be dynamically adjusted to make certain sensitive data blocks more difficult to locate.

[0199] In this embodiment, the execution subject of the method may be a computing device such as a computer or a server, and no limitation is made in this embodiment.

[0200] In this embodiment, the execution subject of the method may also be a smart device such as a smart phone, a tablet computer, etc., which is not limited in this embodiment.

[0201] It can be seen that implementing the storage query method for unstructured data described in this embodiment can improve the query capability and query efficiency of unstructured data, and can also effectively ensure the security of data query.

[0202] Example 3

[0203] This embodiment provides three example processes of storage query methods for unstructured data. Among them, when facing unstructured data of text data type, the example text is as follows: "User information: Name: Alice, Address: 123 Street, Transaction details: Purchased 500 goods", based on this, the method includes:

[0204] S1. Data block retrieval

[0205] The system retrieves the database block requested by the user according to the query conditions through the index. For example, if the user queries "transaction details", the system locates data block 2 (block 2);

[0206] S2. Compression processing

[0207] According to the metadata of block 2, select the appropriate decompression algorithm for processing. After the above steps, the metadata indicates that block 2 uses the LZ4 compression algorithm:

[0208] DecompressedBlock2=LZ4_Decompress(CompressedBlock2)

[0209] S3. Integrity check

[0210] (1) Calculate the checksum of the decompressed data block:

[0211] Checksum′ 2 =Hash(DecompressedBlock2)

[0212] (2) Compare the checksum to ensure that the data has not been tampered with:

[0213]

[0214] (3) If the verification passes, proceed to the next step.

[0215] (4) If the verification fails, an error log is recorded and the processing of the current data block is terminated.

[0216] S4. Decryption processing

[0217] After decompression, the system identifies the encryption algorithm as AES-64 based on the metadata of block 2:

[0218] DecryptedBlock2=AES64_Decrypt(DecompressedBlock2,Key2′)

[0219] S5. Reorganization and mapping of query results

[0220] (1) Mapping relationship determination: Determine the positional relationship between the data block and the original data based on the query conditions and the mapping information in the metadata. In this example, block 2 corresponds to the "transaction details" section.

[0221] (2) Result assembly:

[0222] ① Assemble the decrypted data blocks according to the structure of the original text.

[0223] ② If the query involves multiple data blocks (e.g., block 1 and block 2), they are concatenated according to their order in the original text.

[0224] ③Result after assembly: Result = DecryptedBlock1 + DecryptedBlock2.

[0225] ④For this query, only the “transaction details” need to be returned, so the result is: Result = DecryptedBlock2.

[0226] S6. Return query results: Return the assembled query results to the user. Ensure that the result format meets the user's expectations, for example, keep the original text format and structure.

[0227] S7. Record the relevant information of this query, including query time, user ID, query content, processing results, etc., for the convenience of auditing and monitoring.

[0228] S8、 Metadata management: record the encryption algorithm change and compression status of block 2:

[0229] S9, index update: Update the encrypted keywords and positions of block 2 in the index to ensure correct matching during query.

[0230] In addition, when facing unstructured data of image data type, the example scenario is as follows: a user uploads an ID card photo containing personal face and ID card number, and the image needs to be securely stored and managed while supporting efficient query and defending against potential attacks. Based on this, the method includes:

[0231] S1. Target data block retrieval

[0232] In the metadata set M, the target data block set TargetBlocks that meets the query condition is retrieved. It includes the facial data block {c 1,1 ,c 1,2 ,…}, ID card number data block c 2,1 and the associated instance data blocks.

[0233] S2. For each data block c i ∈TargetBlocks:

[0234] (1) Obtain metadata mi.

[0235] (2) Decompress (if necessary). i , perform the decompression operation: c i ′=compressAlgo i .decompress(c i ).

[0236] (3) Data integrity check. Calculate the checksum and compare:

[0237] checksum i ′=Hash(c i ′)

[0238]

[0239] (4) Decryption. According to encryptAlgo i , select the decryption algorithm decryptAlgo i . Using the corresponding key k i To decrypt: c i =decryptAlgo i (c i ′,k i ).

[0240] (5) The decrypted data block c i Add to the query result set R.

[0241] S3. Reorganization of query results

[0242] (1) Reorganize data blocks according to the mapping relationship:

[0243] ①Use the coordinate information in the metadata to convert each data block c i Place it in the correct position and reconstruct the complete image.

[0244] ② Processing image edges and fusion: For image data, edge smoothing and color correction may be required to ensure consistent visual effects of the stitched images.

[0245] S4. Return the final query result: assign the reorganized complete image to R. Return R to the user who initiated the query request.

[0246] S5. Record the relevant information of this query, including query time, user ID, query content, processing results, etc., for the convenience of auditing and monitoring.

[0247] S6、 For each data block, record the following information:

[0248] (1) Basic information: data block ID, type, size, etc.

[0249] (2) Encryption information: encryption algorithm, key identifier, and encryption status.

[0250] (3) Compression information: compression algorithm, compression ratio, and compression status.

[0251] (4) Storage location information: the location of the data block in the physical storage medium.

[0252] S7. Index management

[0253] Create encrypted indexes, use encrypted keywords and data block identifiers, and support secure and efficient queries.

[0254] S8. Index update

[0255] When encryption or compression strategies change, update the index in a timely manner to ensure the accuracy and effectiveness of queries.

[0256] Finally, when facing unstructured data of the video data type, an example scenario is as follows: a company's internal meeting record video contains sensitive business secrets, and the video needs to be securely stored and managed while supporting efficient query and defending against potential attacks. Based on this, the method includes:

[0257] S1. Target data block retrieval

[0258] In the metadata set M, the target data block set TargetBlocks that meets the query condition is retrieved. It includes the facial data block {c 1,1 ,c 1,2 ,…}, ID card number data block c 2,1 and the associated instance data blocks.

[0259] S2. For each data block c i ∈TargetBlocks:

[0260] (1) Obtain metadata mi.

[0261] (2) Decompress (if necessary). i , perform the decompression operation:

[0262] c i ′=compressAlgo i .decompress(c i ).

[0263] (3) Data integrity check. Calculate the checksum and compare:

[0264] checksum i ′=Hash(c i ′)

[0265]

[0266] (4) Decryption. According to encryptAlgo i , select the decryption algorithm decryptAlgo i . Using the corresponding key k i To decrypt: ci =decryptAlgo i (c i ′,k i ).

[0267] (5) The decrypted data block c i Add to the query result set R.

[0268] S3, reorganization of query results: according to the mapping relationship in the metadata and the structural information of the original video, the decrypted data block B i,x Perform correct time series restoration and splicing to reconstruct the complete video clip.

[0269] (1) Data block sorting: Sort the decrypted data blocks according to their timestamps or sequence numbers.

[0270] (2) Video reconstruction: splicing the data blocks in chronological order to form a continuous video clip.

[0271] (3) Processing re-blocked data blocks: If some data blocks are re-blocked during the dynamic encryption adjustment process (such as highly sensitive data blocks are split into smaller granularities), they need to be spliced ​​according to the new block information.

[0272] S4. Return the final query result: assign the reorganized complete image to R. Return R to the user who initiated the query request.

[0273] S5. Record the relevant information of this query, including query time, user ID, query content, processing results, etc., for the convenience of auditing and monitoring.

[0274] S6. For each data block, record the following information:

[0275] (1) Basic information: data block ID, type, size, etc.

[0276] (2) Encryption information: encryption algorithm, key identifier, and encryption status.

[0277] (3) Compression information: compression algorithm, compression ratio, and compression status.

[0278] (4) Storage location information: the location of the data block in the physical storage medium.

[0279] S7. Create an encrypted index

[0280] Use encrypted keywords and data block identifiers to support secure and efficient queries.

[0281] S8. Index update

[0282] When encryption or compression strategies change, update the index in a timely manner to ensure the accuracy and effectiveness of queries.

[0283] It can be seen that the implementation of the storage query device for unstructured data described in this embodiment can improve the query capability and efficiency of unstructured data, and can also effectively ensure the security of data query.

[0284] Example 4

[0285] Please see Figure 6 , Figure 6 A schematic diagram of the structure of a storage query device for unstructured data provided in this embodiment. Figure 6 As shown, the storage query device for unstructured data includes:

[0286] The storage unit 310 is used to obtain compressed data corresponding to the unstructured data, store the compressed data in layers, and store a metadata set corresponding to the compressed data;

[0287] The receiving unit 320 is used to receive a user query request; the user query request includes a query condition and user identity information;

[0288] A detection unit 330, configured to detect a target data set in a metadata set based on a query condition when the user identity information is verified;

[0289] The processing unit 340 is used to decompress and decrypt the target data set to obtain a query data set;

[0290] The reorganization unit 350 is used to reorganize the query data set to obtain a query result; the query result is unstructured data.

[0291] In this embodiment, the explanation of the storage query device for unstructured data can refer to the description in Embodiment 1 or Embodiment 2, which will not be further elaborated in this embodiment.

[0292] It can be seen that the implementation of the storage query device for unstructured data described in this embodiment can improve the query capability and efficiency of unstructured data, and can also effectively ensure the security of data query.

[0293] Example 5

[0294] Please see Figure 7 , Figure 7 A schematic diagram of the structure of a storage query device for unstructured data provided in this embodiment. Figure 7 As shown, the storage query device for unstructured data includes:

[0295] The storage unit 310 is used to obtain compressed data corresponding to the unstructured data, store the compressed data in layers, and store a metadata set corresponding to the compressed data;

[0296] The receiving unit 320 is used to receive a user query request; the user query request includes a query condition and user identity information;

[0297] A detection unit 330, configured to detect a target data set in a metadata set based on a query condition when the user identity information is verified;

[0298] The processing unit 340 is used to decompress and decrypt the target data set to obtain a query data set;

[0299] The reorganization unit 350 is used to reorganize the query data set to obtain a query result; the query result is unstructured data.

[0300] As an optional implementation, the storage unit 310 further includes:

[0301] The first acquisition subunit 311 is used to obtain compressed data obtained after layering, segmenting, checking for duplicates, encrypting and compressing the unstructured data;

[0302] The storage subunit 312 is used to store the compressed data in layers based on the metadata information of the compressed data, and record the storage location of the compressed data; wherein the metadata information includes encryption information, compression information and attribute information;

[0303] A generating subunit 313, used to generate index information corresponding to the compressed data;

[0304] The storage subunit 312 is used to integrate and store the compressed data, metadata information, storage location and index information to obtain a metadata set.

[0305] In this embodiment, the encryption information includes the encryption algorithm, encryption status and key information of the compressed data;

[0306] The compression information includes the compression algorithm, compression status, compression ratio and checksum of the compressed data;

[0307] The attribute information includes data sensitivity, access frequency, usage scenario, and data type of the compressed data;

[0308] The storage location includes the storage hierarchy and physical storage location of the compressed data;

[0309] The index information includes a unique hash value of the compressed data and an index identifier.

[0310] As an optional implementation, the storage subunit 312 is specifically configured to store the compressed data in a high-sensitivity data layer located in a high-security storage medium when the data sensitivity of the compressed data is highly sensitive; or

[0311] When the data sensitivity of the compressed data is medium sensitivity, the compressed data is stored in the medium sensitivity data layer; or

[0312] When the data sensitivity of the compressed data is low sensitivity, storing the compressed data in a low sensitivity data layer located in a high-performance storage medium;

[0313] Among them, the high-security storage medium has higher security than the high-performance storage medium, and the high-performance storage medium has higher performance than the high-security storage medium.

[0314] As an optional implementation, the processing unit 340 includes:

[0315] The second acquisition subunit 341 is used to acquire a metadata target set corresponding to a target data set;

[0316] The extraction subunit 342 is used to extract the compression algorithm and encryption algorithm corresponding to the target data set from the metadata target set;

[0317] A decompression subunit 343 is used to decompress the target data set based on a decompression algorithm corresponding to the compression algorithm to obtain a decompressed data set;

[0318] The decryption subunit 344 is used to decrypt the decompressed data set based on the decryption algorithm corresponding to the encryption algorithm to obtain the query data set.

[0319] As an optional implementation, the processing unit 340 further includes:

[0320] A calculation subunit 345, configured to calculate a first checksum of each decompressed data in the decompressed data set;

[0321] A determination subunit 346, configured to determine whether the first checksum is consistent with a second checksum stored in the metadata target set;

[0322] The decryption subunit 344 is specifically configured to decrypt the decompressed data set based on a decryption algorithm corresponding to the encryption algorithm to obtain a query data set when the judgment result of the judgment subunit 346 is yes.

[0323] As an optional implementation, the storage query device for unstructured data further includes:

[0324] The acquisition unit 360 is used to acquire the query behavior record of the user;

[0325] The adjustment unit 370 is used to dynamically adjust the metadata set based on the query behavior record.

[0326] As an optional implementation, the adjustment unit 370 is specifically configured to increase the priority of the metadata corresponding to the target data set in the metadata set when the query behavior record shows that the query frequency of the target data set is higher than the first preset frequency threshold; or

[0327] When the query behavior record shows that the query frequency of the target data set is lower than a second preset frequency threshold, lowering the priority of the metadata corresponding to the target data set in the metadata set, or removing the metadata corresponding to the target data set from the metadata set; or

[0328] When the query behavior record shows that there are abnormal queries or potential attacks, the index complexity of the metadata collection is increased.

[0329] In this embodiment, the explanation of the storage query device for unstructured data can refer to the description in Embodiment 1 or Embodiment 2, which will not be further elaborated in this embodiment.

[0330] It can be seen that the implementation of the storage query device for unstructured data described in this embodiment can improve the query capability and efficiency of unstructured data, and can also effectively ensure the security of data query.

[0331] An embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the storage query method for unstructured data in Embodiment 1 or Embodiment 2 of the present application.

[0332] An embodiment of the present application provides a computer-readable storage medium storing computer program instructions. When the computer program instructions are read and executed by a processor, the storage query method for unstructured data in Embodiment 1 or Embodiment 2 of the present application is executed.

[0333] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0334] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0335] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0336] The above description is only an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0337] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0338] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

Claims

1. A storage query method for unstructured data, characterized in that: include: Obtaining compressed data corresponding to the unstructured data, and storing the compressed data in layers, while storing a metadata set corresponding to the compressed data; Receiving a user query request; the user query request includes query conditions and user identity information; When the user identity information is verified, detecting a target data set in the metadata set based on the query condition; Decompressing and decrypting the target data set to obtain a query data set; Reorganizing the query data set to obtain a query result; The query result is unstructured data.

2. The storage query method for unstructured data according to claim 1, characterized in that: The step of obtaining compressed data corresponding to the unstructured data, storing the compressed data in layers, and storing a metadata set corresponding to the compressed data includes: Obtain compressed data after layering, segmenting, checking for duplicates, encrypting and compressing unstructured data; Based on the metadata information of the compressed data, the compressed data is stored in layers, and the storage location of the compressed data is recorded; wherein the metadata information includes encryption information, compression information and attribute information; generating index information corresponding to the compressed data; The compressed data, the metadata information, the storage location and the index information are integrated and stored to obtain a metadata set.

3. The storage query method for unstructured data according to claim 2, characterized in that: The encryption information includes the encryption algorithm, encryption status and key information of the compressed data; The compression information includes the compression algorithm, compression state, compression ratio and checksum of the compressed data; The attribute information includes data sensitivity, access frequency, usage scenario and data type of the compressed data; The storage location includes a storage hierarchy and a physical storage location of the compressed data; The index information includes a unique hash value of the compressed data and an index identifier.

4. The storage query method for unstructured data according to claim 3, characterized in that: The step of storing the compressed data in layers based on the metadata information of the compressed data includes: When the data sensitivity of the compressed data is highly sensitive, storing the compressed data in a high-sensitivity data layer located in a high-security storage medium; or When the data sensitivity of the compressed data is medium sensitivity, storing the compressed data in a medium sensitivity data layer; or When the data sensitivity of the compressed data is low sensitivity, storing the compressed data in a low sensitivity data layer located in a high-performance storage medium; The high-security storage medium has higher security than the high-performance storage medium, and the high-performance storage medium has higher performance than the high-security storage medium.

5. The storage query method for unstructured data according to claim 1, characterized in that: The decompressing and decrypting the target data set to obtain the query data set includes: Acquire a metadata target set corresponding to the target data set; Extracting the compression algorithm and encryption algorithm corresponding to the target data set from the metadata target set; Decompressing the target data set based on a decompression algorithm corresponding to the compression algorithm to obtain a decompressed data set; The decompressed data set is decrypted based on a decryption algorithm corresponding to the encryption algorithm to obtain a query data set.

6. The storage query method for unstructured data according to claim 5, characterized in that: The method further comprises: Calculating a first checksum of each decompressed data in the decompressed data set; determining whether the first checksum is consistent with a second checksum stored in the metadata target set; If so, the decryption algorithm corresponding to the encryption algorithm is executed to decrypt the decompressed data set to obtain the query data set.

7. The storage query method for unstructured data according to claim 1, characterized in that: The method further comprises: Get the user's query behavior record; Based on the query behavior record, the metadata set is dynamically adjusted.

8. The storage query method for unstructured data according to claim 7, characterized in that: The dynamically adjusting the metadata set based on the query behavior record includes: When the query behavior record shows that the query frequency of the target data set is higher than a first preset frequency threshold, increasing the priority of the metadata corresponding to the target data set in the metadata set; or When the query behavior record shows that the query frequency of the target data set is lower than a second preset frequency threshold, lowering the priority of the metadata corresponding to the target data set in the metadata set, or removing the metadata corresponding to the target data set from the metadata set; or When the query behavior record shows that there is an abnormal query or a potential attack, the index complexity of the metadata set is increased.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the storage query method for unstructured data according to any one of claims 1 to 8.

10. A readable storage medium, characterized in that: The readable storage medium stores computer program instructions, and when the computer program instructions are read and executed by a processor, the storage query method for unstructured data according to any one of claims 1 to 8 is executed.

11. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the computer program executes the storage query method for unstructured data according to any one of claims 1 to 8.

Images