Preprocessing method and device for unstructured data, medium and program product

Abstract

The invention provides an unstructured data preprocessing method, electronic equipment, a readable storage medium and a computer program product, and the method comprises the steps: recognizing the layering dimension of unstructured data, and determining a layering standard based on the layering dimension; performing hierarchical processing on the unstructured data based on a hierarchical standard to obtain hierarchical data; identifying data characteristics of the hierarchical data, and determining a corresponding partitioning algorithm and a partitioning size based on the data characteristics; carrying out blocking processing on the hierarchical data based on a blocking algorithm and a blocking size to obtain blocking data; and performing de-duplication processing on the block data to obtain pre-processed data. The method can ensure that the system can flexibly meet the storage requirements of various data formats, so that the data storage efficiency is guaranteed, and the storage cost and the management difficulty are reduced.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method for preprocessing unstructured data, an electronic device, a readable storage medium, and a computer program product. Background Art

[0002] With the rapid advancement of information technology, the demand for storage and management of unstructured data (including documents, pictures, videos, etc.) has increased dramatically. Based on this, the traditional storage method is usually to store unstructured data directly on the server or cloud storage platform. However, this method will lead to poor storage efficiency due to the diverse types and complex formats of unstructured data; at the same time, the large amount of duplicate data that may exist will also increase storage costs and management difficulties. Summary of the invention

[0003] The purpose of this application is to provide a method and device for preprocessing unstructured data, which can ensure that the system can flexibly respond to storage requirements of various data formats, thereby ensuring data storage efficiency and reducing storage costs and management difficulties.

[0004] The first aspect of the present application provides a method for preprocessing unstructured data, comprising:

[0005] Identifying hierarchical dimensions of unstructured data and determining hierarchical criteria based on the hierarchical dimensions;

[0006] Performing hierarchical processing on the unstructured data based on the hierarchical standard to obtain hierarchical data;

[0007] Identifying data characteristics of the hierarchical data, and determining corresponding block algorithms and block sizes based on the data characteristics;

[0008] Performing block processing on the hierarchical data based on the block algorithm and the block size to obtain block data;

[0009] De-duplication processing is performed on the block data to obtain pre-processed data.

[0010] In the above implementation process, the method can identify the hierarchical dimensions of unstructured data and determine the hierarchical standards, so that the system can understand and organize data more finely, provide a basis for subsequent storage and processing, thereby enhancing the flexibility and adaptability of data processing, and helping to better meet the storage requirements of different data formats. Secondly, the hierarchical processing method can make the data more orderly in storage and management, which is conducive to improving the efficiency of data processing and providing convenience for the rapid retrieval and access of subsequent data. Then, according to the data characteristics of the hierarchical data, the appropriate block algorithm and block size are selected to optimize the storage layout of the data, reduce storage fragmentation, and improve the utilization of storage space, thereby reducing storage costs. After that, through block processing, the data can be stored and transmitted more efficiently, and it also provides the possibility for parallel processing and distributed storage of data, thereby improving the storage and processing capabilities of the system. Finally, deduplication processing can eliminate the redundant parts in the data, reduce the waste of storage space, and also help to improve the efficiency of data processing and reduce the difficulty of system management.

[0011] Furthermore, the identifying the hierarchical dimensions of the unstructured data includes:

[0012] Identify the data type and data attributes of unstructured data through a preset data identification model or preset data identification rules;

[0013] Based on the data type and the data attribute, determine the hierarchical dimension of the unstructured data; the hierarchical dimension includes at least one of data sensitivity, data access frequency, and data usage scenario.

[0014] In the above implementation process, the method can accurately identify the data type and attributes of unstructured data through preset data recognition models or rules, thereby determining its hierarchical dimensions (such as sensitivity, access frequency, usage scenarios, etc.), and then can enhance the system's understanding and organization capabilities of unstructured data, help optimize data management strategies, and reduce storage and management costs.

[0015] Furthermore, determining the stratification standard based on the stratification dimension includes:

[0016] When the stratification dimension is data sensitivity, determining the stratification criteria includes at least high sensitivity and low sensitivity; or

[0017] When the stratification dimension is data access frequency, determining the stratification criteria includes at least high-frequency access and low-frequency access; or

[0018] When the stratification dimension is a data usage scenario, the stratification standard is determined to include at least a specific business scenario and a general business scenario.

[0019] In the above implementation process, the method can determine clear stratification standards (such as high / low sensitivity, high / low frequency access, specific / regular business scenarios) based on stratification dimensions (such as data sensitivity, access frequency, and usage scenarios), thereby helping the system to finely classify and manage different types of data, thereby improving the pertinence and efficiency of data storage, and at the same time facilitating the optimization of resource allocation.

[0020] Further, the determining of the corresponding block algorithm and block size based on the data characteristics includes:

[0021] When the data characteristic is highly sensitive, determining the corresponding block algorithm to be a block algorithm based on content definition, and determining the corresponding block size to be a small block; or

[0022] When the data characteristic is low sensitivity, determining the corresponding block algorithm as a preset standard block algorithm, and determining the corresponding block size as a large block; or

[0023] When the data characteristic is high-frequency access, determining the corresponding block size to be small; or

[0024] When the data characteristic is low-frequency access, determining the corresponding block size to be a large block; or

[0025] When the data characteristic is a specific business scenario, determining the corresponding block size to be a first customized size corresponding to the specific business scenario; or

[0026] When the data characteristic is a conventional business scenario, the corresponding block size is determined to be a second customized size corresponding to the business entity.

[0027] In the above implementation process, the method can flexibly select the block algorithm and block size according to data characteristics (such as sensitivity, access frequency, and business scenarios), which is conducive to optimizing the data storage layout and improving storage efficiency.

[0028] Furthermore, the deduplication processing is performed on the block data to obtain preprocessed data, including:

[0029] Performing hash calculation on the block data to obtain a block hash value;

[0030] Performing duplication detection on the block hash values ​​based on a global hash table;

[0031] When the block hash value fails the duplication detection, a reference pointer is generated; the reference pointer points to the stored duplicate data;

[0032] The block data is discarded.

[0033] In the above implementation process, the method can effectively identify and remove duplicate data by performing hash calculation and duplicate detection on the block data, only store unique data blocks, and generate reference pointers to point to duplicate data, significantly reducing storage space occupancy, improving storage efficiency, and reducing storage costs.

[0034] Further, the method comprises:

[0035] When the block hash value passes the duplication detection, recording the block hash value into the full set hash table;

[0036] The block data are stored to obtain preprocessed data.

[0037] In the above implementation process, when the block data is unique, the method can record its hash value to the full set hash table and store the data to ensure data integrity. At the same time, it can use the hash table for fast retrieval to improve data storage and retrieval efficiency and optimize the data management process.

[0038] Furthermore, the deduplication processing is performed on the block data to obtain preprocessed data, including:

[0039] Deduplication processing is performed on the block data to obtain a plurality of block non-duplication data;

[0040] The plurality of block non-duplicate data are classified to obtain pre-processed data.

[0041] In the above implementation process, the method can effectively reduce redundant data by deduplicating and classifying the block data, ensure the uniqueness and orderliness of the stored data, improve the utilization of storage space, and optimize data management, providing an efficient and accurate data foundation for subsequent data processing and analysis.

[0042] A second aspect of the present application provides a preprocessing device for unstructured data, the preprocessing device for unstructured data comprising:

[0043] A first identification unit, configured to identify a hierarchical dimension of the unstructured data and determine a hierarchical standard based on the hierarchical dimension;

[0044] A stratification unit, configured to perform stratification processing on the unstructured data based on the stratification standard to obtain stratified data;

[0045] A second identification unit, used to identify data characteristics of the hierarchical data, and determine a corresponding block algorithm and block size based on the data characteristics;

[0046] A block division unit, configured to perform block processing on the layered data based on the block division algorithm and the block size to obtain block data;

[0047] The deduplication unit is used to perform deduplication processing on the block data to obtain pre-processed data.

[0048] Furthermore, the first identification unit includes:

[0049] An identification subunit, used to identify the data type and data attribute of the unstructured data through a preset data identification model or a preset data identification rule;

[0050] A determination subunit is used to determine the hierarchical dimension of the unstructured data based on the data type and the data attribute; the hierarchical dimension includes at least one of data sensitivity, data access frequency, and data usage scenario.

[0051] Furthermore, the first identification unit is specifically configured to, when the stratification dimension is data sensitivity, determine that the stratification standard at least includes high sensitivity and low sensitivity; or

[0052] When the stratification dimension is data access frequency, determining the stratification criteria includes at least high-frequency access and low-frequency access; or

[0053] When the stratification dimension is a data usage scenario, the stratification standard is determined to include at least a specific business scenario and a general business scenario.

[0054] Further, the second identification unit is specifically configured to, when the data characteristic is highly sensitive, determine that the corresponding block algorithm is a block algorithm based on content definition, and determine that the corresponding block size is a small block; or

[0055] When the data characteristic is low sensitivity, determining the corresponding block algorithm as a preset standard block algorithm, and determining the corresponding block size as a large block; or

[0056] When the data characteristic is high-frequency access, determining the corresponding block size to be small; or

[0057] When the data characteristic is low-frequency access, determining the corresponding block size to be a large block; or

[0058] When the data characteristic is a specific business scenario, determining the corresponding block size to be a first customized size corresponding to the specific business scenario; or

[0059] When the data characteristic is a conventional business scenario, the corresponding block size is determined to be a second customized size corresponding to the business entity.

[0060] Furthermore, the deduplication unit comprises:

[0061] A calculation subunit, used to perform hash calculation on the block data to obtain a block hash value;

[0062] A duplicate checking subunit, used for performing duplicate detection on the block hash values ​​based on a global hash table;

[0063] A generating subunit, used for generating a reference pointer when the block hash value fails the duplication detection; the reference pointer points to the stored duplicate data;

[0064] The discarding subunit is used to discard the block data.

[0065] Furthermore, the deduplication unit further includes:

[0066] A recording subunit, configured to record the block hash value into a full set hash table when the block hash value passes the duplication detection;

[0067] The storage subunit is used to store the block data to obtain pre-processed data.

[0068] Furthermore, the deduplication unit is specifically used to perform deduplication processing on the block data to obtain multiple block non-duplication data; and classify the multiple block non-duplication data to obtain pre-processed data.

[0069] A third aspect of the present application provides an electronic device, comprising a memory and a processor, the memory being used to store a computer program, the processor running the computer program to enable the electronic device to perform the unstructured data preprocessing method described in any one of the first aspect of the present application.

[0070] A fourth aspect of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer program instructions. When the computer program instructions are read and executed by a processor, the method for preprocessing unstructured data described in any one of the first aspect of the present application is executed.

[0071] A fifth aspect of the present application provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it executes the unstructured data preprocessing method described in any one of the first aspects of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0073] Figure 1A schematic diagram of a process for preprocessing unstructured data provided in an embodiment of the present application;

[0074] Figure 2 A system architecture diagram of a secure encryption system for unstructured data provided in an embodiment of the present application;

[0075] Figure 3 A schematic diagram of a process for preprocessing unstructured data provided in an embodiment of the present application;

[0076] Figure 4 A schematic diagram of the structure of a preprocessing device for unstructured data provided in an embodiment of the present application;

[0077] Figure 5 A schematic diagram of the structure of another unstructured data preprocessing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0078] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0079] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0080] Example 1

[0081] Please see Figure 1 , Figure 1 A flowchart of a method for preprocessing unstructured data provided in this embodiment. The method for preprocessing unstructured data includes:

[0082] S101. Identify hierarchical dimensions of unstructured data, and determine hierarchical standards based on the hierarchical dimensions.

[0083] S102: Perform stratification processing on the unstructured data based on the stratification standard to obtain stratified data.

[0084] S103: Identify data characteristics of the hierarchical data, and determine corresponding block algorithms and block sizes based on the data characteristics.

[0085] S104: performing block processing on the layered data based on a block algorithm and a block size to obtain block data.

[0086] S105: De-duplicate the block data to obtain pre-processed data.

[0087] As an optional implementation, deduplication processing is performed on the block data to obtain preprocessed data, including:

[0088] De-duplicate the block data to obtain multiple block non-duplicate data;

[0089] The non-duplicate data of multiple blocks are classified to obtain pre-processed data.

[0090] In this embodiment, the method is to obtain multiple block data after layering and block processing of unstructured data, and store the multiple block data. Among them, the multiple stored block data are collectively referred to as preprocessed data. It should be understood that each block data is a part of the preprocessed data, which is preprocessed data but not completely preprocessed data.

[0091] Please see Figure 2 , Figure 2 A system architecture diagram of a secure encryption system for unstructured data is shown. The secure encryption system for unstructured data includes six components, namely a data preprocessing system, a data dynamic encryption system, a data dynamic compression system, a data storage system, a data query system, and a key management system. It can be seen that the above-mentioned data preprocessing system corresponds to the "data stratification, block division and deduplication" in the figure, that is, the data preprocessing system is used to execute the preprocessing method of unstructured data proposed in this application, specifically for data stratification, block division and deduplication of unstructured data.

[0092] In this embodiment, the execution subject of the method may be a computing device such as a computer or a server, and no limitation is made in this embodiment.

[0093] In this embodiment, the execution subject of the method may also be a smart device such as a smart phone, a tablet computer, etc., which is not limited in this embodiment.

[0094] It can be seen that the implementation of the unstructured data preprocessing method described in this embodiment can provide corresponding data preprocessing methods for different types of unstructured data such as text, pictures, and videos according to the sensitivity, access frequency, and usage scenarios of the data, so that it can meet the data security requirements of different businesses, and at the same time it is also conducive to improving the efficiency of data storage and query processing. On the other hand, the method can also be combined with data deduplication technology to significantly reduce the storage space occupied, thereby improving the storage efficiency of unstructured data and optimizing the utilization of system storage resources.

[0095] Example 2

[0096] Please see Figure 3 , Figure 3A flowchart of a method for preprocessing unstructured data provided in this embodiment. The method for preprocessing unstructured data includes:

[0097] S201. Identify the data type and data attribute of unstructured data through a preset data identification model or preset data identification rule.

[0098] As an optional implementation, before identifying the data type and data attribute of the unstructured data by using a preset data identification model or a preset data identification rule, the method may further include:

[0099] Receive unstructured data uploaded by users.

[0100] In this embodiment, based on the above content, the method can automatically identify the data type and data attributes of unstructured data through a preset algorithm or model. Specifically, the method can use machine learning or rule matching methods to identify the type of data; at the same time, extract the sensitivity, access frequency, usage scenario and other attributes of the data to provide a basis for subsequent stratification and segmentation.

[0101] In this embodiment, in order to ensure that data is protected during transmission and identification, the method may use the SSL / TLS protocol to encrypt data transmission.

[0102] S202. Determine the hierarchical dimensions of unstructured data based on data types and data attributes; the hierarchical dimensions include at least one of data sensitivity, data access frequency, and data usage scenarios.

[0103] S203. Determine a stratification standard based on the stratification dimension.

[0104] As an optional implementation, determining the stratification standard based on the stratification dimension includes:

[0105] When the stratification dimension is data sensitivity, the stratification criteria shall include at least high sensitivity and low sensitivity; or

[0106] When the stratification dimension is data access frequency, the stratification criteria include at least high-frequency access and low-frequency access; or

[0107] When the stratification dimension is the data usage scenario, the stratification criteria are determined to include at least specific business scenarios and general business scenarios.

[0108] In this embodiment, when the stratification dimension is data sensitivity, the data stratification results may include a highly sensitive data layer, a medium sensitive data layer, and a low sensitive data layer. The three sensitive data layers are described as follows:

[0109] (1) Highly sensitive data layer: contains data involving personal privacy, commercial secrets or protected by law. For data in this layer, the highest level of security measures will be adopted in the future, including strong encryption algorithms (such as AES-256), strict access control, detailed audit logs, etc.

[0110] (2) Moderately sensitive data layer: Contains routine data related to operations and business. For data in this layer, a medium-strength encryption algorithm (such as AES-192) will be used in the future, and appropriate access rights and audit policies will be set.

[0111] (3) Low-sensitivity data layer: includes logs, statistical data, etc. For the data in this layer, a lighter encryption algorithm (such as AES-128) will be used in the future, or encryption will not be performed while ensuring security, in order to improve processing efficiency.

[0112] In this embodiment, when the stratification dimension is data access frequency, the result of data stratification may include a high-frequency access data layer and a low-frequency access data layer. The two data layers are described as follows:

[0113] (1) High-frequency access data layer: contains frequently accessed data. This data is subsequently stored in high-performance storage media (such as SSDs), and the data structure and index are optimized to increase access speed.

[0114] (2) Low-frequency access data layer: contains data that is rarely accessed. This type of data is subsequently stored in lower-cost storage media (such as HDD or cold storage) to save storage costs.

[0115] In this embodiment, when the stratification dimension is the data usage scenario, the result of the data stratification may include a specific business scenario data layer and a general business scenario data layer. The two data layers are described as follows:

[0116] (1) Specific business scenario data layer: including machine learning data sets, log data, etc. According to the data characteristics of specific scenarios, optimized storage and processing strategies can be customized. For example, machine learning data sets can use distributed storage and parallel processing technology.

[0117] (2) Conventional business scenario data layer: Contains data from common applications. Standard storage and processing methods are used to ensure system stability and compatibility.

[0118] S204: Perform stratification processing on the unstructured data based on the stratification standard to obtain stratified data.

[0119] In this embodiment, data tiering is a method of classifying and hierarchically managing data of different sensitivities, access frequencies, or usage scenarios, which can achieve refined encryption strategies and storage optimization effects. Furthermore, this multi-dimensional data tiering method can enable subsequent methods to more flexibly formulate storage and encryption strategies based on the characteristics of the data.

[0120] S205: Identify data characteristics of the hierarchical data, and determine corresponding block algorithms and block sizes based on the data characteristics.

[0121] As an optional implementation, determining a corresponding block algorithm and block size based on data characteristics includes:

[0122] When the data characteristic is highly sensitive, determining the corresponding block algorithm to be a content-defined block algorithm, and determining the corresponding block size to be a small block; or

[0123] When the data characteristic is low sensitivity, the corresponding block algorithm is determined to be a preset standard block algorithm, and the corresponding block size is determined to be a large block; or

[0124] When the data feature is high-frequency access, the corresponding block size is determined to be small; or

[0125] When the data feature is low-frequency access, the corresponding block size is determined to be a large block; or

[0126] When the data characteristic is a specific business scenario, determining the corresponding block size to be a first customized size corresponding to the specific business scenario; or

[0127] When the data characteristic is a conventional business scenario, the corresponding block size is determined to be a second customized size corresponding to the business entity.

[0128] In this embodiment, after the unstructured data is layered, each layer of data will have its corresponding data characteristics, and the data characteristics reflect why the data in the data layer is divided into the data layer. Based on this, the data characteristics can be understood as the characteristics shared by all data in the data layer, such as highly sensitive data, high-frequency data access, etc.

[0129] In this embodiment, the data characteristics of the layered data can be known after the data is layered. When the data characteristics are highly sensitive, the method can divide the data into smaller blocks (for example, each block size is 4KB to 16KB) and encrypt each block separately. Use a secure chunking algorithm, such as the Content-DefinedChunking (CDC) algorithm, to ensure the reliability and consistency of the chunking process. Thereafter, the chunked data can be combined with a finer granularity and a strong encryption algorithm to ensure that even if a chunk is leaked, the attacker cannot obtain useful information.

[0130] In addition, when the data characteristics are moderately sensitive, the method can divide the data into medium-sized blocks (for example, each block size is 64KB to 256KB), thereby taking into account both security and storage efficiency. In this case, the method can use a standard block algorithm.

[0131] For data layer data with low sensitivity characteristics, this method can divide the data into larger blocks (for example, each block size is more than 1MB), thereby reducing the number of blocks and encryption overhead. According to the data type, the appropriate block size is selected. Among them, for this type of data, the blocks may not be encrypted to improve processing efficiency.

[0132] In this embodiment, when the data feature is high-frequency access, the method can divide the data into smaller blocks for fast loading and processing. At the same time, the method can optimize the data index and cache mechanism, reduce I / O operations, and improve query speed.

[0133] When the data is accessed infrequently, this method can divide the data into larger blocks to reduce management complexity. Then batch processing and asynchronous loading techniques can be used to save system resources.

[0134] In this embodiment, when the data characteristics are specific business scenarios, the method can be customized according to business needs. For example, machine learning data can be divided into blocks according to feature dimensions or samples, and efficient data storage and access solutions can be designed in combination with business logic.

[0135] When the data characteristics are regular business scenario data, this method can be divided into blocks according to business entities (such as orders, customers, products), establish indexes based on business keys, and optimize query performance.

[0136] S206: Perform block processing on the layered data based on the block algorithm and the block size to obtain block data.

[0137] In this embodiment, the method can further select appropriate block partitioning schemes for different data levels (ie, stratification results) according to the above-mentioned stratification strategy, so that the block partitioning results can meet the security requirements and performance optimization goals.

[0138] In this embodiment, the method can determine the block strategy corresponding to the hierarchical data based on the data type and data attribute, and divide the data into multiple data blocks using a specific block algorithm. Among them, the block algorithm can select a suitable data block algorithm, such as fixed size block, content defined block (CDC), rolling hash block, etc.

[0139] In this embodiment, the method can select an appropriate encryption algorithm according to the data characteristics to encrypt each block data after completing the block processing (or after the subsequent duplication checking step). Among them, the encryption algorithm can select a symmetric encryption algorithm such as AES-256, AES-192, AES-128 according to the sensitivity of the data. At this time, a secure key management system (KMS) can be used to ensure the secure storage and access control of the encryption key.

[0140] In this embodiment, during the encryption process, the generation, distribution and storage of keys require strict security measures to prevent key leakage. A hardware security module (HSM) or a key management solution combining software and hardware can be used.

[0141] S207: Perform hash calculation on the block data to obtain a block hash value.

[0142] In this embodiment, the method generates a unique identifier (hash value) for each block, so as to facilitate deduplication processing in subsequent steps and avoid the storage of redundant block data.

[0143] In this embodiment, the method uses a secure hash algorithm, such as SHA-256 or SHA-3, when performing hash calculations.

[0144] S208. Perform duplication detection on the block hash values ​​based on the global hash table, and execute step S209 or S211.

[0145] In this embodiment, the method maintains a global hash table to record the hash values ​​of all stored block data. In order to prevent security issues caused by the leakage of hash values, the method can encrypt the hash values ​​or add salt values.

[0146] In this embodiment, the method compares the newly generated hash value with the value in the hash table to determine whether there is duplicate block data.

[0147] S209: When the block hash value fails the duplication detection, a reference pointer is generated; the reference pointer points to the stored duplicate data.

[0148] In this embodiment, if duplicate block data is found, the method will no longer store a new copy, but generate a reference pointer to point to the stored data block.

[0149] S210: discard the block data and end this process.

[0150] S211. When the block hash value passes the duplication detection, the block hash value is recorded in the full set hash table.

[0151] In this embodiment, if no duplicate block data is found, the method stores the block data and adds the hash value to the hash table.

[0152] S212: Store the block data to obtain pre-processed data, and end this process.

[0153] In this embodiment, the method can use hash values ​​to perform data integrity verification during data storage and transmission, thereby ensuring that the data has not been tampered with.

[0154] In this embodiment, the method can generate a corresponding metadata list based on all stored block data to record the number, data type, sensitivity classification, encryption method, storage location and other information of each block data. Among them, the method can use a database or index file to manage the metadata of the data block to support fast query and retrieval. At the same time, the association between the block data and the original file, user permissions, etc. is established to facilitate data reorganization and access control.

[0155] In this embodiment, the method can encrypt or control access to metadata to prevent unauthorized access. At the same time, based on user identity and authority, access to metadata and data blocks is controlled to ensure data security.

[0156] In this embodiment, the execution subject of the method may be a computing device such as a computer or a server, and no limitation is made in this embodiment.

[0157] In this embodiment, the execution subject of the method may also be a smart device such as a smart phone, a tablet computer, etc., which is not limited in this embodiment.

[0158] It can be seen that the implementation of the unstructured data preprocessing method described in this embodiment can provide corresponding data preprocessing methods for different types of unstructured data such as text, pictures, and videos according to the sensitivity, access frequency, and usage scenarios of the data, so that it can meet the data security requirements of different businesses, and at the same time it is also conducive to improving the efficiency of data storage and query processing. On the other hand, the method can also be combined with data deduplication technology to significantly reduce the storage space occupied, thereby improving the storage efficiency of unstructured data and optimizing the utilization of system storage resources.

[0159] Example 3

[0160] This embodiment shows three example processes of preprocessing methods for unstructured data based on different data types.

[0161] 1. The data type targeted by the first method is the text data type, and the specific description is as follows:

[0162] First, get the sample text "User information: Name: Alice, Address: 123 Street, Transaction details: Purchased 500 items".

[0163] Secondly, the text is stratified by sensitivity to obtain:

[0164] (1) High sensitivity: user privacy information, such as name and address;

[0165] (2) Medium sensitivity: transaction information;

[0166] (3) Low sensitivity: other non-sensitive content.

[0167] Then, according to the sensitivity and type of the data, the data is divided into blocks to obtain:

[0168] (1) Block 1 (high sensitivity): “Name: Alice, Address: 123 Street”;

[0169] (2) Block 2 (medium sensitivity): “Transaction details: 500 items purchased”;

[0170] (3) Block 3 (low sensitivity): other non-sensitive content (none in this example).

[0171] Then, a hash value is calculated for each data block, for example, using the SHA-256 algorithm to perform hash calculation, and the result is:

[0172] Hash(Block1) = SHA256("Name: Alice, Address: 123 Street")

[0173] Hash(B1ock2)=SHA256("Transaction details: purchased $500 worth of merchandise")

[0174] Finally, the hash value is compared with the existing hash table in the system. If a duplication is found, it will no longer be stored and the reference relationship will be recorded.

[0175] 2. The data type targeted by the second method is the image data type, and the specific description is as follows:

[0176] An example scenario is: a user uploads an ID card photo that contains his / her face and ID card number. The image needs to be securely stored and managed while supporting efficient queries and defending against potential attacks.

[0177] At this point, the method first obtains an ID card photo that includes the individual's face and ID card number.

[0178] Then, divide the photo into the following parts:

[0179] (1) Highly sensitive data:

[0180] ①Facial area: involves personal biometric information and needs to be kept highly confidential.

[0181] ②ID number area: contains personal identity information and must be strictly protected.

[0182] (2) Low-sensitivity data: background areas and other non-sensitive information. This part of information has relatively low security requirements.

[0183] Then, according to the results of data stratification, a face detection algorithm (such as MTCNN) is used to locate the facial area in the image and obtain the coordinate information of the facial area. At the same time, optical character recognition (OCR) technology is used to locate the area where the ID card number is located and obtain its coordinate information.

[0184] At this point, the method can be obtained by block processing:

[0185] (1) Block 1 (high sensitivity): The facial area is divided into fine-grained blocks of 128 × 128 pixels to generate a series of facial data blocks {B 1，1 , B 1，2 , ..., B 1，n}.

[0186] (2) Block 2 (high sensitivity): Extract the ID card number area separately as data block B 2 , accurately protect sensitive information.

[0187] (3) Block 3 (low sensitivity): The background area is divided into blocks of 512 × 512 pixels to generate background data blocks {B 3，1 , B 3，2 , ..., B 3，m}.

[0188] Finally, for each data block B i,j Calculate the hash value H(B i,j ), for example, using the SHA-256 algorithm to calculate: H(B i,j )=SHA 256(B i,j ). At this point, the method compares the calculated hash value with the hash table stored in the system. If H(B i,j ) already exists: indicates that the data block has been stored, and the reference relationship is recorded to avoid duplicate storage; if H(B i,j ) does not exist: data block B i,j Store and update the hash table.

[0189] 3. The data type targeted by the third method is the video data type, and the specific description is as follows:

[0190] Example scenario: Company internal meeting recording videos containing sensitive business secrets need to be securely stored and managed while supporting efficient query and defending against potential attacks.

[0191] Based on the above scenario, this method first obtains the company's internal meeting record video and sets it in layers to obtain:

[0192] (1) Highly sensitive data: Discussion fragments involving commercial secrets, such as new product development and strategic planning, need to be kept highly confidential.

[0193] (2) Medium-sensitivity data: General business discussion snippets, such as daily operations and project progress reports, require appropriate protection.

[0194] (3) Low-sensitivity data: Meeting opening, breaks, and non-business-related clips have relatively low security requirements.

[0195] Secondly, the method uses video content analysis technology to identify different scenes and discussion topics in the video, determine the sensitivity level of each clip, and perform hierarchical processing to obtain:

[0196] (1) Automatic speech recognition (ASR): Convert audio content into text and analyze the sensitivity of the discussion content.

[0197] (2) Image recognition: Detect sensitive information in video images, such as confidential content on a whiteboard and confidential documents.

[0198] Then, based on the above stratification results, the data is divided into blocks to obtain:

[0199] (1) Block 1 (high sensitivity): The confidential discussion segment is divided into fine-grained blocks of 30 frames (about 1 second) to generate high-sensitivity data blocks {B 1，1 , B 1，2 , ..., B 1，p}.

[0200] (2) Block 2 (medium sensitivity): The general business discussion segment is divided into blocks of 150 frames (about 5 seconds) to generate medium sensitivity data blocks {B 2，1 , B 2，2 , ..., B 2，q}.

[0201] (c) Block 3 (low sensitivity): The non-business segment is divided into blocks every 300 frames (about 10 seconds) to generate low-sensitivity data blocks {B 3，1 , B 3，2 , ..., B 3，r}.

[0202] Finally, the method performs ai，j Extract key frames or features and then calculate the hash value H(B i，j ), for example, using the SHA-256 algorithm to calculate: H(B i，j )=SHA256(FeatureExtract(B i，j )). At the same time, the calculated hash value is compared with the hash table stored in the system. If H(B i，j ) already exists: indicates that the data block has been stored, and the reference relationship is recorded to avoid duplicate storage; if H(B i，j ) does not exist: data block B i，j Store and update the hash table.

[0203] It can be seen that the implementation of the unstructured data preprocessing method described in this embodiment can provide corresponding data preprocessing methods for different types of unstructured data such as text, pictures, and videos according to the sensitivity, access frequency, and usage scenarios of the data, so that it can meet the data security requirements of different businesses, and at the same time it is also conducive to improving the efficiency of data storage and query processing. On the other hand, the method can also be combined with data deduplication technology to significantly reduce the storage space occupied, thereby improving the storage efficiency of unstructured data and optimizing the utilization of system storage resources.

[0204] Example 4

[0205] Please see Figure 4 , Figure 4 The following is a schematic diagram of the structure of a preprocessing device for unstructured data provided in this embodiment. Figure 4 As shown, the unstructured data preprocessing device includes:

[0206] A first identification unit 310 is used to identify the hierarchical dimensions of the unstructured data and determine the hierarchical standards based on the hierarchical dimensions;

[0207] A stratification unit 320, configured to perform stratification processing on the unstructured data based on a stratification standard to obtain stratified data;

[0208] A second identification unit 330, used to identify data characteristics of the hierarchical data, and determine a corresponding block algorithm and block size based on the data characteristics;

[0209] A block division unit 340 is used to perform block processing on the layered data based on a block division algorithm and a block size to obtain block data;

[0210] The deduplication unit 350 is used to perform deduplication processing on the block data to obtain pre-processed data.

[0211] In this embodiment, the explanation of the preprocessing device for unstructured data can refer to the description in Embodiment 1 or Embodiment 2, and will not be further elaborated in this embodiment.

[0212] It can be seen that the implementation of the unstructured data preprocessing device described in this embodiment can provide corresponding data preprocessing methods for different types of unstructured data such as text, pictures, and videos according to the sensitivity, access frequency, and usage scenarios of the data, so that it can meet the data security requirements of different businesses, and at the same time it is also conducive to improving the efficiency of data storage and query processing. On the other hand, this method can also be combined with data deduplication technology to significantly reduce the storage space occupied, thereby improving the storage efficiency of unstructured data and optimizing the utilization of system storage resources.

[0213] Example 5

[0214] Please see Figure 5 , Figure 5 The following is a schematic diagram of the structure of a preprocessing device for unstructured data provided in this embodiment. Figure 5 As shown, the unstructured data preprocessing device includes:

[0215] A first identification unit 310 is used to identify the hierarchical dimensions of the unstructured data and determine the hierarchical standards based on the hierarchical dimensions;

[0216] A stratification unit 320, configured to perform stratification processing on the unstructured data based on a stratification standard to obtain stratified data;

[0217] A second identification unit 330, used to identify data characteristics of the hierarchical data, and determine a corresponding block algorithm and block size based on the data characteristics;

[0218] A block division unit 340 is used to perform block processing on the layered data based on a block division algorithm and a block size to obtain block data;

[0219] The deduplication unit 350 is used to perform deduplication processing on the block data to obtain pre-processed data.

[0220] As an optional implementation manner, the first identification unit 310 includes:

[0221] The identification subunit 311 is used to identify the data type and data attribute of the unstructured data through a preset data identification model or a preset data identification rule;

[0222] The determination subunit 312 is used to determine the hierarchical dimensions of the unstructured data based on the data type and data attribute; the hierarchical dimensions include at least one of data sensitivity, data access frequency, and data usage scenario.

[0223] As an optional implementation, the first identification unit 310 is specifically configured to, when the stratification dimension is data sensitivity, determine that the stratification standard at least includes high sensitivity and low sensitivity; or

[0224] When the stratification dimension is data access frequency, the stratification criteria include at least high-frequency access and low-frequency access; or

[0225] When the stratification dimension is the data usage scenario, the stratification criteria are determined to include at least specific business scenarios and general business scenarios.

[0226] As an optional implementation, the second identification unit 330 is specifically configured to, when the data characteristic is highly sensitive, determine that the corresponding block algorithm is a content-defined block algorithm, and determine that the corresponding block size is a small block; or

[0227] When the data characteristic is low sensitivity, the corresponding block algorithm is determined to be a preset standard block algorithm, and the corresponding block size is determined to be a large block; or

[0228] When the data feature is high-frequency access, the corresponding block size is determined to be small; or

[0229] When the data feature is low-frequency access, the corresponding block size is determined to be a large block; or

[0230] When the data characteristic is a specific business scenario, determining the corresponding block size to be a first customized size corresponding to the specific business scenario; or

[0231] When the data characteristic is a conventional business scenario, the corresponding block size is determined to be a second customized size corresponding to the business entity.

[0232] As an optional implementation, the deduplication unit 350 includes:

[0233] The calculation subunit 351 is used to perform hash calculation on the block data to obtain the block hash value;

[0234] The duplicate checking subunit 352 is used to perform duplicate detection on the block hash values ​​based on the global hash table;

[0235] The generating subunit 353 is used to generate a reference pointer when the block hash value fails the duplication detection; the reference pointer points to the stored duplicate data;

[0236] The discarding subunit 354 is used to discard the block data.

[0237] As an optional implementation, the deduplication unit 350 further includes:

[0238] A recording subunit 355, configured to record the block hash value into the full set hash table when the block hash value passes the duplication detection;

[0239] The storage subunit 356 is used to store the block data to obtain pre-processed data.

[0240] As an optional implementation, the deduplication unit 350 is specifically used to perform deduplication processing on the block data to obtain a plurality of block non-duplicate data; and classify the plurality of block non-duplicate data to obtain pre-processed data.

[0241] In this embodiment, the explanation of the preprocessing device for unstructured data can refer to the description in Embodiment 1 or Embodiment 2, and will not be further elaborated in this embodiment.

[0242] It can be seen that the implementation of the unstructured data preprocessing device described in this embodiment can provide corresponding data preprocessing methods for different types of unstructured data such as text, pictures, and videos according to the sensitivity, access frequency, and usage scenarios of the data, so that it can meet the data security requirements of different businesses, and at the same time it is also conducive to improving the efficiency of data storage and query processing. On the other hand, this method can also be combined with data deduplication technology to significantly reduce the storage space occupied, thereby improving the storage efficiency of unstructured data and optimizing the utilization of system storage resources.

[0243] An embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to perform the unstructured data preprocessing method in embodiment 1 or embodiment 2 of the present application.

[0244] An embodiment of the present application provides a computer-readable storage medium storing computer program instructions. When the computer program instructions are read and executed by a processor, the unstructured data preprocessing method in Embodiment 1 or Embodiment 2 of the present application is executed.

[0245] An embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the unstructured data preprocessing method in Embodiment 1 or Embodiment 2 of the present application is executed.

[0246] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0247] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0248] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0249] The above description is only an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0250] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0251] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

Claims

1. A method for preprocessing unstructured data, characterized in that: include: Identifying hierarchical dimensions of unstructured data and determining hierarchical criteria based on the hierarchical dimensions; Performing hierarchical processing on the unstructured data based on the hierarchical standard to obtain hierarchical data; Identifying data characteristics of the hierarchical data, and determining corresponding block algorithms and block sizes based on the data characteristics; Performing block processing on the hierarchical data based on the block algorithm and the block size to obtain block data; De-duplication processing is performed on the block data to obtain pre-processed data.

2. The method for preprocessing unstructured data according to claim 1, characterized in that: The identification of the hierarchical dimensions of unstructured data includes: Identify the data type and data attributes of unstructured data through a preset data identification model or preset data identification rules; Based on the data type and the data attribute, determine the hierarchical dimension of the unstructured data; the hierarchical dimension includes at least one of data sensitivity, data access frequency, and data usage scenario.

3. The method for preprocessing unstructured data according to claim 1, characterized in that: The determining of the stratification standard based on the stratification dimension includes: When the stratification dimension is data sensitivity, determining the stratification criteria includes at least high sensitivity and low sensitivity; or When the stratification dimension is data access frequency, determining the stratification criteria includes at least high-frequency access and low-frequency access; or When the stratification dimension is a data usage scenario, the stratification standard is determined to include at least a specific business scenario and a general business scenario.

4. The method for preprocessing unstructured data according to claim 1, characterized in that: The determining the corresponding block algorithm and block size based on the data characteristics includes: When the data characteristic is highly sensitive, determining the corresponding block algorithm to be a block algorithm based on content definition, and determining the corresponding block size to be a small block; or When the data characteristic is low sensitivity, determining the corresponding block algorithm as a preset standard block algorithm, and determining the corresponding block size as a large block; or When the data characteristic is high-frequency access, determining the corresponding block size to be small; or When the data characteristic is low-frequency access, determining the corresponding block size to be a large block; or When the data characteristic is a specific business scenario, determining the corresponding block size to be a first customized size corresponding to the specific business scenario; or When the data characteristic is a conventional business scenario, the corresponding block size is determined to be a second customized size corresponding to the business entity.

5. The method for preprocessing unstructured data according to claim 1, characterized in that: The performing deduplication processing on the block data to obtain preprocessed data includes: Performing hash calculation on the block data to obtain a block hash value; Performing duplication detection on the block hash values ​​based on a global hash table; When the block hash value fails the duplication detection, a reference pointer is generated; the reference pointer points to the stored duplicate data; The block data is discarded.

6. The method for preprocessing unstructured data according to claim 1, characterized in that: The method comprises: When the block hash value passes the duplication detection, recording the block hash value into the full set hash table; The block data are stored to obtain preprocessed data.

7. The method for preprocessing unstructured data according to claim 1, characterized in that: The performing deduplication processing on the block data to obtain preprocessed data includes: Deduplication processing is performed on the block data to obtain a plurality of block non-duplication data; The plurality of block non-duplicate data are classified to obtain pre-processed data.

8. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to perform the unstructured data preprocessing method according to any one of claims 1 to 7.

9. A readable storage medium, characterized in that: The readable storage medium stores computer program instructions, and when the computer program instructions are read and executed by a processor, the method for preprocessing unstructured data according to any one of claims 1 to 7 is executed.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the computer program executes the method for preprocessing unstructured data according to any one of claims 1 to 7.

Images