A method for storing, migrating and identifying unstructured data

Abstract

The application belongs to the technical field of unstructured data processing, in particular to a kind of unstructured data storage, migration and identification method, the method comprises the following steps: S1: obtaining unstructured data, establishing index label for unstructured data, and saving index label to boot partition;S2: the unstructured data that has established index label is analyzed, same part and different part in unstructured data are determined and split, and the unstructured data after splitting is saved in storage area;S3: the same part of unstructured data after splitting is saved in redundancy partition in storage area, and the different part of unstructured data after splitting is saved in total storage area in storage area;The application can classify and store after analyzing unstructured data, improve the storage compression efficiency of data, and facilitate data identification and migration.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of unstructured data processing, and particularly relates to a storage, migration and identification method of unstructured data. BACKGROUND

[0002] In the information socialization era, various industries have accumulated a large amount of data information in the process of processing related businesses. With the popularization and development of IT application, the traditional paper storage mode is continuously reduced, and more electronic information storage mode is stored in the computer. For unstructured data storage, similar to pictures, images and videos, the characteristics of various formats of unstructured data make it inconvenient to use a two-dimensional table structure to express data compression storage.

[0003] The existing unstructured data is generally stored in the storage in sequence, so that there is no relationship between the data, which makes the identification and retrieval time long, difficult to use, and the unstructured data has a large amount of redundant information, which causes a large waste of storage space when storing the data information, and further leads to low data storage compression efficiency. SUMMARY

[0004] In order to make up for the deficiencies of the prior art, the unstructured data is analyzed, classified and stored, the data storage compression efficiency is improved, and the data identification and migration are facilitated, and the application provides a storage, migration and identification method of unstructured data.

[0005] The technical scheme adopted by the application to solve the technical problem is that: the application provides a storage, migration and identification method of unstructured data, and the method comprises the following steps:

[0006] S1: obtaining unstructured data, establishing an index tag for the unstructured data, and saving the index tag to a boot partition;

[0007] S2: analyzing the unstructured data which has established the index tag, determining and splitting the same part and the different part in the unstructured data, and saving the split unstructured data in a storage area;

[0008] S3: saving the same part of the split unstructured data in a redundant partition in the storage area, and saving the different part of the split unstructured data in a total storage area in the storage area;

[0009] The redundant partition stores one copy of the same part in different unstructured data, the redundant partition stores a plurality of groups of different source same part data, and the groups of different source same part data are respectively provided with independent labels, and the redundant partition in the storage area has a backup.

[0010] Preferably, the index tag includes verification information for checking the integrity of unstructured data. The verification information includes, but is not limited to, the MD5 value, SHA1 value, and CRC32 value of the unstructured data. The unstructured data needs to be verified by the verification information recorded in the index tag when it is stored, viewed, or used.

[0011] Preferably, the index labels established for unstructured data in step S1 include type information, and the redundant partition and the total storage area are divided into sub-partitions corresponding to different types of unstructured data. The data blocks after unstructured data analysis and splitting are still saved to the corresponding sub-partitions according to the type information recorded in the index labels.

[0012] Preferably, the index label includes a main label and sub-labels. The main label stores the index labels of all currently stored unstructured data. The sub-labels are generated according to fixed time intervals and store the index labels of unstructured data stored within the corresponding time period.

[0013] Preferably, during the migration of unstructured data, the index tags, data in the redundant partition, and data in the total storage area are migrated sequentially. The data blocks in the redundant partition and the total storage area are assigned migration numbers before migration, and the migration number data is migrated after the index tags.

[0014] Preferably, during the unstructured data migration process, the migrated unstructured data is verified at fixed intervals.

[0015] Preferably, the unstructured data is divided into hot data and ordinary data based on the number of times it is used and the time it is used, and the hot data is transmitted first during the migration process.

[0016] Preferably, the hotspot data is stored in the same storage device or storage area after migration.

[0017] The beneficial effects of this invention are as follows:

[0018] 1. The present invention provides a method for storing, migrating, and identifying unstructured data. By analyzing and splitting the unstructured data to be stored, the method selects and saves the common parts in different unstructured data, effectively reducing the redundant parts in the stored unstructured data, reducing the size of the stored data, improving the storage compression efficiency of unstructured data, and reducing the waste of storage space.

[0019] 2. The method for storing, migrating and identifying unstructured data described in this invention enables index labels to be compared with each other by setting a main index and sub-indexes, reducing the situation where stored data cannot be used due to errors in index labels. At the same time, the sub-indexes improve the speed of retrieval and use of stored data. Attached Figure Description

[0020] The invention will now be further described with reference to the accompanying drawings.

[0021] Figure 1 This is a flowchart of the identification method of the present invention. Detailed Implementation

[0022] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments.

[0023] like Figure 1 As shown, the present invention provides a method for storing, migrating, and identifying unstructured data, the method comprising the following steps:

[0024] S1: Obtain unstructured data, create index labels for the unstructured data, and save the index labels to the boot partition;

[0025] S2: Analyze the unstructured data that has been indexed and labeled, identify and split the identical and different parts of the unstructured data, and save the split unstructured data in the storage area;

[0026] S3: After splitting unstructured data, the identical parts are stored in redundant partitions in the storage area, and the different parts of the unstructured data are stored in the main storage area in the storage area.

[0027] The redundant partition stores a copy of the same part in different unstructured data. The multiple sets of identical data from different sources stored in the redundant partition are each assigned an independent label. The redundant partition in the storage area has a backup.

[0028] When storing unstructured data, index tags are created on the acquired unstructured data to facilitate searching, reading, and migrating of the stored unstructured data. This prevents the stored unstructured data from becoming disorganized, affecting storage compression efficiency and making it difficult to access and use. Simultaneously, by analyzing and splitting the indexed unstructured data, the identical and different parts of the split data blocks are identified. Based on these differences, identical parts are randomly selected and stored in a redundant partition, while all different parts are stored in the main storage area. This effectively reduces redundancy in the stored unstructured data, shrinks the overall size of the stored unstructured data, and improves the efficiency of unstructured data storage. This system improves storage compression efficiency, reduces storage space waste, and facilitates the identification, retrieval, and migration of unstructured data. When unstructured data needs to be used, it retrieves the corresponding identical and different parts of the unstructured data from the redundant partition and the main storage area based on index tags. These identical and different parts are then combined according to the order recorded in the index tags to obtain the initial unstructured data. Simultaneously, the tag information corresponding to the identical parts of the unstructured data stored in the redundant partition is also recorded in the index tags. This avoids data retrieval errors that could occur when retrieving the corresponding identical parts from the redundant partition due to multiple sets of identical data stored there, thus preventing disruption to the normal use of the unstructured data.

[0029] As one embodiment of the present invention, the index tag includes verification information for checking the integrity of unstructured data. The verification information includes, but is not limited to, the MD5 value, SHA1 value, and CRC32 value of the unstructured data. The unstructured data needs to be verified by the verification information recorded in the index tag when it is stored, viewed, or used.

[0030] When creating index tags for unstructured data, the verification information of the unstructured data is calculated simultaneously and then attached to the index tags. Furthermore, when using unstructured data, the same and different parts of data retrieved from redundant partitions and the main storage area are combined and verified using the verification information recorded in the index tags. This ensures that the unstructured data to be retrieved is complete and error-free, preventing errors from occurring when retrieving data blocks from redundant partitions and the main storage area or when combining data blocks. Errors could lead to changes in the unstructured data, making it impossible to restore its initial state and thus affecting its normal use.

[0031] In one embodiment of the present invention, the index label established for unstructured data in step S1 includes type information, the redundant partition and the total storage area are divided into sub-partitions corresponding to different types of unstructured data, and the data blocks after unstructured data analysis and splitting are still saved to the corresponding sub-partitions according to the type information recorded in the index label.

[0032] When creating index tags for unstructured data, the type of unstructured data, such as images, audio, video, or text, is appended to the index tags to indicate its type. Afterward, the unstructured data is analyzed and split to obtain data blocks. Based on the type information recorded in the index tags, these data blocks are stored in redundant partitions or corresponding sub-partitions within the main storage area. This ensures that similar types of unstructured data are stored in the same partition, thereby improving the retrieval and reading speeds of the stored unstructured data, guaranteeing its usability, and accelerating response times when using it.

[0033] In one embodiment of the present invention, the index tag includes a main tag and a sub-tag. The main tag stores the index tags of all currently stored unstructured data. The sub-tags are generated according to a fixed time interval and store the index tags of the unstructured data stored in the corresponding time period.

[0034] By setting main and sub-labels, the index labels created for unstructured data can be backed up and compared with each other. This prevents the unstructured data from becoming disordered or lost due to interference with the index labels, thus affecting the normal use of the unstructured data. At the same time, by creating sub-labels at fixed time intervals, the index labels are first searched from the sub-labels when using unstructured data, which effectively reduces the size of the index labels and avoids the problem of increased retrieval time and slower retrieval speed caused by directly searching for index labels from the main label. In addition, the sub-labels are created in chronological order, which also facilitates the search and location of index labels, thereby speeding up the retrieval and use of unstructured data.

[0035] In one embodiment of the present invention, during the migration of unstructured data, the index tags, data in the redundant partition, and data in the total storage area are migrated sequentially. The data blocks in the redundant partition and the total storage area are numbered before migration, and the migration number data is migrated after the index tags.

[0036] When migrating unstructured data, data blocks in redundant partitions and the main storage area are numbered. This allows for the determination of the number of transmitted and untransmitted data blocks during the migration process. Furthermore, because each data block is numbered, even if interference occurs during the migration process leading to data block loss, the location of the lost data block can still be determined by its migration number, allowing for retransmission. This ensures that the migration process can be interrupted and resumed at any time, enabling the migration to complete normally. Additionally, by numbering data blocks during transmission, if a data block is lost, its location can be determined directly by the migration number without having to search the index tags from scratch. This reduces the time and computational resources spent searching the index tags, accelerating the migration of unstructured data and ensuring the integrity of the migrated data.

[0037] As one embodiment of the present invention, the migrated unstructured data is verified at fixed intervals during the unstructured data migration process;

[0038] Because interference during data migration is uncontrollable, data block loss can occur at any point in the transmission process. By verifying the migrated data at fixed intervals using verification information and migration numbers during the migration process, lost or erroneous data blocks in the transmitted unstructured data can be detected in a timely manner, allowing for retransmission. This ensures the integrity of the migrated unstructured data, quickly corrects errors that occur during the data migration process, and improves the efficiency of data migration.

[0039] In one embodiment of the present invention, the unstructured data is divided into hot data and ordinary data according to the number of times it is used and the time it is used, and the hot data is transmitted first during the migration process;

[0040] Hot data and ordinary data are determined by the frequency and duration of use of stored unstructured data in daily use. Hot data, which is used more frequently and for longer periods, is relatively important to users. Therefore, when migrating unstructured data, hot data is prioritized for migration, allowing it to be migrated to the target storage area earlier than ordinary data. This reduces the time hot data is occupied during the migration process, ensuring that users can use hot data normally. At the same time, since transmission links may fluctuate over time, prioritizing the migration of hot data also reduces the possibility of damage or loss after migration, ensuring normal use of stored data by users.

[0041] In one embodiment of the present invention, the hotspot data is stored in the same storage device or storage area after migration;

[0042] After migrating unstructured data, hot data is saved to the same storage device or storage area, changing the storage location of the corresponding unstructured data. This helps to improve the access speed of the migrated hot data, reduce the latency when users use hot data, and improve the user experience.

[0043] The specific workflow is as follows:

[0044] When storing unstructured data, index tags are created on the acquired unstructured data to facilitate searching, reading, and migrating of the stored unstructured data. This prevents the stored unstructured data from becoming disorganized, affecting storage compression efficiency and making it difficult to access and use. Simultaneously, by analyzing and splitting the indexed unstructured data, the identical and different parts of the split data blocks are identified. Based on these differences, identical parts are randomly selected and stored in a redundant partition, while all different parts are stored in the main storage area. This effectively reduces redundancy in the stored unstructured data, shrinks the overall size of the stored unstructured data, and improves the efficiency of unstructured data storage. This system improves storage compression efficiency, reduces storage space waste, and facilitates the identification, retrieval, and migration of unstructured data. Furthermore, when unstructured data needs to be used, it retrieves the corresponding identical and different parts of the unstructured data from the redundant partition and the main storage area based on the index tags. These identical and different parts are then combined according to the order recorded in the index tags to obtain the initial unstructured data. Simultaneously, the tag information corresponding to the identical parts of the unstructured data stored in the redundant partition is also recorded in the index tags. This avoids data retrieval errors that occur when retrieving the corresponding identical parts from the redundant partition due to multiple sets of identical data stored there, thus preventing disruption to the normal use of unstructured data.

[0045] When creating index tags for unstructured data, the verification information of the unstructured data is calculated simultaneously and then attached to the index tags. At the same time, when using unstructured data, the same and different parts of data retrieved from redundant partitions and the main storage area are combined and verified using the verification information recorded in the index tags. This ensures that the unstructured data to be retrieved and used is complete and error-free, and avoids errors that occur when retrieving data blocks from redundant partitions and the main storage area or when combining data blocks, which could cause changes to the unstructured data, making it impossible to restore the initial state of the unstructured data and thus affecting the normal use of the unstructured data.

[0046] When creating index tags for unstructured data, the type information of the unstructured data, such as images, audio, video, and text, is appended to the index tags. Then, the unstructured data is analyzed and split to obtain data blocks. Based on the type information recorded in the index tags, the data blocks are stored in redundant partitions or corresponding sub-partitions within the main storage area. This ensures that unstructured data of the same type are stored in the same partition, thereby improving the retrieval and reading speed of the stored unstructured data, ensuring the usability of unstructured data, and accelerating the response speed when using unstructured data.

[0047] By setting main tags and sub-tags, the index tags created for unstructured data can be backed up and compared with each other. This prevents the stored unstructured data from becoming disordered or lost due to interference with the index tags, thus affecting the normal use of the unstructured data. At the same time, by creating sub-tags at fixed time intervals, the index tags are first searched from the sub-tags when using unstructured data, which effectively reduces the size of the index tags. This avoids the problem of increased retrieval time and slower retrieval speed caused by directly searching for index tags from the main tags. In addition, the sub-tags are created in chronological order, which also facilitates the search and location of index tags, thereby speeding up the retrieval and use of unstructured data.

[0048] When migrating unstructured data, data blocks in redundant partitions and the total storage area are numbered. This allows for the determination of the number of transmitted and untransmitted data blocks during the migration process. Furthermore, since each data block is numbered during migration, even if data blocks are lost due to interference, their migration numbers can still be used to locate the lost blocks and facilitate retransmission. This ensures that the migration process can be interrupted and resumed at any time, allowing the migration to complete normally. Additionally, by numbering data blocks during migration, if a data block is lost during transmission, there is no need to search the index tags from scratch; the location of the lost data block can be determined by the migration number. This reduces the time and computational resources spent searching the index tags, speeds up the migration of unstructured data, and ensures the integrity of the migrated data.

[0049] Because interference during data migration is uncontrollable, data block loss can occur at any point in the transmission process. By verifying the migrated data at fixed intervals using verification information and migration number during the migration process, lost or erroneous data blocks in the transmitted unstructured data can be detected in a timely manner, allowing for retransmission, ensuring the integrity of the migrated unstructured data, and quickly correcting errors that occur during the data migration process, thereby improving the efficiency of data migration.

[0050] Hot data and ordinary data are determined by the frequency and duration of use of stored unstructured data in daily use. Hot data is used more frequently and for longer periods, making it relatively important to users. Therefore, when migrating unstructured data, hot data is prioritized for migration, allowing it to be migrated to the target storage area earlier than ordinary data. This reduces the time hot data is occupied during the migration process, ensuring that users can use hot data normally. At the same time, since transmission links may fluctuate over time, prioritizing the migration of hot data also reduces the possibility of damage or loss after migration, ensuring that users can use the stored data normally.

[0051] After migrating unstructured data, hot data is saved to the same storage device or storage area, changing the storage location of the corresponding unstructured data. This helps to improve the access speed of the migrated hot data, reduce the latency when users use hot data, and improve the user experience.

[0052] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for storing, migrating, and identifying unstructured data, characterized in that: The method includes the following steps: S1: Obtain unstructured data, create index labels for the unstructured data, and save the index labels to the boot partition; S2: Analyze the unstructured data that has been indexed and labeled, identify and split the identical and different parts of the unstructured data, and save the split unstructured data in the storage area; S3: After splitting unstructured data, the identical parts are stored in redundant partitions in the storage area, and the different parts are stored in the main storage area in the storage area. The redundant partition stores a copy of the same part in different unstructured data. The multiple sets of identical data from different sources stored in the redundant partition are each assigned an independent label. The redundant partition in the storage area has a backup. The index tags include a main tag and sub tags. The main tag stores the index tags of all currently stored unstructured data. The sub tags are generated according to a fixed time interval and store the index tags of the unstructured data stored in the corresponding time period. During the unstructured data migration process, the migrated unstructured data is verified at fixed intervals. The unstructured data is divided into hot data and ordinary data based on the number of times it is used and the time it is used. The hot data is transmitted first during the migration process.

2. The method for storing, migrating, and identifying unstructured data according to claim 1, characterized in that: The index tag includes verification information for checking the integrity of unstructured data. The verification information includes, but is not limited to, the MD5 value, SHA1 value, and CRC32 value of the unstructured data. The unstructured data needs to be verified against the verification information recorded in the index tag when it is stored, viewed, or used.

3. The method for storing, migrating, and identifying unstructured data according to claim 1, characterized in that: The index labels created for unstructured data in step S1 include type information. The redundant partition and the total storage area are divided into sub-partitions corresponding to different types of unstructured data. The data blocks after unstructured data analysis and splitting are still saved to the corresponding sub-partitions based on the type information recorded in the index labels.

4. The method for storing, migrating, and identifying unstructured data according to claim 2, characterized in that: During the migration of unstructured data, the index tags, data in the redundant partition, and data in the total storage area are migrated sequentially. Before the migration, the data blocks in the redundant partition and the total storage area are assigned migration numbers, and the migration numbers are migrated after the index tags.

5. The method for storing, migrating, and identifying unstructured data according to claim 4, characterized in that: The hot data is migrated and stored in the same storage device or storage area.

Images