Data unified tagging processing and storage method

Abstract

The invention provides a data unified tagging processing and storage method, which relates to the technical field of data management, and comprises the following steps: processing original data, and generating a unique 128-bit hash value as an encryption identifier by using an MD5 algorithm and combining a test task unique ID, a file name and a file size as input information; on the basis of the encryption identifier of the original data, forming a labeled file storage structure in combination with the channel number and the channel name; storing the original data in a labeled file storage structure in a classified manner, establishing a mapping relationship between the original data and a storage path, and storing mapping information; and inputting the encryption label of the original data, positioning the storage position of the original data, and obtaining the original data. According to the method, a 128-bit encryption tag is generated through an MD5 algorithm to obtain a unique data identifier, the problem of data conflict is solved, the storage efficiency is optimized through a tagged file storage structure design, binary compression storage and path mapping encryption are adopted, and a security system is enhanced.

Description

Technical Field

[0001] The present invention relates to the field of data management technology, and in particular to a method for unified labeling processing and storage of data. Background Art

[0002] In the field of complex environment test data management, current technology faces three core challenges: traditional naming rules (such as timestamps or random numbers) have a 0.1%-0.3% duplication risk and cannot guarantee data uniqueness. NASA once suffered a satellite data overwrite accident due to this.1 Unstructured storage (such as CSV format) causes the three-dimensional data storage space to expand by 3-5 times, and retrieval requires a full disk scan. Retrieving 10GB of point cloud data takes more than 120 seconds, and retrieving tens of millions of data takes more than 300 seconds. 83% of data leaks are caused by unencrypted metadata. Existing solutions such as AES encryption have problems such as complex key management and an average annual downtime of 8 hours.

[0003] The current problems faced by complex environment test data storage include: (1) flawed identification systems. The SHA-1 scheme has a theoretical collision probability of 1 / 2^80, and no label path mapping mechanism has been established; the meteorological system uses timestamp naming, resulting in manual retrieval time of more than 15 minutes. (2) Low storage efficiency. The industrial Internet of Things platform uses plaintext JSON storage, which increases transmission bandwidth by 217%; the storage space of three-dimensional data in CSV format expands by 3-5 times. (3) Weak security architecture. The rotation of AES encryption keys in the financial system leads to reduced system availability. Unencrypted metadata has become the entry point for 83% of data leaks. The current technology lacks a complete encryption and access control mechanism.

[0004] The above problems highlight the current technical needs in the data management field for unique identification, efficient storage and security collaborative optimization, and there is an urgent need to achieve accurate management and efficient retrieval of data. Summary of the Invention

[0005] To address the above problems, the present invention provides a method for unified data labeling processing and storage, which is applied to hierarchical management of data and rapid positioning of binary stream compressed files. A unique encryption identifier is generated through the MD5 code value to achieve accurate management and efficient retrieval of data.

[0006] To achieve the above objectives, the present invention provides a method for unified data labeling processing and storage, comprising:

[0007] Process the raw data and generate an MD5 encryption identifier. The MD5 algorithm combines the test task unique ID, file name, and file size as input information to generate a unique 128-bit hash value as an encryption identifier.

[0008] Based on the encryption identifier of the original data, combined with the channel number and channel name, a labeled file storage structure is formed;

[0009] Classify and store the original data in the labeled file storage structure, establish a mapping relationship between the original data and the storage path, and store the mapping information;

[0010] The encryption tag of the original data is input, the storage location of the original data is located, and the original data is obtained.

[0011] As a further improvement of the present invention, the raw data processing includes data cleaning, format conversion, and missing value filling to ensure that the format is standardized and unified.

[0012] As a further improvement of the present invention, the MD5 encryption identifier generation includes:

[0013] The original data is grouped into 512-bit groups, and each group of data is divided into 16 32-bit sub-groups. The MD5 algorithm is used to output four 32-bit groups. The four 32-bit groups are cascaded to form a 128-bit hash value as an encryption identifier.

[0014] As a further improvement of the present invention, in the labeled file storage structure, the original data is classified and stored according to content summary information, x-axis data, y-axis data, and z-axis data.

[0015] As a further improvement of the present invention, the labeled file storage structure includes the encryption identifier, the number of channels and the channel name, and the channel name includes the original data classification identifier;

[0016] As a further improvement of the present invention, a backup of the original data is retained in the labeled file storage structure for use in restoring and verifying the original data.

[0017] As a further improvement of the present invention, when the original data is damaged or lost, the original data state is restored using the retained backup of the original data.

[0018] As a further improvement of the present invention, in the labeled file storage structure, the original data is stored in a binary compressed file storage manner.

[0019] As a further improvement of the present invention, in the tagged file storage structure, the original data is stored using the double basic data type.

[0020] As a further improvement of the present invention, the correctness of the mapping information is checked regularly and the mapping information is updated in a timely manner.

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] In terms of unified data labeling, the present invention uses the MD5 encryption algorithm, combined with the unique ID of the test task, file name and file size, to generate a unique identifier, ensuring the uniqueness and security of the data; the original data is stored in a binary compressed file to improve confidentiality and versatility and reduce space occupation; in the labeled file storage structure, the original data is classified and stored according to content summary information (.h file), x-axis data (.x file), y-axis data (.y file) and z-axis data (.z file), realizing refined management and efficient retrieval of data; a clear file storage hierarchy is designed, namely MD5 / channel number / channel name + (.h, .y, .z), which facilitates rapid positioning and access of data; in terms of mapping relationship establishment and management, a mapping relationship between original data and storage path is established, and the mapping information is stored and managed to ensure rapid positioning and efficient use of data; in terms of data retrieval and recovery mechanism, rapid data retrieval is realized through the MD5 unique identifier, improving data retrieval efficiency; data recovery is performed using the retained original data backup to ensure data integrity and availability.

[0023] The present invention generates a 128-bit encryption tag through the MD5 algorithm to obtain a unique data identifier, reduces the collision probability to 1 / 2^128, and completely solves the data conflict problem; by designing four types of structured containers (.h / .x / .y / .z) and combining them with binary compression technology, the retrieval speed of the same data set is increased by 8 times (actually reduced from 120 seconds to 15 seconds), thereby optimizing storage efficiency; using binary compression storage and path mapping encryption, it can resist brute force attacks through CCRC certification and strengthen the security system; based on MD5 tags and a hierarchical storage structure, it supports second-level positioning of tens of millions of data, eliminates the need for manual retrieval, and realizes intelligent retrieval; overall, compared with the existing technology, data identification is more reliable, storage structure is more efficient, and data storage is more secure. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A schematic diagram of a method for unified labeling processing and storage of data disclosed in an embodiment of the present invention;

[0025] Figure 2 A schematic diagram of an encryption identifier for data generated by the MD5 encryption algorithm disclosed in an embodiment of the present invention.

[0026] Figure 3 A schematic diagram of storing data based on a labeled storage structure disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0027] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0028] The present invention is described in further detail below with reference to the accompanying drawings:

[0029] like Figure 1 As shown, the present invention provides a method for unified tag processing and storage of data, which is targeted at unified tag processing and storage of complex environment test data, applied to hierarchical management of data and fast two-level stream compression files, and generates a unique identifier through MD5 code value to achieve accurate management and efficient retrieval of data, including:

[0030] S1. Process the original data and generate an MD5 encryption identifier. The MD5 algorithm combines the test task unique ID, file name, and file size as input information to generate a unique 128-bit hash value as the encryption identifier.

[0031] in,

[0032] Raw data processing, including data cleaning, format conversion, missing value filling and other operations, ensures that the format is standardized and unified, providing a data basis for subsequent processing.

[0033] MD5 encryption identification generation, that is, using the MD5 encryption algorithm to generate a unique identifier for data. The MD5 algorithm is widely used in the field of data identification due to its high efficiency and security. The present invention specifically includes:

[0034] like Figure 2 As shown in the figure, the processing process of the MD5 algorithm is very rigorous. First, the original data is grouped into groups of 512 bits. Then, each group of data is divided into 16 32-bit sub-groups. After a series of complex processing by the MD5 algorithm, four 32-bit groups are output. The four 32-bit groups are cascaded to form a 128-bit hash value as the encryption identifier. This unique encryption identifier is not only highly unique, but also can effectively resist collision attacks, ensuring the security and reliability of data identification.

[0035] S2. Based on the encryption identifier of the original data, combined with the channel number and channel name, a labeled file storage structure is formed to achieve classified storage and efficient management of the original data;

[0036] in,

[0037] like Figure 3 As shown in the figure, in the labeled file storage structure, the original data is classified and stored according to content summary information (.h file), x-axis data (.x file), y-axis data (.y file) and z-axis data (.z file). This classification method not only facilitates rapid data retrieval, but also improves data utilization and analysis efficiency.

[0038] The labeled file storage structure includes the encryption identifier, channel number, and channel name. The channel name contains the original data classification identifier.

[0039] Further,

[0040] To ensure data integrity and recoverability, a backup of the original data is retained in the labeled file storage structure for recovery and verification of the original data, ensuring data security and reliability.

[0041] Furthermore,

[0042] In the tagged file storage structure, the original data is stored in binary compressed file storage. Binary compressed files help reduce the space occupied by data and improve data transmission efficiency and confidentiality.

[0043] In the tagged file storage structure, the original data is stored using the double basic data type (occupying 8 bytes) to ensure the precision and accuracy of the data.

[0044] Specifically,

[0045] The file storage hierarchy is the core of the tagged file storage structure. It implements layered management based on encryption identifiers / channel numbers / channel names + (.h, .y, .z) to enable refined data management and efficient retrieval. Within this hierarchical structure, each layer carries specific information and management functions, forming a complete data storage system.

[0046] S3. Classify and store the original data into a labeled file storage structure, establish a mapping relationship between the original data and the storage path, and store the mapping information;

[0047] in,

[0048] According to the storage hierarchy in the designed labeled file storage structure, the classified data is stored in the corresponding location. This process needs to be strictly operated in accordance with the specifications of the hierarchy to ensure accurate storage and efficient retrieval of data.

[0049] At the same time, a mapping relationship is established between the original data and the storage path, and this mapping information is stored and managed. This mapping relationship is the key to achieving rapid data location. Through this mapping relationship, the storage location of the required data can be quickly found, thereby improving data retrieval efficiency and utilization.

[0050] Specifically,

[0051] In terms of mapping relationship management, effective measures need to be taken to ensure the accuracy and completeness of mapping information, regularly verify the correctness of mapping information, and promptly update mapping information to reflect data changes.

[0052] S4. Input the encryption tag of the original data, locate the storage location of the original data and obtain the original data, that is, realize fast retrieval of data through the MD5 unique identifier.

[0053] in,

[0054] Because MD5 values ​​are highly unique and secure, they can ensure the accuracy and reliability of search results. During the search process, you only need to enter the MD5 value of the data, and the system can quickly locate the corresponding storage location and provide the required data.

[0055] In addition, when necessary, the retained original data backup can be used for data recovery. When the original data is damaged or lost, the retained original data backup can be used to quickly restore the original data status to ensure data integrity and availability. This data recovery function is of great significance to ensuring data security and reliability.

[0056] The present invention utilizes the MD5 encryption algorithm in combination with the test task unique ID, file name and file size to generate a unique identifier, which can realize data storage and rapid retrieval.

[0057] The present invention classifies and stores the original data, and designs a labeled file storage structure, including file type, content description, storage format and storage path design, so that the storage structure is standardized and the search is more efficient.

[0058] The present invention constructs a mapping relationship, a storage method and an update mechanism to ensure the accuracy and integrity of the mapping information.

[0059] The data retrieval and recovery process, implementation method and technical means of the present invention ensure that data can be retrieved and recovered quickly and accurately.

[0060] Advantages of the present invention:

[0061] In terms of unified data labeling, the present invention uses the MD5 encryption algorithm, combined with the unique ID of the test task, file name and file size, to generate a unique identifier, ensuring the uniqueness and security of the data; the original data is stored in a binary compressed file to improve confidentiality and versatility and reduce space occupation; in the labeled file storage structure, the original data is classified and stored according to content summary information (.h file), x-axis data (.x file), y-axis data (.y file) and z-axis data (.z file), realizing refined management and efficient retrieval of data; a clear file storage hierarchy is designed, namely MD5 / channel number / channel name + (.h, .y, .z), which facilitates rapid positioning and access of data; in terms of mapping relationship establishment and management, a mapping relationship between original data and storage path is established, and the mapping information is stored and managed to ensure rapid positioning and efficient use of data; in terms of data retrieval and recovery mechanism, rapid data retrieval is realized through the MD5 unique identifier, improving data retrieval efficiency; data recovery is performed using the retained original data backup to ensure data integrity and availability.

[0062] The present invention generates a 128-bit encryption tag through the MD5 algorithm to obtain a unique data identifier, reduces the collision probability to 1 / 2^128, and completely solves the data conflict problem; by designing four types of structured containers (.h / .x / .y / .z) and combining them with binary compression technology, the retrieval speed of the same data set is increased by 8 times (actually reduced from 120 seconds to 15 seconds), thereby optimizing storage efficiency; using binary compression storage and path mapping encryption, it can resist brute force attacks through CCRC certification and strengthen the security system; based on MD5 tags and a hierarchical storage structure, it supports second-level positioning of tens of millions of data, eliminates the need for manual retrieval, and realizes intelligent retrieval; overall, compared with the existing technology, data identification is more reliable, storage structure is more efficient, and data storage is more secure.

[0063] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A method for unified data labeling processing and storage, characterized in that: include: Process the raw data and generate an MD5 encryption identifier. The MD5 algorithm combines the test task unique ID, file name, and file size as input information to generate a unique 128-bit hash value as an encryption identifier. Based on the encryption identifier of the original data, combined with the channel number and channel name, a labeled file storage structure is formed; Classify and store the original data in the labeled file storage structure, establish a mapping relationship between the original data and the storage path, and store the mapping information; The encryption tag of the original data is input, the storage location of the original data is located, and the original data is obtained.

2. The method for unified data labeling processing and storage according to claim 1, characterized in that: The raw data processing includes data cleaning, format conversion, and missing value filling to ensure that the format is standardized and unified.

3. The method for unified data labeling processing and storage according to claim 1, characterized in that: The MD5 encryption identifier generation includes: The original data is grouped into 512-bit groups, and each group of data is divided into 16 32-bit sub-groups. The MD5 algorithm is used to output four 32-bit groups. The four 32-bit groups are cascaded to form a 128-bit hash value as an encryption identifier.

4. The method for unified data labeling processing and storage according to claim 1, characterized in that: In the labeled file storage structure, the original data is classified and stored according to content summary information, x-axis data, y-axis data, and z-axis data.

5. The method for unified data labeling processing and storage according to claim 1 or 4, characterized in that: The labeled file storage structure includes the encryption identifier, the number of channels and the channel name, and the channel name includes the original data classification identifier.

6. The method for unified data labeling processing and storage according to claim 1, characterized in that: A backup of the original data is retained in the labeled file storage structure for recovery and verification of the original data.

7. The method for unified data labeling processing and storage according to claim 6, characterized in that: When the original data is damaged or lost, the original data state is restored using the retained original data backup.

8. The method for unified data labeling processing and storage according to claim 1, characterized in that: In the labeled file storage structure, the original data is stored in a binary compressed file storage manner.

9. The method for unified data labeling processing and storage according to claim 1, characterized in that: In the tagged file storage structure, the original data is stored using the double basic data type.

10. The method for unified data labeling processing and storage according to claim 1, characterized in that: The correctness of the mapping information is checked regularly and the mapping information is updated in a timely manner.

Images