Data storage method, cloud storage system and storage medium
By performing format conversion and hashing operations on heterogeneous data, the target storage server is determined, and the problem of dispersed storage of heterogeneous data in the prior art is solved, and the unified storage and management of massive heterogeneous data is realized.
Patent Information
- Application Number
- CN202411138710.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-19
- Publication Date
- 2024-12-06
AI Technical Summary
The prior art is difficult to effectively manage and store massive heterogeneous data, resulting in homogeneous heterogeneous data being scattered and stored in different database systems and cannot be effectively managed.
By converting the stored data in format, generating data description text and structural description text, hashing operations to determine the target storage server, and storing the data to the same server to achieve unified storage and management of massive heterogeneous data.
It realizes effective storage and management of massive heterogeneous data, avoids data dispersed storage, and improves data unity and manageability.
Smart Images

Figure CN119088803A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of data storage, and in particular to a data storage method, a cloud storage system and a storage medium. Background Art
[0002] With the continuous development of highway traffic construction, the types and quantity of data related to highway traffic construction are increasing day by day. And due to the diversification of data sources, the data forms are different and there is heterogeneity. How to effectively store and manage these heterogeneous data has become an urgent problem to be solved.
[0003] In related technologies, relational database systems are usually used to store structured data, and NoSQL (NotOnly SQL, non-relational database) systems are used to store unstructured or semi-structured data. Using different databases to store heterogeneous data will result in part of the same-source heterogeneous data being stored in the relational database system and the other part being stored in the NoSQL system. The same-source heterogeneous data is scattered and stored in different database systems and cannot be effectively managed.
[0004] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention
[0005] The embodiments of the present application provide a data storage method, a cloud storage system, and a storage medium to solve the technical problem of being unable to effectively manage massive heterogeneous data.
[0006] To achieve the above object, an embodiment of the present application provides a data storage method, which includes the following:
[0007] Performing format conversion on the data to be stored, and generating data description text corresponding to the data to be stored;
[0008] Generate a structural description text according to the current timestamp, the attribute description and the classification code of each data in the data to be stored, wherein the data description text and the structural description text constitute a basic storage object of the data to be stored;
[0009] Performing a hash operation on the structural description text to obtain a hash value;
[0010] A target storage server for the basic storage object is determined according to the hash value, and the basic storage object is stored in the target storage server.
[0011] Optionally, before the step of generating a structural description text according to the current timestamp, the attribute description of each data in the data to be stored, and the classification code, the following steps are included:
[0012] According to a preset classification standard, the classification code of each data in the data to be stored is determined.
[0013] Optionally, the step of converting the format of the data to be stored and generating a data description text corresponding to the data to be stored includes:
[0014] Classifying the data to be stored according to the preset classification standard to obtain data to be stored of the corresponding category;
[0015] Format conversion is performed on the different categories of the data to be stored respectively to generate data description texts corresponding to the different categories of the data to be stored.
[0016] Optionally, the classification standard is the "Uniform Standard for Application of Highway Engineering Information Model".
[0017] Optionally, the corresponding categories include: engineering geographic information data category, engineering design file category and sensor data category.
[0018] Optionally, the step of performing format conversion on the data to be stored and generating a data description text corresponding to the data to be stored, or the step of performing format conversion on different categories of the data to be stored and generating data description texts corresponding to different categories of the data to be stored, includes:
[0019] Performing binary serialization processing on the data to be stored to generate byte content corresponding to the data to be stored;
[0020] The byte content is encoded based on a preset encoding method to generate the data description text corresponding to the data to be stored.
[0021] Optionally, the preset encoding method is Base64 encoding.
[0022] Optionally, the step of determining a target storage server of the basic storage object according to the hash value includes:
[0023] Convert the data type of the hash value to an integer;
[0024] Performing a modulus operation on the hash value according to a preset modulus to obtain a modulus result, wherein the modulus is the number of storage servers of the cloud storage system;
[0025] The target storage server is determined according to the modulo result.
[0026] In addition, to achieve the above-mentioned purpose, the present application also provides a cloud storage system, which includes: a memory, a processor, and a data storage program stored on the memory and executable on the processor, and the data storage program implements the steps of the data storage method described above when executed by the processor.
[0027] In addition, to achieve the above-mentioned purpose, the present application also provides a computer-readable storage medium, on which a data storage program is stored, and when the data storage program is executed by a processor, the steps of the data storage method described above are implemented.
[0028] The first embodiment of the present application proposes a data storage method, a cloud storage system and a storage medium, which converts the format of the data to be stored to generate a data description text corresponding to the data to be stored, and then generates a structural description text according to the current timestamp, the attribute description of each data in the data to be stored and the classification code, wherein the data description text and the structural description text constitute the basic storage object of the data to be stored, and then obtains a hash value by performing a hash operation on the structural description text, determines the target storage server of the basic storage object according to the hash value, and stores the basic storage object to the target storage server. The present application converts the data to be stored in different data forms into data description texts, so that the data to be stored in different forms can be stored in the same storage server without having to be stored in a dispersed manner. In addition, the present application determines the target storage server of the data to be stored by performing a hash operation on the structural description text, so that when managing massive heterogeneous data, the storage location can be directly determined by the structural description text corresponding to the data to be stored, thereby realizing effective storage and management of massive heterogeneous data. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] Figure 1 This is a schematic diagram of the steps of the first embodiment of the data storage method of the present application;
[0030] Figure 2 This is a schematic diagram of the structural flow of the first embodiment of the data storage method of the present application;
[0031] Figure 3 This is a detailed flow chart of the second embodiment of the data storage method of the present application;
[0032] Figure 4 This is a schematic diagram of the structural flow of the second embodiment of the data storage method of the present application;
[0033] Figure 5 It is a schematic diagram of the cloud storage system structure of the hardware operating environment involved in the embodiment of the present application.
[0034] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0035] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0036] In the related technologies, relational database systems are usually used to store structured data, and NoSQL (NotOnly SQL, non-relational database) systems are used to store unstructured or semi-structured data. Using different databases to store heterogeneous data will result in part of the same-source heterogeneous data being stored in the relational database system and the other part being stored in the NoSQL system. The same-source heterogeneous data is scattered and stored in different database systems and cannot be effectively managed.
[0037] In order to solve the above defects in the related art, the present application proposes a data storage method, the main solution steps of which include the following:
[0038] By converting the format of the data to be stored, a data description text corresponding to the data to be stored is generated, and then a structural description text is generated according to the current timestamp, the attribute description of each data in the data to be stored, and the classification code, wherein the data description text and the structural description text constitute the basic storage object of the data to be stored, and then a hash value is obtained by performing a hash operation on the structural description text, and the target storage server of the basic storage object is determined according to the hash value, and the basic storage object is stored to the target storage server. The present application converts the data to be stored in different data forms into a data description text, so that the data to be stored in different forms can be stored in the same storage server without having to be stored in a dispersed manner. In addition, the present application determines the target storage server of the data to be stored by performing a hash operation on the structural description text to obtain the hash value, so that when managing massive heterogeneous data, the storage location can be directly determined by the structural description text corresponding to the data to be stored, thereby realizing effective storage and management of massive heterogeneous data.
[0039] In order to better understand the above technical solution, exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.
[0040] Reference Figure 1 In the first embodiment of the data storage method of the present application, the data storage method is applied to a cloud storage system, and the method includes steps S10 to S40:
[0041] Step S10: converting the format of the data to be stored, and generating a data description text corresponding to the data to be stored;
[0042] In this embodiment, the data to be stored is binary serialized to generate the byte content corresponding to the data to be stored, and then the byte content is encoded based on a preset encoding method to generate a data description text corresponding to the data to be stored. The purpose is to convert the data to be stored in different data forms into a unified format so that the data to be stored in different forms can be stored in the same storage server without having to be stored in a dispersed manner. The data to be stored is data related to the field of highway traffic construction. In addition, the data to be stored can also be data related to other fields, which is not specifically limited in this embodiment.
[0043] Optionally, you can use Python 3 to read the data to be stored. Use the pickle.dump function to define the Python data object of the data to be stored, and use the Pickle module to perform binary serialization on the defined data object.
[0044] Optionally, the preset encoding method may be preferably Base64 encoding. In addition, the preset encoding method may also be other encoding methods, such as Hex encoding, ASCII encoding, UTF-8 encoding, URL encoding, and Binary encoding, and the user may flexibly select.
[0045] Step S20: generating a structural description text according to the current timestamp, the attribute description of each data in the data to be stored, and the classification code, wherein the data description text and the structural description text constitute a basic storage object of the data to be stored;
[0046] In this embodiment, by determining the classification code of each data in the data to be stored in advance according to the preset classification standard, the current timestamp, the attribute description of each data in the data to be stored and the classification code are assembled and written into the corresponding text file, thereby generating a structural description text. The data description text and the structural description text constitute the basic storage object of the data to be stored.
[0047] Optionally, the current timestamp can be obtained by using the Python built-in time library datetime module. In addition, other methods can be used to obtain the current timestamp, and users can flexibly choose.
[0048] Optionally, the preset classification standard may be a data classification standard related to the field corresponding to the data to be stored, thereby ensuring that the generated classification code has wide applicability, enabling it to maintain consistency and compatibility in different cloud storage systems, thereby facilitating data exchange and integration, and helping to achieve unified data management and access in various systems.
[0049] Specifically, when the data to be stored is data related to the field of highway traffic construction, the preset classification standard is the "Uniform Standard for the Application of Highway Engineering Information Models".
[0050] Optionally, since there are multiple versions of the data classification standard, the latest version of the data classification standard may be selected as the preset classification standard, or a version of the data classification standard that is compatible with the cloud storage system may be selected as the preset classification standard.
[0051] Specifically, when the data to be stored is related to the field of highway traffic construction, the recommended industry standard of the People's Republic of China JTG / T2420-2021 "Uniform Standard for the Application of Highway Engineering Information Models" officially implemented on June 1, 2021 can be selected as the preset classification standard.
[0052] Optionally, when the data to be stored is data related to the field of highway traffic construction, each data in the data to be stored can be classified according to the "Uniform Standard for Application of Highway Engineering Information Models" and then encoded into a classification code consisting of 5 segments of decimal numbers.
[0053] For example, assuming that the data to be stored is data related to the field of highway traffic construction, the content of the structure descriptive text file generated by the data to be stored is:
[0054] 51-01.03.19.01 1722945794
[0056] id INTEGER(11)
[0057] pipeline_id INTEGER(11)
[0058] pipeline_name CHAR(40)
[0059] longitude DECIMAL(11,4)
[0060] latitude DECIMAL(11,4)
[0061] altitude DECIMAL(11,4)
[0062] Among them, the first line of the above-mentioned structure descriptive text file is the classification code obtained by classifying the business data according to JTG / T2420-2021 "Uniform Standard for Application of Highway Engineering Information Model". This code specifically expresses that the first-level category of this business data is "terrain address", the second-level category is "terrain, landform, landform", the third-level category is "landform", the fourth-level category is "pipeline", and the fifth-level category is "on-road pipeline". The second line is the timestamp, and the specific time expressed is 2024-08-0620:03:13 Beijing time. Starting from the third line, each line represents the attribute description of the data structure of this data, and each line represents an attribute. The attribute named id is an integer with a maximum of 11 digits, representing the number of a data row; the attribute named pipeline_id is an integer with a maximum of 11 digits, representing the number of the pipeline on the road; the attribute named pipeline_name is a string with a maximum of 40 characters, representing the name of the pipeline on the road; the attribute named longitude is a fixed-point decimal number with a maximum of 11 digits and 4 decimal places, representing the GPS longitude information of an inflection point of the pipeline on the road; the attribute named latitude is a fixed-point decimal number with a maximum of 11 digits and 4 decimal places, representing the GPS latitude information of an inflection point of the pipeline on the road; the attribute named altitude is a fixed-point decimal number with a maximum of 11 digits and 4 decimal places, representing the GPS altitude information of an inflection point of the pipeline on the road.
[0063] Step S30: performing a hash operation on the structural description text to obtain a hash value;
[0064] In this embodiment, the structure description text is hashed according to the SHA-256 hash algorithm to obtain a hash value. In addition, other hash algorithms can be used to perform hash operations on the structure description file, for example, MD5, SHA-1 and other hash algorithms, and users can flexibly choose.
[0065] Step S40: determining a target storage server for the basic storage object according to the hash value, and storing the basic storage object in the target storage server.
[0066] In this embodiment, the data type of the hash value is converted to an integer, and then a modulus operation is performed on the hash value according to a preset modulus to obtain a modulus result, where the modulus is the number of storage servers in the cloud storage system, and then the target storage server is determined according to the modulus result. The modulus result is the unique identifier corresponding to the target storage server. Figure 2 , Figure 2 This is a schematic diagram of the structural flow of the first embodiment of the data storage method of the present application.
[0067] For example, assuming that the data to be stored is data related to the field of highway traffic construction, the number of storage servers in the cloud storage system is 3, and the content of the structure descriptive text file generated by the data to be stored is:
[0068] 51-01.03.19.01 1722945794
[0070] id INTEGER(11)
[0071] pipeline_id INTEGER(11)
[0072] pipeline_name CHAR(40)
[0073] longitude DECIMAL(11,4)
[0074] latitude DECIMAL(11,4)
[0075] altitude DECIMAL(11,4)
[0076] The structure descriptive text is hashed according to the SHA-256 hash algorithm, and the hash value M of the data to be stored is calculated and represented by HEX text as: b539259bdb2e187895758c48f2bcdf38e308fc2845ba7c4071ba28435087d904. The hash value M is further converted to an integer, and then a modulo 3 operation is performed on M, that is, the remainder is taken after dividing M by 3, and the result is recorded as i, i=0, 1, 2. When i=0, the basic storage object is stored on cloud storage server 0; when i=1, the basic storage object is stored on cloud storage server 1; when i=2, the basic storage object is stored on cloud storage server 2. According to the above structure descriptive text file, the result i value calculated after the modulo 3 operation of M is 1, so the storage server with the unique identifier of 1 is the target storage server.
[0077] In the technical solution provided in this embodiment, the data to be stored is converted into a format to generate a data description text corresponding to the data to be stored, and then a structural description text is generated according to the current timestamp, the attribute description of each data in the data to be stored, and the classification code, wherein the data description text and the structural description text constitute the basic storage object of the data to be stored, and then a hash value is obtained by performing a hash operation on the structural description text, and the target storage server of the basic storage object is determined according to the hash value, and the basic storage object is stored to the target storage server. The present application converts the data to be stored having different data forms into a data description text, so that the data to be stored in different forms can be stored in the same storage server without having to be stored in a dispersed manner. In addition, the present application determines the target storage server of the data to be stored by performing a hash operation on the structural description text to obtain the hash value, so that when managing massive heterogeneous data, the storage location can be directly determined by the structural description text corresponding to the data to be stored, thereby realizing effective storage and management of massive heterogeneous data.
[0078] Reference Figure 2 In the second embodiment, based on the first embodiment, the step S10 includes: steps S11-12:
[0079] Step S10: classifying the data to be stored according to the preset classification standard to obtain data to be stored of the corresponding category;
[0080] Step S12: performing format conversion on the different categories of the data to be stored respectively, and generating data description texts corresponding to the different categories of the data to be stored.
[0081] In this embodiment, after the data to be stored is classified according to the classification standard, different categories of data to be stored can be obtained, thereby generating different categories of data description texts respectively. The purpose is to generate detailed data description texts, thereby improving the structured degree of the data to be stored, and further making the management and retrieval of data more efficient and orderly.
[0082] Specifically, when the data to be stored is data related to the field of highway traffic construction, the corresponding categories include: engineering geographic information data category, engineering design file category and sensor data category. Thus, engineering geographic information data description text, engineering design file data description text and sensor data description text can be generated. Then, the engineering geographic information data description text, engineering design file data description text, sensor data description text and structural description text constitute the basic storage object. Please refer to Figure 4 , Figure 4 This is a schematic diagram of the structural flow of the second embodiment of the data storage method of the present application.
[0083] In the technical solution provided in this embodiment, the data to be stored is classified according to a preset classification standard to obtain the corresponding category of data to be stored, so that the formats of the different categories of data to be stored are converted respectively, and data description texts corresponding to the different categories of data to be stored are generated, aiming to improve the degree of structuring of the data to be stored and further make the management and retrieval of the data more efficient and orderly.
[0084] Reference Figure 5 , Figure 5 A schematic diagram of the structure of a cloud storage system in the hardware operating environment involved in an embodiment of the present application.
[0085] like Figure 5 As shown, the cloud storage system may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, and a communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), a mouse, etc., and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.
[0086] Those skilled in the art will understand that Figure 5 The cloud storage system structure shown in the figure does not constitute a limitation on the cloud storage system, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.
[0087] like Figure 5 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module and a data storage program.
[0088] exist Figure 5 In the cloud storage system shown, the network interface 1004 is mainly used to connect to the backend server and perform data communication with the backend server; the processor 1001 can be used to call the data storage program stored in the memory 1005 and perform the following operations:
[0089] Performing format conversion on the data to be stored, and generating data description text corresponding to the data to be stored;
[0090] Generate a structural description text according to the current timestamp, the attribute description and the classification code of each data in the data to be stored, wherein the data description text and the structural description text constitute a basic storage object of the data to be stored;
[0091] Performing a hash operation on the structural description text to obtain a hash value;
[0092] A target storage server for the basic storage object is determined according to the hash value, and the basic storage object is stored in the target storage server.
[0093] Further, the processor 1001 may call the data storage program stored in the memory 1005, and further perform the following operations:
[0094] According to a preset classification standard, the classification code of each data in the data to be stored is determined.
[0095] Classifying the data to be stored according to the preset classification standard to obtain data to be stored of the corresponding category;
[0096] Format conversion is performed on the different categories of the data to be stored respectively to generate data description texts corresponding to the different categories of the data to be stored.
[0097] Further, the processor 1001 may call the data storage program stored in the memory 1005, and further perform the following operations:
[0098] Performing binary serialization processing on the data to be stored to generate byte content corresponding to the data to be stored;
[0099] The byte content is encoded based on a preset encoding method to generate the data description text corresponding to the data to be stored.
[0100] Further, the processor 1001 may call the data storage program stored in the memory 1005, and further perform the following operations:
[0101] Convert the data type of the hash value to an integer;
[0102] Performing a modulus operation on the hash value according to a preset modulus to obtain a modulus result, wherein the modulus is the number of storage servers of the cloud storage system;
[0103] The target storage server is determined according to the modulo result.
[0104] In addition, to achieve the above-mentioned purpose, the present application also provides a cloud storage system, which includes: a memory, a processor, and a data storage program stored on the memory and executable on the processor, and the data storage program implements the steps of the data storage method described above when executed by the processor.
[0105] In addition, to achieve the above-mentioned purpose, the present application also provides a computer-readable storage medium, on which a data storage program is stored, and when the data storage program is executed by a processor, the steps of the data storage method described above are implemented.
[0106] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.
[0107] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.
[0108] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, including a number of instructions for a cloud storage system to execute the methods described in each embodiment of the present application.
[0109] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.
Claims
1. A data storage method, characterized in that: Applied to a cloud storage system, the data storage method includes: Performing format conversion on the data to be stored, and generating data description text corresponding to the data to be stored; Generate a structural description text according to the current timestamp, the attribute description and the classification code of each data in the data to be stored, wherein the data description text and the structural description text constitute a basic storage object of the data to be stored; Performing a hash operation on the structural description text to obtain a hash value; A target storage server for the basic storage object is determined according to the hash value, and the basic storage object is stored in the target storage server.
2. The method according to claim 1, characterized in that Before the step of generating a structural description text according to the current timestamp, the attribute description of each data in the data to be stored, and the classification code, the step includes: According to a preset classification standard, the classification code of each data in the data to be stored is determined.
3. The method according to claim 2, characterized in that The step of converting the format of the data to be stored and generating a data description text corresponding to the data to be stored includes: Classifying the data to be stored according to the preset classification standard to obtain data to be stored of the corresponding category; Format conversion is performed on the different categories of the data to be stored respectively to generate data description texts corresponding to the different categories of the data to be stored.
4. The method according to claim 2 or 3, characterized in that The preset classification standard is the "Uniform Standard for the Application of Highway Engineering Information Models".
5. The method according to claim 3, characterized in that The corresponding categories include: engineering geographic information data category, engineering design document category and sensor data category.
6. The method as claimed in claims 1 to 3, characterized in that The step of performing format conversion on the data to be stored and generating a data description text corresponding to the data to be stored, or the step of performing format conversion on different categories of the data to be stored and generating data description texts corresponding to different categories of the data to be stored, comprises: Performing binary serialization processing on the data to be stored to generate byte content corresponding to the data to be stored; The byte content is encoded based on a preset encoding method to generate the data description text corresponding to the data to be stored.
7. The method according to claim 6, characterized in that The preset encoding method is Base64 encoding.
8. The method according to claim 1, characterized in that The step of determining the target storage server of the basic storage object according to the hash value comprises: Convert the data type of the hash value to an integer; Performing a modulus operation on the hash value according to a preset modulus to obtain a modulus result, wherein the modulus is the number of storage servers of the cloud storage system; The target storage server is determined according to the modulo result.
9. A cloud storage system, characterized in that: The cloud storage system includes: a memory, a processor, and a data storage program of the cloud storage system stored in the memory and executable on the processor. When the data storage program of the cloud storage system is executed by the processor, the steps of the data storage method according to any one of claims 1 to 8 are implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a data storage program of the cloud storage system, and when the data storage program of the cloud storage system is executed by the processor, the steps of the data storage method according to any one of claims 1 to 8 are implemented.
Citation Information
Cited By
Unified access and standardized conversion method for OT / IT heterogeneous data
CN122470595A
