Method and system for storing production city data based on distributed storage architecture

Abstract

The invention discloses a production city data storage method and system based on a distributed storage architecture, the distributed storage architecture comprises a data acquisition layer, a data storage layer and a data access layer, and the data storage layer comprises a structured data storage library and an unstructured data storage library. The method comprises the following steps: acquiring production city data acquired by a data acquisition layer, and respectively acquiring corresponding structured data and unstructured data from the production city data; obtaining a geographical location area to which the unstructured data belongs, determining a target storage pool corresponding to the unstructured data from the plurality of storage pools according to the geographical location area to which the unstructured data belongs, and correspondingly storing the unstructured data into the target storage pool; metadata corresponding to the unstructured data are obtained, and the metadata and the structured data are stored in a structured database. The problem that storage management of massive and multi-type production city data cannot be handled in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and particularly to a storage method and system for industrial city data based on a distributed storage architecture. Background Art

[0002] With the rapid advancement of the urbanization process, the integrated development of industry and city has become increasingly close, and industrial city data has emerged as the times require. Industrial city data refers to various types of information related to the development of industry and city, which covers multiple dimensions such as industrial operation status, urban construction planning, population flow distribution, and infrastructure utilization, and is of great significance to the development strategies of cities and industries.

[0003] In the process of digital transformation of cities and industries, more and more devices and systems start to generate data. For example, Internet of Things sensors scattered throughout the city, and these data acquisition sources continuously generate a large amount of industrial city data. These data are not only huge in quantity but also complex in variety, including both structured data that is easy to represent in tables or relational models, such as urban population statistics data, and unstructured data that is difficult to store and process according to a fixed format, such as urban construction planning documents and urban traffic monitoring videos.

[0004] The scale and complexity of industrial city data pose severe challenges to the effective management of data. If these data cannot be properly collected and stored, it will not only lead to data loss and waste of information resources, but also make subsequent data-based analysis and decision-making lack accurate and comprehensive data support. However, traditional data management models are difficult to meet the processing requirements of large-scale and multi-type data. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a storage method and system for industrial city data based on a distributed storage architecture, aiming to solve the problem in the prior art that it is impossible to cope with the storage management of massive and multi-type industrial city data.

[0006] The present invention is implemented as follows:

[0007] A storage method for industrial city data based on a distributed storage architecture is used to store industrial city data through a distributed storage architecture. The distributed storage architecture includes a data acquisition layer, a data storage layer, and a data access layer. The data storage layer includes a structured data repository and an unstructured data repository. The unstructured data repository includes storage pools corresponding to multiple different geographical location regions. The method includes:

[0008] Obtain the industrial city data collected by the data acquisition layer, and respectively obtain the corresponding structured data and unstructured data from the industrial city data;

[0009] Obtain the geographical location area to which the unstructured data belongs, determine the target storage pool corresponding to the unstructured data from multiple storage pools according to the geographical location area to which it belongs, and store the unstructured data correspondingly in the target storage pool;

[0010] Obtain the metadata corresponding to the unstructured data, and store the metadata and the structured data in the structured database respectively.

[0011] Further, for the above storage method of industrial city data based on a distributed storage architecture, in the step of obtaining the geographical location area to which the unstructured data belongs, determining the target storage pool corresponding to the unstructured data from multiple storage pools according to the geographical location area to which it belongs, and storing the unstructured data correspondingly in the target storage pool, the following steps are further included:

[0012] Within a preset period of time, after the storage volume in the target storage pool reaches a preset threshold, add a random storage time;

[0013] During the random storage time, obtain the target unstructured data that needs to be stored in the target storage pool, obtain the attribute information of the target unstructured data, and map the attribute information according to a preset rule to obtain the location information in other storage pools;

[0014] Store the target unstructured data correspondingly according to the location information of the target unstructured data in other storage pools.

[0015] Further, for the above storage method of industrial city data based on a distributed storage architecture, where the attribute information includes file type, geographical location area to which it belongs, and creation time, the step of mapping the attribute information according to a preset rule to obtain the location information in other storage pools includes:

[0016] Respectively obtain the letter elements and digital elements corresponding to the file type, geographical location area to which it belongs, and creation time, and perform hash processing on the letter elements and digital elements to obtain a storage sequence containing the storage information of the target unstructured data;

[0017] Each storage pool is set with a corresponding number. Select a random number of digital element combinations in the storage sequence, and perform a modulo operation on the total number of other storage pools to obtain a value, and determine the target other storage pool that needs to be stored.

[0018] Further, for the above storage method of industrial city data based on a distributed storage architecture, after the step of storing the target unstructured data correspondingly according to the location information of the target unstructured data in other storage pools, the following steps are further included:

[0019] After the random storage time, continue to execute the steps of obtaining the geographical location area to which the unstructured data belongs, determining the target storage pool corresponding to the unstructured data from multiple storage pools according to the geographical location area to which it belongs, and storing the unstructured data correspondingly in the target storage pool.

[0020] Further, in the above storage method of urban and industrial data based on a distributed storage architecture, the step of determining the target other storage pool to be stored by taking the modulus of the total number of other storage pools with the value obtained by combining a random number of digital elements selected from the storage sequence includes:

[0021] Obtain the storage path set of the target unstructured data within the random storage time, where the storage path set consists of different storage paths;

[0022] Obtain the keywords of the file name of the target unstructured data, and assign corresponding identifiers to each corresponding storage path according to the keywords;

[0023] Set a corresponding cache layer in the query path of the data access layer, and cache the storage path set in the cache layer to achieve fast query of the target unstructured data through the cache layer when querying the target unstructured data.

[0024] Further, in the above storage method of urban and industrial data based on a distributed storage architecture, after the step of setting a corresponding cache layer in the query path of the data access layer and caching the storage path set in the cache layer, it further includes:

[0025] When the actual storage path of the target unstructured data changes, trigger an update event, and the update event includes relevant information about the storage path change;

[0026] Send the triggered update event information to the message queue, and the set path listening service continuously receives event information from the message queue;

[0027] The path listening service updates the storage path set in the cache layer according to the received event content to ensure the consistency between the storage path in the cache layer and the actual storage path.

[0028] Further, in the above storage method of urban and industrial data based on a distributed storage architecture, the method further includes:

[0029] Determine a corresponding verification chain for each storage path in the cache layer, and store the verification chain in the cache layer;

[0030] Regularly calculate the actual verification chain of the actual storage path, compare it with the verification chain in the cache layer, and determine whether the storage path has changed according to the verification result.

[0031] Another object of the present invention is to provide a storage system for urban-industrial data based on a distributed storage architecture, which is used to store urban-industrial data through the distributed storage architecture. The distributed storage architecture includes a data acquisition layer, a data storage layer, and a data access layer. The data storage layer includes a structured data repository and an unstructured data repository. The unstructured data repository includes storage pools corresponding to multiple different geographical location regions. The system includes:

[0032] An acquisition module, configured to acquire the urban-industrial data collected by the data acquisition layer, and respectively acquire the corresponding structured data and unstructured data from the urban-industrial data;

[0033] A determination module, configured to obtain the geographical location region to which the unstructured data belongs, and determine the target storage pool corresponding to the unstructured data from multiple storage pools according to the geographical location region to which it belongs, and store the unstructured data correspondingly in the target storage pool;

[0034] A storage module, configured to obtain the metadata corresponding to the unstructured data, and store the metadata and the structured data in the structured database respectively.

[0035] Another object of the present invention is to provide a readable storage medium, on which a computer program is stored, and characterized in that when the program is executed by a processor, the steps of the method described in any one of the above are implemented.

[0036] Another object of the present invention is to provide an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the steps of the method described in any one of the above are implemented.

[0037] The present invention acquires the urban-industrial data collected by the data acquisition layer, and divides it into structured data and unstructured data, avoiding the confusion of different types of data, and improving the pertinence and efficiency of data management. In terms of unstructured data processing, first obtain its geographical location region, and then determine the target storage pool from multiple storage pools accordingly and store the data, which not only makes full use of the space of multiple storage pools to cope with the problem of the huge scale of urban-industrial data, but also facilitates data retrieval and management. After that, obtain the metadata of the unstructured data, and store it together with the structured data in the structured database. With the powerful query and management functions of the structured database, data loss is avoided and data security is guaranteed, breaking the limitations of the traditional data management mode in dealing with large-scale and multi-type data. It solves the problem that the prior art cannot cope with the storage management of massive and multi-type urban-industrial data. Description of the Drawings

[0038] Figure 1Flowchart of the storage method of industrial city data based on a distributed storage architecture in the first embodiment of the present invention;

[0039] Figure 2 Block diagram of the structure of the storage system of industrial city data based on a distributed storage architecture in the third embodiment of the present invention.

[0040] The following specific embodiments will further illustrate the present invention in conjunction with the above-mentioned drawings. Specific Embodiments

[0041] To facilitate the understanding of the present invention, the present invention will be described more comprehensively below with reference to the relevant drawings. Several embodiments of the present invention are shown in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, these embodiments are provided to make the disclosure of the present invention more thorough and comprehensive.

[0042] It should be noted that when an element is referred to as being "fixed to" another element, it can be directly on the other element or there can also be an intermediate element. When an element is considered to be "connected" to another element, it can be directly connected to the other element or there may be an intermediate element at the same time. The terms "vertical", "horizontal", "left", "right" and similar expressions used herein are for illustrative purposes only.

[0043] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field to which the present invention belongs. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more of the related listed types.

[0044] The following will combine specific embodiments and drawings to elaborate on how to solve the problem in the prior art that the storage management of a large amount of and multi-type industrial city data cannot be handled.

[0045] Example 1

[0046] Please refer to Figure 1 , which shows the storage method of industrial city data based on a distributed storage architecture in the first embodiment of the present invention, for storing industrial city data through a distributed storage architecture. The distributed storage architecture includes a data acquisition layer, a data storage layer, and a data access layer. The data storage layer includes a structured data repository and an unstructured data repository. The unstructured data repository includes storage pools corresponding to multiple different geographical location regions. The method includes steps S10 to S12.

[0047] Among them, the data acquisition layer is the entrance for urban-industrial data to flow into the data management system or platform. It is responsible for collecting data from diverse data sources, including sensors deployed in various corners of the city or industrial park, and data acquisition interfaces, etc., to collect information from all corners of the city. For example, traffic sensors collect real-time traffic condition data, and devices in industrial parks collect enterprise production data, etc., covering multiple dimensions of urban-industrial data.

[0048] The data storage layer receives the urban-industrial data transmitted from the data acquisition layer. Specifically, the data storage layer includes a structured database and an unstructured database, which are respectively used to store structured data and unstructured data. The structured data repository receives the structured data transmitted from the data acquisition layer, such as the census data of the city, etc. Structured data can be managed by a relational database, enabling efficient data query, statistics, and analysis. For example, when the government department formulates industrial support policies, it can quickly query the financial data and employee scale data of relevant enterprises from the structured data repository. The unstructured data repository consists of storage pools corresponding to multiple different geographical location regions and is used to store unstructured data, such as urban surveillance videos, construction planning documents, etc. By obtaining the geographical location region to which the unstructured data belongs, the data is stored in the corresponding storage pool, which not only optimizes the utilization of storage resources but also facilitates data location and retrieval. For example, the traffic surveillance videos of a certain region can be directly stored in the storage pool corresponding to that region, and relevant video data can be quickly found when investigating traffic incidents in that region later.

[0049] The data access layer provides a unified data access interface for users or other application systems. This enables users to query and obtain the required data through this interface without having to know the specific storage location and format of the data. For example, when the urban planning department formulates a new regional plan, it only needs to send a query request through the data access layer to obtain data such as population distribution data from the structured data repository and regional satellite images from the unstructured data repository to support the planning decision-making.

[0050] Step S10, obtain the urban-industrial data collected by the data acquisition layer, and respectively obtain the corresponding structured data and unstructured data from the urban-industrial data.

[0051] Among them, in the distributed storage system of urban-industrial data, the data acquisition layer plays a key role as a data "collector", which widely collects urban-industrial data from diverse and complex data sources.

[0052] After the data acquisition layer successfully collects the industrial and urban data, it is necessary to classify these data in a refined manner, and identify and separate structured data and unstructured data from them. Structured data has a fixed format and a clear logical structure, and is easy to store and retrieve through a database. For example, financial statement data, employee attendance data generated during the operation of an enterprise, and census data in urban management all belong to structured data. Unstructured data, on the other hand, is different. Its format is free and lacks a predefined structure, such as urban traffic monitoring videos, engineering design drawings, policy document texts, etc. Accurately classifying the collected industrial and urban data into structured and unstructured categories can make different types of data adapt to subsequent corresponding storage and processing mechanisms, ensuring the efficient and orderly progress of the entire industrial and urban data management process.

[0053] Step S11, obtain the geographical location area to which the unstructured data belongs, and determine the target storage pool corresponding to the unstructured data from multiple storage pools according to the geographical location area to which it belongs, and store the unstructured data correspondingly in the target storage pool.

[0054] After separating the unstructured data from the industrial and urban data, since the unstructured data is not only huge in volume but also contains various forms such as videos, documents, and images, efficient storage becomes a key challenge. This link optimizes the storage process of unstructured data by determining the geographical location of the data.

[0055] First of all, the system will analyze and extract the geographical location area information associated with each piece of unstructured data. Specifically, the area where the industrial and urban data is generated can be divided first, the number of storage pools can be set according to the number of divisions, and then the corresponding storage can be determined according to the regional source of the data in the corresponding storage pool. In specific implementation, the GPS positioning information of the acquisition device can be used to confirm the regional source of the data, or each device can be numbered in advance, and the regional source of the data can be determined according to the number information.

[0056] After obtaining the geographical location area information, match this information with the location attributes of multiple storage pools. These storage pools are pre-divided according to geographical locations and are distributed in different regions. For example, each urban area, a specific industrial park, or a transportation hub corresponds to a storage pool. Taking the monitoring video of a certain traffic artery as an example, if it is determined that it is shot at a certain section in the city center, then it is matched with the storage pool corresponding to the city center area, and this storage pool is selected as the target storage pool.

[0057] Finally, the system will transfer and store the unstructured data into the corresponding target storage pool. In this way, the physical location of the unstructured data corresponds to the geographical location area of the data source, which not only greatly improves the rationality of data storage, avoids the over - occupation or idleness of storage resources, but also can quickly locate the target data according to the geographical location during subsequent data retrieval and invocation, greatly improving the retrieval efficiency.

[0058] Step S12: Obtain the metadata corresponding to the unstructured data, and store the metadata and the structured data into the structured database respectively.

[0059] Among them, after the storage of the unstructured data is completed, in order to further improve the systematicness and efficiency of the industrial - urban data management, it is necessary to obtain the corresponding metadata for the unstructured data, and properly store the metadata of the unstructured data and the structured data.

[0060] Metadata is data that describes data. It can provide key information for unstructured data, such as data source, creation time, data format, etc. Taking a city traffic surveillance video as an example, its metadata may include information about the shooting device of the video, the start and end times of shooting, video format, actual storage path, etc.; for a planning document of an industrial park, the metadata may cover information such as the creator of the document, creation date, file version, actual storage path, etc. These metadata are extracted from the unstructured data and its associated file attributes through preset algorithms and programs.

[0061] At the same time, the structured data has already been identified and separated due to its regular format and high organization. In order to achieve centralized and standardized data management, both the extracted metadata of the unstructured data and the existing structured data are uniformly stored in the structured database. The structured database has powerful indexing, query, and transaction - processing functions, and can efficiently manage data. Storing the metadata of the unstructured data and the structured data together in the structured database can provide great convenience for subsequent data mining, query, and management.

[0062] In summary, for the storage method of urban-industrial data based on the distributed storage architecture in the above embodiments of the present invention, by acquiring the urban-industrial data collected by the data acquisition layer and dividing it into structured data and unstructured data, it avoids the confusion of different types of data, improves the pertinence and efficiency of data management. In terms of unstructured data processing, first obtain its geographical location area, and then determine the target storage pool from multiple storage pools based on this and store the data, which not only makes full use of the space of multiple storage pools to cope with the problem of the huge scale of urban-industrial data, but also facilitates data retrieval and management. After that, obtain the metadata of the unstructured data and store it together with the structured data in the structured database. By virtue of the powerful query and management functions of the structured database, data loss is avoided and data security is guaranteed, breaking the limitations of the traditional data management mode in dealing with large-scale and multi-type data. It solves the problem in the prior art that the storage management of massive and multi-type urban-industrial data cannot be handled.

[0063] Example 2

[0064] This embodiment also proposes a storage method of urban-industrial data based on the distributed storage architecture. The difference between the storage method of urban-industrial data based on the distributed storage architecture proposed in this embodiment and the storage method of urban-industrial data based on the distributed storage architecture proposed in Embodiment 1 of this invention is as follows:

[0065] After the step of obtaining the geographical location area to which the unstructured data belongs and determining the target storage pool corresponding to the unstructured data from multiple storage pools according to the geographical location area and storing the unstructured data corresponding to the target storage pool, the following steps are further included:

[0066] Within a preset period of time, after the storage amount in the target storage pool reaches the preset threshold, a random storage time is added;

[0067] During the random storage time, obtain the target unstructured data that needs to be stored in the target storage pool, obtain the attribute information of the target unstructured data, and map it according to the attribute information according to the preset rules to obtain the location information in other storage pools;

[0068] Store the target unstructured data correspondingly according to the location information of the target unstructured data in other storage pools.

[0069] Among them, after the storage pool is divided according to regions, when centralizing the storage of data in the same region, the centralized storage of data is likely to cause problems such as low security and unbalanced load. Therefore, random and cross-storage of unstructured data among multiple storage pools is introduced. Specifically, within a preset period of time, after the storage volume in the target storage pool reaches the preset threshold, a random storage time is added, that is, the trigger condition for random storage is set. By setting the storage volume in the target storage pool within a preset period of time as the trigger condition, the purpose is to store a large amount of data centrally in the target storage pool. During the random storage time, when new target unstructured data needs to be stored in the target storage pool, the attribute information of these data will be obtained first. These attribute information may include data type, size, creation time, etc. Then, according to the preset rules, the location information of other storage pools will be mapped from these attribute information, that is, to determine which storage pool among the other remaining storage pools the target unstructured data to be stored in the target storage pool during this period will be stored in. Finally, according to the obtained other storage pools, the target unstructured data will be stored in the corresponding other storage pools.

[0070] Specifically, the attribute information includes file type, geographical location area to which it belongs, and creation time. The step of mapping the location information in other storage pools according to the attribute information according to the preset rules includes:

[0071] Respectively obtain the letter elements and digital elements corresponding to the file type, geographical location area to which it belongs, and creation time, and perform hash processing on the letter elements and digital elements to obtain a storage sequence containing the storage information of the target unstructured data;

[0072] Each storage pool is set with a corresponding number. The value obtained by taking the modulus operation of the total number of other storage pools with the value obtained by combining a random number of digital elements selected from the storage sequence determines the target other storage pool to be stored.

[0073] Specifically, extract the key attribute information such as the file type, geographical location area to which it belongs, and creation time of the target unstructured data, and convert this information into letter elements and digital elements. The reason for doing this is that these elements can become the basic data for generating the storage sequence in the future. Among them, for the file type, the letter elements representing the type can be selected, such as the extension name, and the pinyin letter elements of the geographical location area to which it belongs. The area division has been carried out corresponding to the initial establishment of the storage pool, so that the geographical location area to which it belongs can be obtained. Immediately afterwards, hash processing is performed on the obtained letters and digital elements. Hash processing can convert data of any length into a hash value of a fixed length through a specific algorithm, generating a storage sequence containing the storage information of the target unstructured data.

[0074] Each storage pool is pre-allocated with a corresponding number. From the previously generated storage sequence, a certain number of digital elements are randomly selected and combined into a value. The reason for random selection is to make the data more evenly distributed among the storage pools. Subsequently, this combined value is used to perform a modulo operation on the total number of other storage pools except the current target storage pool. The result of the modulo operation is actually a remainder with the total number of other storage pools as the period. And based on this remainder, the corresponding storage pool number can be obtained, which is the number of the target other storage pool where the target unstructured data needs to be stored, thereby determining the storage location of the target unstructured data in other storage pools. Specifically, since the storage pool numbers are sequentially numbered, the obtained remainder can be added to the number of the target storage pool to get the number of the target other storage pool for storing the data, or the remainder can directly determine the number of the target other storage pool for storing the data.

[0075] In addition, in some alternative embodiments of the present invention, after the step of storing the target unstructured data correspondingly according to the location information of the target unstructured data in other storage pools, the following steps are further included:

[0076] After the random storage time has passed, continue to execute the steps of obtaining the geographical location area to which the unstructured data belongs, and determining the target storage pool corresponding to the unstructured data from multiple storage pools according to the geographical location area to which it belongs, and storing the unstructured data correspondingly into the target storage pool.

[0077] Among them, after the target unstructured data is stored in other storage pools according to the location information, when the preset random storage time ends, it will return to the normal data storage process of dividing storage pools based on geographical location. That is to say, the system will collect newly generated unstructured data again and determine the geographical location areas to which these data belong. This is because the geographical location area is the key basis for determining the initial storage pool of the data. Subsequently, the system finds the target storage pool that matches it from many storage pools based on the geographical location area to which the data belongs. Finally, the system stores the unstructured data into the determined target storage pool.

[0078] In addition, in some alternative embodiments of the present invention, the step of determining the target other storage pool that needs to be stored by performing a modulo operation on the total number of other storage pools with the value obtained by combining a random number of digital elements selected from the storage sequence includes:

[0079] Obtain the storage path set of the target unstructured data within the random storage time, where the storage path set is composed of different storage paths;

[0080] Obtain the keywords of the file name of the target unstructured data, and assign corresponding identifiers to each corresponding storage path according to the keywords;

[0081] A corresponding cache layer is set in the query path of the data access layer, and the storage path set is cached in the cache layer to achieve fast query of the target unstructured data through the cache layer when querying the target unstructured data.

[0082] Among them, the storage path set of the target unstructured data within the random storage time is collected. These paths are composed of different storage paths and cover the specific location information where the target unstructured data is randomly stored in other storage pools. Collecting the storage path set is the basis for subsequent fast data query. Subsequently, the file name of the target unstructured data is processed to extract the keywords in the file name. Through these keywords, the system assigns a corresponding identifier to each storage path, that is, a "tag" is attached to each data storage location, which helps to clarify the location and attributes of the data during subsequent query and access. The extraction of keywords and the assignment of identifiers actually construct a mapping relationship between the data and the storage path, facilitating the efficient positioning of the data in the system.

[0083] Furthermore, a cache layer is set in the query path of the data access layer, and the storage path set is cached in this cache layer. When a query is initiated for the target unstructured data, the system first searches in the cache layer. Since the storage path set of the target unstructured data has been saved in the cache layer and each path has been assigned an identifier related to the data content, the system can quickly match the corresponding storage path according to the keywords in the query content without conducting a comprehensive search in the entire storage system, thus greatly shortening the query time, achieving fast query of the target unstructured data, and effectively solving the problem of difficult data query caused by random storage.

[0084] In addition, when the actual storage path of the target unstructured data changes, an update event is triggered, and the update event includes relevant information about the change in the storage path;

[0085] The triggered update event information is sent to the message queue, and the set path listening service continuously receives the event information from the message queue;

[0086] The path listening service updates the storage path set in the cache layer according to the received event content to ensure the consistency between the storage path in the cache layer and the actual storage path.

[0087] Among them, after setting up the cache layer, the query efficiency of data is improved. However, when the storage path of data changes, it will cause the path in the cache layer to not match the actual storage path. Therefore, a path monitoring service is set up to continuously receive event information from the message queue. Once there is new event information in the message queue, it can quickly capture it. After receiving the event content, the path monitoring service will update the storage path set in the cache layer according to the detailed information of the storage path change therein. Since the cache layer previously stored the data storage paths for quick query, and after the actual storage path changes, if the cache layer path is not updated synchronously, it will cause the queried path to not match the actual path, affecting data acquisition. By timely updating the storage path set of the cache layer, it can be ensured that the storage path in the cache layer is consistent with the actual storage path of the target unstructured data, enabling the data query function to still accurately and efficiently acquire data in the case of a storage path change.

[0088] Meanwhile, a path verification mechanism can also be set up. For example, a corresponding verification chain is determined for each storage path in the cache layer, and this verification chain is stored in the cache layer.

[0089] Regularly calculate the actual verification chain of the actual storage path and compare it with the verification chain in the cache layer to determine whether the storage path has changed according to the verification result.

[0090] First of all, the system will determine the corresponding verification chain for the storage path of each target unstructured data stored in the cache layer through specific algorithms or rules. The verification chain is a combination of information that can reflect the characteristics of the storage path. It is like the "identity identifier" of the storage path and is obtained by calculating and processing the relevant information of the storage path. After generating the verification chain, the system will store it in the cache layer and save it together with the corresponding storage path information.

[0091] Next, at regular time intervals as set, the actual storage path of the target unstructured data will be processed regularly to calculate its actual verification chain. This process also involves using corresponding algorithms to operate on the relevant information of the actual storage path. After calculating the actual verification chain, it will be compared with the verification chain already stored in the cache layer. By comparing the consistency of the two, it can be determined whether the storage path has changed. If the actual verification chain is inconsistent with the verification chain in the cache layer, it indicates that the storage path has changed; if the two are consistent, it means that the storage path has not changed. In this way, through the comparison of the verification chains, the status of the storage path can be grasped in a timely and accurate manner, providing a reliable basis for subsequent data query and management. Exemplarily, the key information of the storage path, such as the storage pool number, the specific location index of the file in the storage pool, the file name, etc., can be combined into a string. Then, a hash algorithm (such as MD5, SHA-1, or SHA-256, etc.) is used to calculate this string to generate a hash value of a fixed length as part of the verification chain. Different hash algorithms can be used multiple times to calculate the original information or intermediate results, and the multiple obtained hash values can be concatenated to form a complete verification chain, or the storage path information can be decomposed into multiple small pieces. For example, it can be divided by bytes or characters. Then, a hash calculation is performed on each small piece, and the adjacent hash values are combined in pairs and then hashed again to form a tree structure, and the value of the root node of the tree is the final verification chain.

[0092] In summary, in the storage method of the industrial-city data based on the distributed storage architecture in the above embodiments of the present invention, by obtaining the industrial-city data collected by the data collection layer and dividing it into structured data and unstructured data, it avoids the confusion of different types of data and improves the pertinence and efficiency of data management. In terms of unstructured data processing, first obtain its geographical location area, and then determine the target storage pool from multiple storage pools based on this and store the data. This not only makes full use of the space of multiple storage pools to address the problem of the large scale of industrial-city data, but also facilitates data retrieval and management. After that, obtain the metadata of the unstructured data and store it together with the structured data in the structured database. Relying on the powerful query and management functions of the structured database, it avoids data loss and ensures data security, breaking the limitations of the traditional data management mode in dealing with large-scale and multi-type data. It solves the problem in the prior art that it is impossible to handle the storage management of massive and multi-type industrial-city data.

[0093] Example 3

[0094] Please refer to Figure 2, shown is a storage system for urban-industrial data based on a distributed storage architecture proposed in the third embodiment of the present invention, which is used to store urban-industrial data through the distributed storage architecture. The distributed storage architecture includes a data collection layer, a data storage layer, and a data access layer. The data storage layer includes a structured data repository and an unstructured data repository. The unstructured data repository includes storage pools corresponding to multiple different geographical location regions. The system includes:

[0095] An acquisition module 100, configured to acquire the urban-industrial data collected by the data collection layer, and respectively acquire the corresponding structured data and unstructured data from the urban-industrial data;

[0096] A determination module 200, configured to acquire the geographical location region to which the unstructured data belongs, and determine the target storage pool corresponding to the unstructured data from multiple storage pools according to the geographical location region to which it belongs, and store the unstructured data correspondingly into the target storage pool;

[0097] A storage module 300, configured to acquire the metadata corresponding to the unstructured data, and store the metadata and the structured data into the structured database respectively.

[0098] The functions or operation steps implemented when the above modules are executed are substantially the same as those in the above method embodiments, and will not be elaborated here.

[0099] Example 4

[0100] On the other hand, the present invention also provides a readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the steps of the method described in any one of the above embodiments 1 to 2 are implemented.

[0101] Example 5

[0102] On the other hand, the present invention also provides an electronic device, the electronic device includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the steps of the method described in any one of the above embodiments 1 to 2 are implemented.

[0103] The technical features of each of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0104] Those skilled in the art will understand that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a definite sequence list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device), or used in combination with these instruction execution systems, apparatus, or devices. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

[0105] More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection portion (electronic device) having one or more wirings, a portable computer diskette (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpretation, or otherwise processing as appropriate, and then stored in a computer memory.

[0106] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application specific integrated circuits having appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.

[0107] In the description of this specification, the description with reference to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc. means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0108] The above-described embodiments merely represent several implementation manners of the present invention. The description thereof is relatively specific and detailed, but it should not be construed as a limitation on the scope of the patent for the present invention. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several modifications and improvements can still be made, and these all fall within the protection scope of the present invention. Therefore, the protection scope of the patent for the present invention shall be subject to the appended claims.

Claims

1. A storage method for industrial city data based on a distributed storage architecture, characterized in that, Used for storing industrial city data through a distributed storage architecture, the distributed storage architecture includes a data collection layer, a data storage layer, and a data access layer. The data storage layer includes a structured data repository and an unstructured data repository. The unstructured data repository includes storage pools corresponding to multiple different geographical location regions. The method includes: Obtain the industrial city data collected by the data collection layer, and respectively obtain the corresponding structured data and unstructured data from the industrial city data; Obtain the geographical location region to which the unstructured data belongs, and determine the target storage pool corresponding to the unstructured data from multiple storage pools according to the geographical location region to which it belongs, and store the unstructured data correspondingly in the target storage pool; Obtain the metadata corresponding to the unstructured data, and store the metadata and the structured data in the structured database respectively.

2. The storage method of the industrial city data based on the distributed storage architecture according to claim 1, wherein, After the step of obtaining the geographical location region to which the unstructured data belongs, and determining the target storage pool corresponding to the unstructured data from multiple storage pools according to the geographical location region to which it belongs, and storing the unstructured data correspondingly in the target storage pool, the following steps are further included: Within a preset period of time, after the storage amount in the target storage pool reaches a preset threshold, add a random storage time; During the random storage time, obtain the target unstructured data that needs to be stored in the target storage pool, obtain the attribute information of the target unstructured data, and map the attribute information according to a preset rule to obtain the location information in other storage pools; Store the target unstructured data correspondingly according to the location information of the target unstructured data in other storage pools.

3. The storage method of the industrial city data based on the distributed storage architecture according to claim 2, wherein The attribute information includes file type, geographical location region to which it belongs, and creation time. The step of mapping the attribute information according to a preset rule to obtain the location information in other storage pools includes: Respectively obtain the letter elements and digital elements corresponding to the file type, geographical location region to which it belongs, and creation time, and perform hash processing on the letter elements and digital elements to obtain a storage sequence containing the storage information of the target unstructured data; Each storage pool is set with a corresponding number. Select a random number of digital element combinations in the storage sequence, and perform a modulo operation on the total number of other storage pools to obtain a value, and determine the target other storage pool that needs to be stored.

4. The storage method of the industrial city data based on the distributed storage architecture according to claim 2, wherein After the step of storing the target unstructured data correspondingly according to the location information of the target unstructured data in other storage pools, the following steps are further included: After the random storage time has passed, continue to execute the step of obtaining the geographical location region to which the unstructured data belongs, and determining the target storage pool corresponding to the unstructured data from multiple storage pools according to the geographical location region to which it belongs, and storing the unstructured data correspondingly in the target storage pool.

5. The storage method of industrial city data based on a distributed storage architecture according to claim 3, wherein, The step of selecting a random number of digital element combinations in the storage sequence, and performing a modulo operation on the total number of other storage pools to obtain a value, and determining the target other storage pool that needs to be stored includes: Obtain the storage path set of the target unstructured data during the random storage time, where the storage path set consists of different storage paths; Obtain the keywords of the file names of the target unstructured data, and assign corresponding identifiers to each corresponding storage path according to the keywords; Set a corresponding cache layer in the query path of the data access layer, and cache the storage path set in the cache layer to achieve fast query of the target unstructured data through the cache layer when querying the target unstructured data.

6. The storage method of the industrial city data based on the distributed storage architecture according to claim 5, characterized in that, After the step of setting a corresponding cache layer in the query path of the data access layer and caching the storage path set in the cache layer, it further includes: When the actual storage path of the target unstructured data changes, an update event is triggered, and the update event includes relevant information on the change of the storage path; Send the triggered update event information to the message queue, and the set path listening service continuously receives the event information from the message queue; The path listening service updates the storage path set in the cache layer according to the received event content to ensure the consistency between the storage path in the cache layer and the actual storage path.

7. The storage method of the industrial city data based on the distributed storage architecture according to claim 6, characterized in that, The method further includes: Determine a corresponding verification chain for each storage path in the cache layer, and store the verification chain in the cache layer; Regularly calculate the actual verification chain of the actual storage path, compare it with the verification chain in the cache layer, and determine whether the storage path has changed according to the verification result.

8. A storage system for industrial city data based on a distributed storage architecture, characterized in that, For storing urban-industry data through a distributed storage architecture, the distributed storage architecture includes a data collection layer, a data storage layer, and a data access layer. The data storage layer includes a structured data repository and an unstructured data repository. The unstructured data repository includes storage pools corresponding to multiple different geographical location regions. The system includes: An acquisition module, configured to acquire the urban-industry data collected by the data collection layer, and respectively acquire the corresponding structured data and unstructured data from the urban-industry data; A determination module, configured to obtain the geographical location region to which the unstructured data belongs, and determine the target storage pool corresponding to the unstructured data from multiple storage pools according to the geographical location region to which it belongs, and store the unstructured data correspondingly in the target storage pool; A storage module, configured to obtain the metadata corresponding to the unstructured data, and store the metadata and the structured data in the structured database respectively.

9. A readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method described in any one of claims 1 to 7.

10. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the steps of the method described in any one of claims 1 to 7.

Images