Digital country data storage system based on cloud computing

Abstract

The invention discloses a data storage system of a digital country based on cloud computing, which comprises a client, a data transmission module and a cloud computing center, relates to the technical field of data storage, stores hot data on a high-performance hot data storage layer, ensures that frequently accessed data can be quickly responded, improves the system performance, and at the same time, stores the hot data on a high-performance hot data storage layer. The cold data is stored in a low-performance cold data storage layer, the storage cost is reduced, effective balance of performance and cost is achieved, the data is automatically migrated among different storage layers along with change of data access frequency, so that the performance and the cost are balanced, the frequently accessed data is migrated into a cache, the access speed of the data is increased, and the data access efficiency is improved. And establishing a relationship between a time interval and a data type of historical data, predicting data which are possibly accessed in a given time interval, adding the data into the cache, dynamically adjusting the cache, and ensuring the validity of the cache.

Description

Technical Field

[0001] The present invention relates to the technical field of data storage, and in particular to a data storage system for a digital village based on cloud computing. Background Art

[0002] Digital villages are an endogenous process of agricultural and rural modernization and transformation that is accompanied by the application of networking, informatization and digitization in the economic and social development of agriculture and rural areas, as well as the improvement of farmers' modern information skills. The emergence of cloud computing technology provides a new solution for digital rural data storage. Cloud computing has powerful computing and storage capabilities, which can realize centralized management and rapid access to data. At the same time, the cloud computing platform provides a variety of security measures, such as data encryption and access control, which can effectively ensure the security of data.

[0003] However, the current digital rural data storage system based on cloud computing still has some shortcomings. For example, first of all, there is a lack of effective classification storage strategy for different types of data, resulting in all data being stored on a single type of storage medium, which cannot balance performance and cost; in terms of data access, there is a lack of effective caching mechanism, and frequently accessed data cannot be cached to memory in time, resulting in slow data access speed and affecting user experience. At the same time, the cache content cannot be dynamically adjusted according to the data access mode. Summary of the invention

[0004] In order to solve the technical problems existing in the background technology, the present invention proposes a data storage system for a digital village based on cloud computing.

[0005] The present invention proposes a data storage system for a digital village based on cloud computing, comprising: a client, a data transmission module and a cloud computing center;

[0006] The client includes:

[0007] Digital rural data writing module: used to input agricultural production data, rural e-commerce data, rural governance data, rural culture and tourism data, and environmental monitoring data;

[0008] Data type setting module: when writing digital village data, it is used to automatically or manually set the data type to regular data, important data and particularly important data;

[0009] Access control module: using identity authentication and authorization management technology, only authenticated and authorized users can access data;

[0010] Data preprocessing module: used for data cleaning, data conversion, and data integration of digital village data, and transmitting the preprocessed data to the data transmission module;

[0011] Data transmission module: used to transmit the pre-processed data to the cloud computing center, and transmit the data from the cloud computing center to the client;

[0012] The cloud computing center includes:

[0013] Data storage module: including hot data storage layer, warm data storage layer and cold data storage layer, storing data in the hot data storage layer, warm data storage layer and cold data storage layer respectively according to the writing time, access frequency and data type of digital village data;

[0014] Data storage layer migration module: used to count and analyze the access frequency of digital village data, and automatically migrate data between hot data storage layer, warm data storage layer and cold data storage layer;

[0015] Data redundancy mechanism module: used to set up a copy for regular data and multiple copies for important data. Multiple copies of important data are stored on different cloud computing nodes. For particularly important data, erasure coding technology is used, and consistency checks and updates are performed on data copies regularly to ensure data consistency between copies. At the same time, a data recovery mechanism is established to quickly recover data from other copies in the event of node failure or data loss.

[0016] Data synchronization strategy module: used to establish a data synchronization mechanism to keep the data between each node of distributed storage consistent;

[0017] Data cache strategy module: used to predict the data that users may access based on their access patterns and historical data, and read this data from the data storage module into the cache in advance to reduce access latency.

[0018] Preferably, in the data storage module, a distributed file system and object storage technology are used to store the digital village data in a dispersed manner on multiple cloud computing nodes.

[0019] Preferably, in the data storage module, the hot data storage layer, the warm data storage layer and the cold data storage layer use storage media with different performances, wherein the performance of the storage media used in the hot data storage layer, the warm data storage layer and the cold data storage layer decreases in sequence.

[0020] Preferably, in the data storage module, the writing time of the digital village data is t:

[0021] Among the general data in the digital village data:

[0022] In the time interval 0<t≤a, the hot data storage layer is used for storage;

[0023] In the time interval a<t≤b, the warm data storage layer is used for storage;

[0024] In the time interval b<t, cold data storage layer is used for storage;

[0025] Important data in digital village data include:

[0026] In the time interval 0<t≤b, the hot data storage layer is used for storage;

[0027] In the time interval b<t≤c, the warm data storage layer is used for storage;

[0028] In the time interval c<t, cold data storage layer is used for storage;

[0029] Among the particularly important data in the digital village data:

[0030] In the time interval 0<t≤c, the hot data storage layer is used for storage;

[0031] In the time interval a<t≤d, the warm data storage layer is used for storage;

[0032] In the time interval d<t, cold data storage layer is used for storage;

[0033] 0<a<b<c<d.

[0034] Preferably, in the data storage layer migration module, the data storage layer migration is set as follows:

[0035] Establish data access monitoring for digital village data, record the access time and number of each piece of data stored in the data storage module, regularly scan and count the data in the data storage module, and screen out data that meets the migration conditions;

[0036] If the number of accesses to data exceeds the set threshold within a certain period of time, the migration condition is met and the data migration task is triggered. The data migration task migrates the selected data from the cold data storage layer to the warm data storage layer, or migrates the selected data from the warm data storage layer to the hot data storage layer, and updates the data storage location information. At the same time, the data is marked as migration data.

[0037] For the marked migration data, keep backups for a certain period of time in the original storage layer to prevent problems during the migration process and allow rollback.

[0038] Continuously monitor the access to migrated data. If the access frequency of migrated data decreases within a certain period of time, gradually migrate the data back to a lower-level storage layer.

[0039] Preferably, in the data synchronization strategy module:

[0040] For the digital rural data in the hot data storage layer, when the parameters of the digital rural data change, a real-time synchronization strategy is adopted to immediately synchronize the new data to all relevant nodes, ensuring the rapid update and consistency of data. For example, when the environmental parameters of agricultural facilities change, the new data is immediately synchronized to all relevant nodes.

[0041] For digital rural data in the warm data storage layer, when the parameters of digital rural data change, a regular synchronization strategy is adopted to regularly synchronize new data to all relevant nodes; for example, synchronize warm data every day or every week to ensure data consistency and availability. For sales trend data of rural e-commerce, village affairs management data of rural governance, etc., regular synchronization can meet most query and analysis needs.

[0042] For the digital rural data in the cold data storage layer, when the parameters of the digital rural data change, an asynchronous synchronization strategy is adopted to synchronize the data when it is requested or at a specific time interval. For example, to reduce the load and cost of the system. At the same time, a lower priority can be used for the synchronization of cold data to ensure that the synchronization of hot and warm data is prioritized.

[0043] Preferably, in the data cache strategy module:

[0044] The settings are as follows:

[0045] Assume that the data in the data set is X = {x 1 , x 2 , ..., x n}, where x j In, j = 1, 2, ..., n, representing different data;

[0046] The time interval is T = {t 1 , t 2 , ..., t m}, where t i In , i = 1, 2, ..., m, representing different time periods;

[0047] The goal is to establish a relationship between time intervals and data based on known historical data, and predict the data that may be accessed in a given time interval;

[0048] According to the principle of Naive Bayes algorithm:

[0049] Bayes’ Theorem: Where P(Y|X) is the posterior probability of category Y given feature X; P(X|Y) is the probability of feature X given category Y; P(Y) is the prior probability of category Y; P(X) is the probability of feature X;

[0050] Here, the data is considered as category Y and the time interval is considered as feature X;

[0051] Algorithm steps:

[0052] Record the access to different data in different time intervals in the past period of time to form a data set D = {(t i , x j )}, where (t i , x j ) represents the time interval t i Accessed data x j , D represents the collection of historical data;

[0053] Calculate the prior probability P(x j ), P(x j ) represents data x j The prior probability of data x j The probability of appearing in the entire data set is calculated as where count(x j ) is the data x j is the number of occurrences, |D| is the size of the data set;

[0054] Calculate the conditional probability P(t i |x j ), which means that when the known data is x j In the case of i The conditional probability of each data x j , count the number of occurrences in different time intervals under the data, and the calculation method is:

[0055] Among them, count((t i , x j )) is in the data x j Next time interval t i Number of occurrences;

[0056] For a given time interval t new , calculate the posterior probability P(x j |t new );

[0057] P(x j |t new ) represents the given new time interval t new In the case of x j The posterior probability of is used to predict the data that may be accessed in the new time interval by calculating this probability;

[0058] According to the Naive Bayes hypothesis, Since P(t new ) is the same for all data and can be ignored;

[0059] Therefore, P(x j |t new )≈P(t new |x j )P(x j );

[0060] After calculating the posterior probability of all data, the data with the highest probability is selected as the prediction result, that is, the data with the highest probability is selected as the data that may be accessed in a given new time interval after calculation;

[0061]

[0062] in, Represents the predicted value at a given new time interval t new The data most likely to be accessed, Indicates that in all different data x j In the above example, find the value such that P(x j |t new )The data x with the largest probability value j , here we are calculating the posterior probability of various data in a given new time interval, and then selecting the data with the highest probability as the prediction result;

[0063] Through the above steps, the Naive Bayes algorithm can be used to predict the data that may be accessed in a given time interval based on historical data, thereby realizing the pre-fetching of digital village data in the cache;

[0064] At the same time, the data is marked as cache data;

[0065] For the marked cache data, keep a backup for a certain period of time in the original storage layer;

[0066] Continuously monitor the access to cached data. When the number of accesses to cached data is 0 within a certain period of time, migrate the cached data to the hot data storage layer in the data storage module, and update the write time of the cached data. Updating the write time of the cached data here means that the data is considered to have re-entered the system, and the time required for data storage layer migration is recalculated according to its original data type.

[0067] A method for using a digital village data storage system based on cloud computing, comprising the following steps: inputting agricultural production data, rural e-commerce data, rural governance data, rural culture and tourism data, and environmental monitoring data through a digital village data writing module on a client;

[0068] At the same time, the data type setting module is used to automatically or manually set the data type to regular data, important data and particularly important data when writing digital village data;

[0069] The access control module uses identity authentication and authorization management technology to ensure that only authenticated and authorized users can access data;

[0070] The data preprocessing module then cleans, converts and integrates the digital village data and transmits it to the data transmission module;

[0071] The data transmission module transmits the pre-processed data to the cloud computing center, and transmits the data from the cloud computing center to the client;

[0072] In the cloud computing center, the data storage module stores the data in the hot data storage layer, the warm data storage layer, and the cold data storage layer according to the writing time and data type of the digital village data;

[0073] The data storage module uses distributed file system and object storage technology to store digital village data in multiple cloud computing nodes;

[0074] The hot data storage layer, warm data storage layer and cold data storage layer in the data storage module use storage media with different performances, and the performance of use decreases in sequence;

[0075] The data storage layer migration module counts and analyzes the access frequency of digital village data, and automatically migrates data between the hot data storage layer, warm data storage layer, and cold data storage layer;

[0076] In the data storage layer migration module, data access monitoring is established for digital village data, the access time and access frequency of each piece of data are recorded, the data in the data storage module are scanned and counted regularly, and data that meets the migration conditions is screened out; within a certain period of time, if the number of data accesses exceeds the set threshold, the migration conditions are met, and the data migration task is triggered to migrate the selected data from the cold data storage layer to the warm data storage layer, or migrate the selected data from the warm data storage layer to the hot data storage layer, and update the data storage location information, and mark the data as migration data; for the marked migration data, a backup is retained in the original storage layer for a certain period of time, and the access to the migration data is continuously monitored. If the access frequency of the migration data decreases within a certain period of time, the data will be gradually migrated back to the lower-level storage layer;

[0077] The data redundancy mechanism module sets up multiple copies of important data, uses multi-copy storage technology to store the copies on different cloud computing nodes, uses erasure coding technology for particularly important data, and regularly checks and updates the consistency of data copies, while establishing a data recovery mechanism;

[0078] The data synchronization strategy module establishes a data synchronization mechanism to keep the data between each node of the distributed storage consistent. For the digital rural data in the hot data storage layer, when the parameters of the digital rural data change, a real-time synchronization strategy is adopted to immediately synchronize the new data to all related nodes; for the digital rural data in the warm data storage layer, when the parameters of the digital rural data change, a periodic synchronization strategy is adopted to regularly synchronize the new data to all related nodes; for the digital rural data in the cold data storage layer, when the parameters of the digital rural data change, an asynchronous synchronization strategy is adopted to synchronize when the data is requested or within a specific time interval, and the synchronization priority is lower than that of hot data and warm data;

[0079] The data cache strategy module predicts the data that users may access based on their access patterns and historical data, and reads the data from the data storage module into the cache in advance. For data that has reached a threshold number of accesses within a certain period of time, it reads it into the cache;

[0080] At the same time, the data is marked as cached data;

[0081] For the marked cache data, keep a backup for a certain period of time in the original storage layer;

[0082] The access situation of the migrated data is continuously monitored. When the number of accesses to the cached data is 0 within a certain period of time, the cached data is migrated to the hot data storage layer in the data storage module, and the write time of the cached data is updated.

[0083] In the present invention, the proposed digital village data storage system based on cloud computing has the following beneficial technical effects:

[0084] 1. According to the writing time and data type of digital village data, the data is stored in the hot data storage layer, warm data storage layer and cold data storage layer respectively. The performance of the storage media used in the hot data storage layer, warm data storage layer and cold data storage layer decreases in sequence. Hot data is stored in the high-performance hot data storage layer to ensure that frequently accessed data can be responded to quickly and improve system performance. At the same time, cold data is stored in the low-performance cold data storage layer to reduce storage costs and achieve an effective balance between performance and cost. A data migration mechanism is established to automatically migrate data between different storage layers as the frequency of data access changes to balance performance and cost.

[0085] 2. The data cache strategy module is set up to cache frequently accessed data into the cache to improve the data access speed. It can also establish a relationship between the time interval and the data type of historical data, predict the data that the user may access in a given time interval, and read this data from the storage medium into the cache in advance to reduce access delays, dynamically adjust the cache strategy, and ensure the effectiveness of the cache.

[0086] 3. The data storage module is used to store data in the hot data storage layer, warm data storage layer and cold data storage layer respectively. In the hot data storage layer, warm data storage layer and cold data storage layer, the distributed file system and object storage technology are used to store the digital village data in multiple cloud computing nodes to improve the reliability and availability of the data. By distributing the data on different nodes, single point failures can be avoided to ensure that the data can still be accessed when problems occur in some nodes. Tiered storage improves the access speed of data by storing frequently accessed hot data on the hot data storage layer, and distributed storage can store data in multiple nodes in a dispersed manner, further improving the parallel access capability of data, thereby reducing access delays and improving the overall performance of the system.

[0087] 4. Setting up the data redundancy mechanism module. Important data is stored in multiple copies on different cloud computing nodes. Even if a node fails, data copies can be obtained from other nodes to ensure the availability of important data. This is crucial for important data in digital villages, reducing the risk of data loss due to hardware failure, network problems or other unexpected situations. The use of erasure coding technology for particularly important data further enhances the protection of particularly important data. Erasure coding can be restored through the remaining data when some data is lost, thereby improving the data's fault tolerance.

[0088] 5. The setting of the data synchronization strategy module adopts different synchronization strategies for data in different storage layers, achieving a balance between data consistency and system performance. The hot data storage layer adopts a real-time synchronization strategy to ensure the immediate update and consistency of key data, meeting the application scenarios with high real-time requirements. The regular synchronization strategy of the warm data storage layer reduces the occupation of system resources by synchronization operations while ensuring relatively timely data updates. It is suitable for some businesses that do not have high real-time requirements but still require a certain degree of data consistency. The asynchronous synchronization strategy of the cold data storage layer ensures that data can be synchronized when needed without affecting system performance, reduces the frequency of synchronization operations, and saves system resources.

[0089] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] Figure 1 The figure is a functional block diagram of the system of the present invention. DETAILED DESCRIPTION

[0091] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar symbols throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0092] like Figure 1 A data storage system for a digital village based on cloud computing is shown, comprising: a client, a data transmission module and a cloud computing center;

[0093] The client includes:

[0094] Digital rural data writing module: used to input agricultural production data, rural e-commerce data, rural governance data, rural culture and tourism data, and environmental monitoring data;

[0095] Data type setting module: when writing digital village data, it is used to automatically or manually set the data type to regular data, important data and particularly important data;

[0096] Access control module: using identity authentication and authorization management technology, only authenticated and authorized legitimate users can access data; implementing strict access control policies to limit user access rights to data.

[0097] For example, only specific village managers can access sensitive financial data.

[0098] Data retrieval module: B-tree index and hash index are used to provide data retrieval. B-tree index and hash index are used to establish appropriate indexes for data and improve data retrieval speed. You can choose the appropriate index type, such as B-tree index, hash index, etc., according to the characteristics of the data and query requirements.

[0099] Regularly optimize and maintain the index to ensure its efficiency. You can improve the performance of the index by rebuilding the index and cleaning up invalid indexes.

[0100] Data preprocessing module: used for data cleaning, data conversion, and data integration of digital village data, and transmitting the preprocessed data to the data transmission module;

[0101] Data transmission module: used to transmit pre-processed data to the cloud computing center, and transmit the data from the cloud computing center to the client; the client retrieves the data stored in the cloud computing center through the data transmission module;

[0102] The cloud computing center includes:

[0103] Data storage module: including hot data storage layer, warm data storage layer and cold data storage layer.

[0104] In the data storage module, the writing time of digital village data is assumed to be t:

[0105] Among the general data in the digital village data:

[0106] In the time interval 0<t≤a, the hot data storage layer is used for storage;

[0107] In the time interval a<t≤b, the warm data storage layer is used for storage;

[0108] In the time interval b<t, cold data storage layer is used for storage;

[0109] Important data in digital village data include:

[0110] In the time interval 0<t≤b, the hot data storage layer is used for storage;

[0111] In the time interval b<t≤c, the warm data storage layer is used for storage;

[0112] In the time interval c<t, cold data storage layer is used for storage;

[0113] Particularly important data in digital village data include:

[0114] In the time interval 0<t≤c, the hot data storage layer is used for storage;

[0115] In the time interval a<t≤d, the warm data storage layer is used for storage;

[0116] In the time interval d<t, cold data storage layer is used for storage;

[0117] In an optional embodiment, it is set:

[0118] a=3(months)

[0119] b=12(months)

[0120] c=24(months)

[0121] d = 36 (months).

[0122] According to the writing time and data type of digital village data, the data is stored in the hot data storage layer, the warm data storage layer and the cold data storage layer respectively;

[0123] In the data storage module, the hot data storage layer, the warm data storage layer and the cold data storage layer use storage media with different performances, wherein the performance of the storage media used by the hot data storage layer, the warm data storage layer and the cold data storage layer decreases in sequence, for example: the storage medium used by the hot data storage layer is a high-speed solid-state hard disk, the storage medium used by the warm data storage layer is an ordinary hard disk, and the storage medium used by the cold data storage layer is a low-cost large-capacity hard disk or a tape library;

[0124] According to the writing time and data type of digital village data, the data is stored in the hot data storage layer, warm data storage layer and cold data storage layer respectively. The performance of the storage media used in the hot data storage layer, warm data storage layer and cold data storage layer decreases in sequence. Hot data is stored in the high-performance hot data storage layer to ensure that frequently accessed data can be responded to quickly and improve system performance. At the same time, cold data is stored in the low-performance cold data storage layer to reduce storage costs and achieve an effective balance between performance and cost.

[0125] In the data storage module, distributed file system and object storage technology are used to store digital village data in multiple cloud computing nodes to improve data reliability and availability.

[0126] For agricultural production data, the data is stored in multiple cloud computing nodes according to different geographical locations and crop types. For example, meteorological data of different regions is stored on nodes close to the region, and growth data of specific crops is stored on nodes that specialize in processing data for that crop. This can increase the speed of data access and facilitate regional agricultural production analysis.

[0127] Rural e-commerce data is distributed and stored according to the types of agricultural products and sales areas. For example, data on specific agricultural products sold in the same area is stored on nodes near the area to quickly process orders and analyze market demand. At the same time, e-commerce platform user data is dispersed and stored on multiple nodes to improve the response speed of user access.

[0128] Rural governance data can be distributed and stored according to different governance areas and administrative divisions. For example, the village affairs management data of a village can be stored on a node close to the village, and the government service processing data of a specific area can be stored on a node in the area, so as to improve the processing efficiency of government services and the security of data.

[0129] Rural culture and tourism data can be distributed and stored according to the geographical location of cultural resources and tourist attractions. For example, the cultural resource data of a traditional village can be stored on a node close to the village, and the visitor data of a specific tourist attraction can be stored on a node near the attraction, so as to quickly provide tourism information services and conduct tourism management.

[0130] Environmental monitoring data can be distributed and stored according to the location of monitoring sites. For example, water quality data of a river monitoring point can be stored on a node close to the river, and air quality monitoring data of a region can be stored on a node in the region, so as to conduct environmental monitoring and early warning in a timely manner.

[0131] The data storage module is used to store data in a hot data storage layer, a warm data storage layer, and a cold data storage layer respectively. In addition, in the hot data storage layer, the warm data storage layer, and the cold data storage layer, a distributed file system and object storage technology are used to disperse and store digital village data on multiple cloud computing nodes to improve data reliability and availability. By distributing data on different nodes, single point failures can be avoided to ensure that data can still be accessed when problems occur in some nodes. Tiered storage improves data access speed by storing frequently accessed hot data on the hot data storage layer, and distributed storage can disperse and store data on multiple nodes, further improving the parallel access capability of data, thereby reducing access delays and improving the overall performance of the system.

[0132] Data storage layer migration module:

[0133] Establish a data migration service to automatically migrate data between different storage tiers based on statistics and analysis of data access frequency.

[0134] In the data storage layer migration module, the following is a mechanism design for migrating data from the cold data storage layer to the warm data storage layer and then to the hot data storage layer based on access time and access count:

[0135] Establish data access monitoring to record the access time and access count for each piece of data stored in the cold data storage layer;

[0136] Scan and count the data in the cold data storage layer regularly (for example, every day) to filter out data that meets the migration conditions.

[0137] Migration conditions:

[0138] Number of accesses: If the number of accesses to the data exceeds a set threshold (e.g., 5 times) within a certain time period (e.g., one month).

[0139] Access time: If the data has been accessed recently (for example, within the last week).

[0140] Migration process:

[0141] When the data meets the migration conditions, the data migration task is triggered.

[0142] The data migration service reads the selected data from the storage media (such as a large-capacity hard disk or a tape library) of the cold data storage layer.

[0143] The data is then written to a storage medium (such as a common hard disk) in the warm data storage layer, and the storage location information of the data is updated.

[0144] At the same time, the status of the data is updated in the data management system, marking that it has been migrated from the cold data storage layer to the warm data storage layer.

[0145] Migration conditions and process from warm data storage layer to hot data storage layer;

[0146] Migration conditions:

[0147] Number of accesses: Within a short period of time (such as a week), the number of accesses to the data frequently exceeds a higher threshold (such as 20 times).

[0148] Access time: If the data has been accessed multiple times in the last few days.

[0149] Migration process:

[0150] Similar to the migration from the cold data storage layer to the warm data storage layer, when the warm data meets the migration conditions, the data migration task is triggered.

[0151] The data migration service reads data from the warm data storage tier and writes it to the high-speed SSDs in the hot data storage tier.

[0152] Update the storage location information and status of the data, marking it as having been migrated from the warm data storage layer to the hot data storage layer.

[0153] Management after data migration:

[0154] For the migrated data, keep backups for a certain period of time in the original storage layer to allow rollback in case of problems during the migration process.

[0155] Continuously monitor the access status of data migrated to the new layer. If the access frequency decreases, the data will be gradually migrated back to the lower-level storage layer according to the reverse migration conditions to ensure the rational use of storage resources.

[0156] Establish a data migration mechanism to automatically migrate data between different storage tiers as data access frequency changes to balance performance and cost.

[0157] Data compression and deduplication module:

[0158] Use lossless data compression algorithms to compress stored data and reduce storage space usage. Especially for data types such as text, images, and videos, it can significantly reduce storage costs;

[0159] Implement data deduplication technology to perform real-time deduplication when data is written, and regularly perform deduplication processing on stored data to remove duplicate data blocks and further improve storage efficiency;

[0160] Existing lossless data compression algorithms include Huffman coding algorithm, arithmetic coding algorithm, LZ77 data compression algorithm, and LZ78 data compression algorithm.

[0161] Data security module:

[0162] Symmetric and asymmetric encryption technologies are used to encrypt data stored in the data storage module to ensure the confidentiality of the data during storage;

[0163] Used to conduct security audits on access to data storage modules and record user operation logs and access history. For example, record the time and content of each user's access to specific data;

[0164] Monitor the security status of data storage systems in real time, and use security information and event management (SIEM) systems to promptly detect and handle security vulnerabilities and attack behaviors.

[0165] Data redundancy mechanism module: used to set up a copy for regular data, use multi-copy storage technology for important data, set up multiple copies for important data, and store these copies on different cloud computing nodes. For example, for key agricultural production data (such as key crop growth stage data for the current season), popular sales data for rural e-commerce, important government service data for rural governance, real-time data on popular attractions in rural culture and tourism, and key indicator data for environmental monitoring, set up at least three copies to ensure that the data is still available when some nodes fail.

[0166] For some particularly important data, erasure coding technology can be used. For example, for financial revenue and expenditure data of rural governance and long-term historical data of environmental monitoring, erasure coding technology can be used to restore data in the event of a certain degree of data loss, thereby improving data reliability.

[0167] Regularly check and update the data copies to ensure data consistency between copies. At the same time, establish a data recovery mechanism to quickly recover data from other copies in the event of node failure or data loss.

[0168] Establish a data redundancy mechanism, use multi-copy storage and erasure coding technology to ensure that data is still accessible when some nodes fail.

[0169] Data synchronization strategy module: establishes a data synchronization mechanism to ensure that the data between each node of the distributed storage remains consistent. In the data synchronization strategy module:

[0170] For the digital rural data in the hot data storage layer, when the parameters of the digital rural data change, a real-time synchronization strategy is adopted to immediately synchronize the new data to all related nodes, ensuring the rapid update and consistency of data. For example, when the environmental parameters of agricultural facilities change, the new data is immediately synchronized to all related nodes.

[0171] For digital rural data in the warm data storage layer, when the parameters of digital rural data change, a regular synchronization strategy is adopted to regularly synchronize new data to all relevant nodes; for example, synchronize warm data every day or every week to ensure data consistency and availability. For sales trend data of rural e-commerce and village affairs management data of rural governance, regular synchronization can meet most query and analysis needs.

[0172] For the digital rural data in the cold data storage layer, when the parameters of the digital rural data change, an asynchronous synchronization strategy is adopted to synchronize the data when it is requested or at a specific time interval. For example, to reduce the load and cost of the system. At the same time, a lower priority can be used for the synchronization of cold data to ensure that the synchronization of hot and warm data is prioritized.

[0173] Different synchronization strategies are adopted for data in different storage layers to achieve a balance between data consistency and system performance. The hot data storage layer adopts a real-time synchronization strategy to ensure the immediate update and consistency of key data, meeting the application scenarios with high real-time requirements. The regular synchronization strategy of the warm data storage layer reduces the occupation of system resources by synchronization operations while ensuring relatively timely data updates. It is suitable for some businesses that do not have high real-time requirements but still require a certain degree of data consistency. The asynchronous synchronization strategy of the cold data storage layer ensures that data can be synchronized when needed without affecting system performance, reduces the frequency of synchronization operations, and saves system resources.

[0174] Data caching strategy module: Introducing cache into the data storage system to cache frequently accessed data to improve data access speed. The cache strategy can be dynamically adjusted according to the access mode and popularity of the data to ensure the effectiveness of the cache.

[0175] For data that has been accessed a certain number of times within a certain period of time, it is read into the cache. For example, data that has been accessed 20 times within 3 days can be read into the cache.

[0176] At the same time, the data is marked as cache data;

[0177] For the marked cache data, keep a backup for a certain period of time in the original storage layer;

[0178] Continuously monitor the access to the migrated data. When the number of accesses to the cached data is 0 within a certain period of time, migrate the cached data to the hot data storage layer in the data storage module, and update the write time of the cached data. Updating the write time of the cached data here means that the data is considered to have re-entered the system, and the time required for data storage layer migration is recalculated according to its original data type.

[0179] By utilizing the elastic computing power of cloud computing, the number and configuration of cache nodes are automatically adjusted according to changes in access load to meet different performance requirements.

[0180] Based on the user's access pattern and historical data, predict the data that the user may access, and read this data from the storage medium into the cache in advance to reduce access latency.

[0181] The data cache strategy module is set up to cache frequently accessed data into the cache to improve the data access speed. It can also establish relationships between time intervals and data types for historical data, predict the data that users may access in a given time interval, and read this data from the storage medium into the cache in advance to reduce access delays, dynamically adjust the cache strategy, and ensure the effectiveness of the cache.

[0182] The goal is to establish a relationship between time intervals and data types based on known historical data and predict the type of data that may be accessed in a given time interval.

[0183] A method for data pre-fetching prediction based on the naive Bayes algorithm:

[0184] In the data cache strategy module:

[0185] The settings are as follows:

[0186] Assume that the data in the data set is X = {x 1 , x 2 , ..., x n}, where x j In, j = 1, 2, ..., n, representing different data;

[0187] The time interval is T = {t 1 , t 2 , ..., t m}, where t i In , i = 1, 2, ..., m, representing different time periods;

[0188] The goal is to establish a relationship between time intervals and data based on known historical data, and predict the data that may be accessed in a given time interval;

[0189] According to the principle of Naive Bayes algorithm:

[0190] Bayes’ Theorem: Where P(Y|X) is the posterior probability of category Y given feature X; P(X|Y) is the probability of feature X given category Y; P(Y) is the prior probability of category Y; P(X) is the probability of feature X;

[0191] Here, the data is considered as category Y and the time interval is considered as feature X;

[0192] Algorithm steps:

[0193] Record the access to different data in different time intervals in the past period of time to form a data set D = {(t i , x j )}, where (t i , x j ) represents the time interval t i Accessed data x j , D represents the collection of historical data;

[0194] Calculate the prior probability P(x j ), P(x j ) represents the data x j The prior probability of data x j The probability of appearing in the entire data set is calculated as where count(x j ) is the data x j is the number of occurrences, |D| is the size of the data set;

[0195] Calculate the conditional probability P(t i |x j ), which means that when the known data is x j In the case of i The conditional probability of each data x j , count the number of occurrences in different time intervals under the data, and the calculation method is:

[0196] Among them, count((t i , x j )) is in the data x j Next time interval t i Number of occurrences;

[0197] For a given time interval t new , calculate the posterior probability P(x j |t new );

[0198] P(x j|t new ) represents the given new time interval t new In the case of x j The posterior probability of is used to predict the data that may be accessed in the new time interval by calculating this probability;

[0199] According to the Naive Bayes hypothesis, Since P(t new ) is the same for all data and can be ignored;

[0200] Therefore, P(x j |t new )≈P(t new |x j )P(x j );

[0201] After calculating the posterior probability of all data, the data with the highest probability is selected as the prediction result, that is, the data with the highest probability is selected as the data that may be accessed in a given new time interval after calculation;

[0202]

[0203] in, Represents the predicted value at a given new time interval t new The data most likely to be accessed, Indicates that in all different data x j In the above example, find the value such that P(x j |t new )The data x with the largest probability value j , here we are calculating the posterior probability of various data in a given new time interval, and then selecting the data with the highest probability as the prediction result;

[0204] Through the above steps, the Naive Bayes algorithm can be used to predict the data that may be accessed in a given time interval based on historical data, thereby realizing the pre-fetching of digital village data in the cache;

[0205] At the same time, the data is marked as cache data;

[0206] For the marked cache data, keep a backup for a certain period of time in the original storage layer;

[0207] The access to the cached data is continuously monitored. When the number of accesses to the cached data is 0 within a certain period of time, the cached data is migrated to the hot data storage layer in the data storage module, and the write time of the cached data is updated.

[0208] A method for using a data storage system of a digital village based on cloud computing comprises the following steps:

[0209] Input agricultural production data, rural e-commerce data, rural governance data, rural culture and tourism data, and environmental monitoring data through the digital rural data writing module on the client;

[0210] At the same time, the data type setting module is used to automatically or manually set the data type to regular data, important data and particularly important data when writing digital village data;

[0211] The access control module uses identity authentication and authorization management technology to ensure that only authenticated and authorized users can access data;

[0212] The data preprocessing module then cleans, converts and integrates the digital village data and transmits it to the data transmission module;

[0213] The data transmission module transmits the pre-processed data to the cloud computing center, and transmits the data from the cloud computing center to the client;

[0214] In the cloud computing center, the data storage module stores the data in the hot data storage layer, the warm data storage layer, and the cold data storage layer according to the writing time and data type of the digital village data;

[0215] The data storage module uses distributed file system and object storage technology to store digital village data in multiple cloud computing nodes;

[0216] The hot data storage layer, warm data storage layer and cold data storage layer in the data storage module use storage media with different performances, and the performance of use decreases in sequence;

[0217] In the data storage module, the writing time of digital village data is set to t:

[0218] For regular data in digital village data, the hot data storage layer is used for storage in the time interval 0<t≤a, the warm data storage layer is used for storage in the time interval a<t≤b, and the cold data storage layer is used for storage in the time interval b<t;

[0219] For important data in the digital village data, the hot data storage layer is used for storage in the time interval 0<t≤b, the warm data storage layer is used for storage in the time interval b<t≤c, and the cold data storage layer is used for storage in the time interval c<t;

[0220] For particularly important data in the digital village data, the hot data storage layer is used for storage in the time interval 0<t≤c, the warm data storage layer is used for storage in the time interval a<t≤d, and the cold data storage layer is used for storage in the time interval d<t;

[0221] 0<a<b<c<d;

[0222] The data storage layer migration module counts and analyzes the access frequency of digital village data, and automatically migrates data between the hot data storage layer, warm data storage layer, and cold data storage layer;

[0223] In the data storage layer migration module, data access monitoring is established for digital village data, the access time and access frequency of each piece of data are recorded, the data in the data storage module are scanned and counted regularly, and data that meets the migration conditions is screened out; within a certain period of time, if the number of data accesses exceeds the set threshold, the migration conditions are met, and the data migration task is triggered to migrate the selected data from the cold data storage layer to the warm data storage layer, or migrate the selected data from the warm data storage layer to the hot data storage layer, and update the data storage location information, and mark the data as migration data; for the marked migration data, a backup is retained in the original storage layer for a certain period of time, and the access to the migration data is continuously monitored. If the access frequency of the migration data decreases within a certain period of time, the data will be gradually migrated back to the lower-level storage layer;

[0224] The data redundancy mechanism module sets up multiple copies of important data, uses multi-copy storage technology to store the copies on different cloud computing nodes, uses erasure coding technology for particularly important data, and regularly checks and updates the consistency of data copies, while establishing a data recovery mechanism;

[0225] The data synchronization strategy module establishes a data synchronization mechanism to keep the data consistent between each node of the distributed storage. For the digital rural data in the hot data storage layer, when the parameters of the digital rural data change, a real-time synchronization strategy is adopted to immediately synchronize the new data to all relevant nodes; for the digital rural data in the warm data storage layer, when the parameters of the digital rural data change, a periodic synchronization strategy is adopted to regularly synchronize the new data to all relevant nodes; for the digital rural data in the cold data storage layer, when the parameters of the digital rural data change, an asynchronous synchronization strategy is adopted to synchronize the data when it is requested or at a specific time interval, and the synchronization priority is lower than that of hot data and warm data.

[0226] The data cache strategy module predicts the data that users may access based on their access patterns and historical data, and reads the data from the data storage module into the cache in advance. For data that has reached a threshold number of accesses within a certain period of time, it reads it into the cache to reduce access delays.

[0227] At the same time, the data is marked as cached data;

[0228] For the marked cache data, keep a backup for a certain period of time in the original storage layer;

[0229] The access situation of the migrated data is continuously monitored. When the number of accesses to the cached data is 0 within a certain period of time, the cached data is migrated to the hot data storage layer in the data storage module, and the write time of the cached data is updated.

[0230] The system optimization solution of this patent application provides an efficient and reliable data storage and access solution for the construction of digital villages, which will help promote the intelligentization of agricultural production, the development of rural e-commerce, the modernization of rural governance, the prosperity of rural culture and tourism, and the precision of environmental monitoring, and comprehensively promote the sustainable development of digital villages.

[0231] Specifically, as an example:

[0232] Digital village data writing module:

[0233] Agricultural production data include:

[0234] Meteorological data: including temperature, humidity, rainfall, wind direction and speed, are crucial for agricultural production arrangements and disaster warnings.

[0235] Soil data: including soil pH, fertility status, and water content, used to guide reasonable fertilization and irrigation.

[0236] Crop growth data: planting area, variety information; crop growth stage data, including the characteristics of different growth stages such as emergence, tillering and jointing; crop disease and pest data, recording the type, occurrence time and severity of diseases and pests, so as to take timely prevention and control measures.

[0237] Agricultural facility operation data: temperature, humidity, light environment parameters of greenhouses, and the operating status of ventilation, irrigation, and fertilization equipment; flow and pressure data of the irrigation system, used to monitor irrigation effects and water resource utilization; operating time, operating area, and fuel consumption data of agricultural machinery equipment, which can be used to evaluate agricultural machinery efficiency and maintenance needs.

[0238] Rural e-commerce data includes:

[0239] Agricultural product sales data: including the types, quantities, prices, and sales areas of agricultural products sold, which can be used to analyze market demand and sales trends.

[0240] E-commerce platform user data: including user browsing history, purchasing behavior, and evaluation information, which is used for precision marketing and service improvement.

[0241] Logistics data: including shipping location, receiving location, transportation time, and logistics cost, in order to optimize logistics distribution efficiency.

[0242] Rural governance data include:

[0243] Population data: household registration population information, including name, gender, age, and ID number; permanent population size and floating population information, used for social management and public service planning.

[0244] Village affairs management data: financial revenue and expenditure data, recording the village collective’s income sources and expenditure items; public facilities construction and maintenance data, including the construction progress and maintenance of roads, bridges, and water conservancy facilities; government service processing data, including processing matters, processing time, and processing results, in order to improve the efficiency and transparency of government services.

[0245] Public security monitoring data: video surveillance image data and alarm information, used to maintain rural security.

[0246] Rural culture and tourism data include:

[0247] Cultural resource data: including the location, characteristics, and protection status of traditional villages, ancient buildings, and folk culture.

[0248] Tourism data: number of tourists, place of origin, length of stay; evaluation and feedback data of tourist attractions, used to improve the quality of tourism services and promote tourism resources.

[0249] Environmental monitoring data include:

[0250] Water quality data: including water quality parameters of rivers, lakes, and drinking water sources, including pH, dissolved oxygen, and chemical oxygen demand.

[0251] Air quality data: Monitor air quality indicators in rural areas, including PM2.5, PM10, sulfur dioxide, and nitrogen oxides.

[0252] Ecosystem data: Record the biodiversity, vegetation cover, and wildlife species and numbers of rural ecosystems.

[0253] It should be noted that in this application, the hierarchical storage architecture and distributed storage of the data storage module do not conflict. The hierarchical storage architecture refers to the hot data storage layer, the warm data storage layer and the cold data storage layer. Distributed storage refers to the use of distributed file systems and object storage technologies to store digital rural data in multiple cloud computing nodes. The hierarchical storage architecture and distributed storage can cooperate with each other to provide a more efficient and reliable data storage solution for the digital rural data storage system based on cloud computing.

[0254] The different focuses of the two:

[0255] The tiered storage architecture focuses on classified storage based on the access frequency and importance of data to balance storage costs and performance requirements. It divides data into hot data, warm data, and cold data, and stores them on storage media with different performance, thereby achieving optimized configuration of storage resources.

[0256] For example, for meteorological data in agricultural production data, recent weather forecasts are stored as hot data on high-speed solid-state hard drives to meet farmers' needs for fast access to real-time meteorological information; historical meteorological data from the past few months are stored as warm data on ordinary hard drives, which can be retrieved more quickly when periodic analysis is required; and meteorological data from historical years are stored as cold data on low-cost large-capacity hard drives or tape libraries, which are only occasionally accessed during long-term research.

[0257] Distributed storage focuses on distributing data across multiple physical nodes to improve data reliability and availability. By distributing data across different nodes, single point failures can be avoided and data can still be accessed when problems occur on some nodes.

[0258] For example, for rural e-commerce data, the data is stored in multiple cloud computing nodes according to the type of agricultural products and sales areas. In this way, even if a node fails, it will not affect the data availability of the entire system. At the same time, distributed storage can also increase the access speed of data because data can be read from the node closest to the user.

[0259] The synergy between the two:

[0260] Performance optimization:

[0261] The tiered storage architecture improves data access speed by storing frequently accessed hot data on high-speed storage media. Distributed storage can store data in multiple nodes to further improve the parallel access capability of data, thereby reducing access delays and improving the overall performance of the system.

[0262] For example, in the rural governance data, the data related to the government service items currently being processed are stored as hot data on high-speed solid-state hard drives, and these data are dispersed and stored on multiple nodes through distributed storage. In this way, different government service processing personnel can quickly access these data at the same time, improving the efficiency of government services.

[0263] Reliability Improvements:

[0264] Distributed storage uses data redundancy mechanisms to ensure that data can still be accessed when some nodes fail, improving system reliability. The tiered storage architecture can store important data on higher-performance storage media, reducing the risk of data loss.

[0265] For example, key indicator data in environmental monitoring data can be stored as hot data on high-speed solid-state drives to facilitate rapid environmental warnings; multiple copies can also be set up through distributed storage to ensure that the data is still available when a node fails.

[0266] Cost-effectiveness:

[0267] The tiered storage architecture can select the appropriate storage medium according to the access frequency and importance of data, thus reducing storage costs. Distributed storage can utilize the elastic computing power of cloud computing to dynamically adjust storage resources according to actual needs, thus avoiding resource waste.

[0268] For example, in agricultural production data, soil data from historical years is stored as cold data in low-cost large-capacity hard disks or tape libraries, which reduces storage costs. At the same time, distributed storage can dynamically adjust the number and configuration of nodes according to actual storage needs to improve resource utilization.

[0269] To sum up, hierarchical storage architecture and distributed storage can cooperate with each other in the digital village data storage system based on cloud computing and play a role together to provide better data support for the construction of digital villages.

[0270] As a supplementary note: the data retrieval speed of the cache and hot data storage layer is faster than that of the cache.

[0271] Meanwhile, the contents not described in detail in this specification belong to the prior art known to those skilled in the art.

[0272] In the embodiments provided by the present invention, it should be understood that the disclosed system or method can be implemented in other ways. For example, the above-described embodiments of the invention are only illustrative, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation.

[0273] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, and may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0274] In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0275] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the basic features of the present invention.

[0276] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A data storage system for a digital village based on cloud computing, characterized in that: include: Client, data transmission module and cloud computing center; The client includes: Digital rural data writing module: used to input agricultural production data, rural e-commerce data, rural governance data, rural culture and tourism data, and environmental monitoring data; Data type setting module: when writing digital village data, it is used to automatically or manually set the data type to regular data, important data and particularly important data; Data transmission module: used to transmit digital village data to the cloud computing center, and transmit the data from the cloud computing center to the client; The cloud computing center includes: Data storage module: including hot data storage layer, warm data storage layer and cold data storage layer, storing data in the hot data storage layer, warm data storage layer and cold data storage layer respectively according to the writing time and data type of digital village data; Data storage layer migration module: used to count and analyze the access frequency of digital village data, and automatically migrate data between hot data storage layer, warm data storage layer and cold data storage layer; Data cache strategy module: used to predict the data that users may access based on their access patterns and historical data, and read these data from the data storage module into the cache in advance. For data whose access times reach a threshold within a certain period of time, it will be read into the cache.

2. The data storage system of a digital village based on cloud computing according to claim 1 is characterized in that: The client also includes: Access control module: using identity authentication and authorization management technology, only authenticated and authorized users can access data; Data preprocessing module: used for data cleaning, data conversion, and data integration of digital village data, and transmits the preprocessed data to the data transmission module.

3. The data storage system of a digital village based on cloud computing according to claim 1 is characterized in that: The cloud computing center also includes: Data redundancy mechanism module: used to set up one copy for regular data and multiple copies for important data. Multi-copy storage technology is used for important data, and these copies are stored on different cloud computing nodes. For particularly important data, erasure coding technology is used, and consistency checks and updates are performed on data copies regularly. At the same time, a data recovery mechanism is established to recover data from other copies in the event of node failure or data loss.

4. The data storage system of a digital village based on cloud computing according to claim 1 is characterized in that: In the data storage module, a distributed file system and object storage technology are used to store digital village data in multiple cloud computing nodes; The hot data storage layer, the warm data storage layer and the cold data storage layer use storage media with different performances, wherein the performance of the storage media used by the hot data storage layer, the warm data storage layer and the cold data storage layer decreases in sequence.

5. The data storage system of a digital village based on cloud computing according to claim 4 is characterized in that: The cloud computing center also includes: Data synchronization strategy module: used to establish a data synchronization mechanism to keep the data between each node of distributed storage consistent.

6. The data storage system of a digital village based on cloud computing according to claim 5 is characterized in that: In the data storage module, the writing time of digital village data is assumed to be t: Among the general data in the digital village data: In the time interval 0<t≤a, the hot data storage layer is used for storage; In the time interval a<t≤b, the warm data storage layer is used for storage; In the time interval b<t, cold data storage layer is used for storage; Important data in digital village data: In the time interval 0<t≤b, the hot data storage layer is used for storage; In the time interval b<t≤c, the warm data storage layer is used for storage; In the time interval c<t, cold data storage layer is used for storage; Particularly important data in digital village data include: In the time interval 0<t≤c, the hot data storage layer is used for storage; In the time interval a<t≤d, the warm data storage layer is used for storage; In the time interval d<t, cold data storage layer is used for storage; 0<a<b<c<d.

7. The data storage system of a digital village based on cloud computing according to claim 1 is characterized in that: In the data storage layer migration module, the data storage layer migration settings are as follows: Establish data access monitoring for digital village data, record the access time and number of each piece of data stored in the data storage module, regularly scan and count the data in the data storage module, and screen out data that meets the migration conditions; If the number of accesses to data exceeds the set threshold within a certain period of time, the migration condition is met and the data migration task is triggered. The data migration task migrates the selected data from the cold data storage layer to the warm data storage layer, or migrates the selected data from the warm data storage layer to the hot data storage layer, and updates the data storage location information. At the same time, the data is marked as migration data. For the marked migrated data, keep backups for a certain period of time in the original storage layer; Continuously monitor the access to migrated data. If the access frequency of migrated data decreases within a certain period of time, gradually migrate the data back to a lower-level storage layer.

8. The data storage system of a digital village based on cloud computing according to claim 5 is characterized in that: In the data synchronization strategy module: For the digital rural data in the hot data storage layer, when the parameters of the digital rural data change, a real-time synchronization strategy is adopted to immediately synchronize the new data to all relevant nodes; For the digital rural data in the warm data storage layer, when the parameters of the digital rural data change, a regular synchronization strategy is adopted to regularly synchronize the new data to all relevant nodes; For the digital rural data in the cold data storage layer, when the parameters of the digital rural data change, an asynchronous synchronization strategy is adopted to synchronize the data when it is requested or at a specific time interval.

9. The data storage system of a digital village based on cloud computing according to claim 1 is characterized in that: In the data cache strategy module: The settings are as follows: Assume that the data in the data set is X = {x1, x2, ..., x n }, where x j In, j = 1, 2, ..., n, representing different data; The time interval is T = {t1, t2, ..., t m }, where t i In , i = 1, 2, ..., m, representing different time periods; The goal is to establish a relationship between time intervals and data based on known historical data, and predict the data that may be accessed in a given time interval; According to the principle of Naive Bayes algorithm: Bayes' Theorem: Where P(X|Y) is the posterior probability of category Y given feature X; P(X|Y) is the probability of feature X given category Y; P(Y) is the prior probability of category Y; P(X) is the probability of feature X; Here, the data is considered as category Y and the time interval is considered as feature X; Algorithm steps: Record the access to different data in different time intervals in the past period of time to form a data set D = {(t i , x j )}, where (t i , x j ) represents the time interval t i Accessed data x j , D represents the collection of historical data; Calculate the prior probability P(x j ), P(x j ) represents data x j The prior probability of data x j The probability of appearing in the entire data set is calculated as where count(x j ) is the data x j is the number of occurrences, |D| is the size of the data set; Calculate the conditional probability P(t i |x j ), which means that when the known data is x j In the case of i The conditional probability of each data x j , count the number of occurrences in different time intervals under the data, and the calculation method is: Among them, count((t i , x j )) is in the data x j Next time interval t i Number of occurrences; For a given time interval t new , calculate the posterior probability P(x j |t new ); P(x j |t new ) represents the given new time interval t new In the case of x j The posterior probability of is used to predict the data that may be accessed in the new time interval by calculating this probability; According to the Naive Bayes hypothesis, Since P(t new ) is the same for all data and can be ignored; Therefore, P(x j |t new )≈P(t new |x j )P(x j ); After calculating the posterior probability of all data, the data with the highest probability is selected as the prediction result, that is, the data with the highest probability is selected as the data that may be accessed in a given new time interval after calculation; in, Represents the predicted value at a given new time interval t new The data most likely to be accessed, Indicates that in all different data x j In the above example, find the value such that P(x j |t new )The data x with the largest probability value j , here we are calculating the posterior probability of various data in a given new time interval, and then selecting the data with the highest probability as the prediction result; Through the above steps, the Naive Bayes algorithm can be used to predict the data that may be accessed in a given time interval based on historical data, thereby realizing the pre-fetching of digital village data in the cache; At the same time, the data is marked as cache data; For the marked cache data, keep a backup for a certain period of time in the original storage layer; The access to the cached data is continuously monitored. When the number of accesses to the cached data is 0 within a certain period of time, the cached data is migrated to the hot data storage layer in the data storage module, and the write time of the cached data is updated.

10. A method for using a data storage system for a digital village based on cloud computing according to claims 1 to 9, characterized in that: The following steps are involved: Input agricultural production data, rural e-commerce data, rural governance data, rural culture and tourism data, and environmental monitoring data through the digital rural data writing module on the client; At the same time, the data type setting module is used to automatically or manually set the data type to regular data, important data and particularly important data when writing digital village data; The access control module uses identity authentication and authorization management technology to ensure that only authenticated and authorized users can access data; The data preprocessing module then cleans, converts and integrates the digital village data and transmits it to the data transmission module; The data transmission module transmits the pre-processed data to the cloud computing center, and transmits the data from the cloud computing center to the client; In the cloud computing center, the data storage module stores the data in the hot data storage layer, the warm data storage layer, and the cold data storage layer according to the writing time and data type of the digital village data; The data storage module uses distributed file system and object storage technology to store digital village data in multiple cloud computing nodes; The hot data storage layer, warm data storage layer and cold data storage layer in the data storage module use storage media with different performances, and the performance of the storage media decreases in sequence; The data storage layer migration module counts and analyzes the access frequency of digital village data, and automatically migrates data between the hot data storage layer, warm data storage layer, and cold data storage layer; In the data storage layer migration module, data access monitoring is established for digital village data, the access time and number of each piece of data are recorded, the data in the data storage module is scanned and counted regularly, and data that meets the migration conditions is screened out; If the number of accesses to data exceeds the set threshold within a certain period of time, the migration condition is met, and the data migration task is triggered to migrate the selected data from the cold data storage layer to the warm data storage layer, or to migrate the selected data from the warm data storage layer to the hot data storage layer, and update the data storage location information, and mark the data as migrated data; for the marked migrated data, a backup is retained in the original storage layer for a certain period of time, and the access to the migrated data is continuously monitored. If the access frequency of the migrated data decreases within a certain period of time, the data is gradually migrated back to the storage layer at a lower level; The data redundancy mechanism module sets up multiple copies of important data, uses multi-copy storage technology to store the copies on different cloud computing nodes, uses erasure coding technology for particularly important data, and regularly checks and updates the consistency of data copies, while establishing a data recovery mechanism; The data synchronization strategy module establishes a data synchronization mechanism to keep the data between each node of the distributed storage consistent. For the digital rural data in the hot data storage layer, when the parameters of the digital rural data change, a real-time synchronization strategy is adopted to immediately synchronize the new data to all related nodes; for the digital rural data in the warm data storage layer, when the parameters of the digital rural data change, a periodic synchronization strategy is adopted to regularly synchronize the new data to all related nodes; for the digital rural data in the cold data storage layer, when the parameters of the digital rural data change, an asynchronous synchronization strategy is adopted to synchronize when the data is requested or within a specific time interval, and the synchronization priority is lower than that of hot data and warm data; The data cache strategy module predicts the data that users may access based on their access patterns and historical data, and reads the data from the data storage module into the cache in advance. For data that has reached a threshold number of accesses within a certain period of time, it reads it into the cache; At the same time, the data is marked as cached data; For the marked cache data, keep a backup for a certain period of time in the original storage layer; The access situation of the migrated data is continuously monitored. When the number of accesses to the cached data is 0 within a certain period of time, the cached data is migrated to the hot data storage layer in the data storage module, and the write time of the cached data is updated.

Images