Remote sensing data distributed storage method and device and remote sensing data reading method

Abstract

The present application relates to a remote sensing data distributed storage method and device and a remote sensing data reading method, wherein the storage method includes: obtaining remote sensing image data to be stored, dividing the remote sensing image data into more than one storage block; storing each storage block in a cluster storage unit, creating an index file, and caching each storage block in a distributed cache layer. By creating an index file, each storage block obtained by segmentation is cached in a distributed cache layer, so that when storing massive remote sensing data, a combination of distributed cache and distributed object storage technology is adopted, which effectively improves the overall storage performance of remote sensing image data. In addition, by creating an index file, the storage blocks obtained by segmentation are cached in a distributed cache layer, and when reading the stored data, the distributed cache layer is given priority, which effectively improves the data access efficiency and reduces the data access delay.

Description

Technical Field

[0001] The present application relates to the technical field of remote sensing data processing, and in particular to a remote sensing data distributed storage method and device and a remote sensing data reading method. Background Art

[0002] In recent years, my country's satellite industry has developed rapidly. With the successful launch of a series of satellites such as Gaofen and Ziyuan, the amount of earth observation data received has increased exponentially. The effective storage and standardized management of massive remote sensing data has become one of the major challenges currently faced. Traditional storage methods mainly include direct-connected storage and centralized storage. Among them, direct-connected storage simply means that the computing server and the storage server are directly connected, and the expansion capacity is limited. Centralized storage mainly includes SAN storage, NAS storage, etc. In this type of storage structure, the computing server and the storage server are separated, and the storage server is an independent device. However, all file reading and writing must pass through an entry (i.e., the controller). Affected by the controller, the expansion capacity of centralized storage is also limited, which makes the storage performance poor when using the above two storage methods to store massive remote sensing data. Summary of the invention

[0003] In view of this, the present application proposes a distributed storage method for remote sensing data, which can effectively improve the storage performance of massive remote sensing data.

[0004] According to one aspect of the present application, a remote sensing data distributed storage method is provided, comprising:

[0005] Acquire remote sensing image data to be stored, and divide the remote sensing image data into one or more storage blocks;

[0006] Each storage block is stored in a cluster storage unit, an index file is created, and each storage block is cached in a distributed cache layer;

[0007] The index file is used to record the mapping relationship between each storage block and each cache in the distributed cache layer.

[0008] In a possible implementation, when the remote sensing image data is divided into more than one storage blocks, the sizes of the storage blocks are the same.

[0009] In a possible implementation, when each of the storage blocks is stored in a cluster storage unit, Ceph distributed storage technology is used for storage.

[0010] In a possible implementation, when the Ceph distributed storage technology is used to store each storage block in a cluster storage unit, the method includes:

[0011] Configuring a corresponding identifier for each storage block;

[0012] Calculate the hash value of the identifier of each storage block, and obtain the PG sequence number corresponding to each storage block based on the hash value of the identifier of each storage block;

[0013] The PG serial number corresponding to each storage block is mapped to the storage server of the cluster storage unit, and each storage block is stored in the corresponding storage server.

[0014] In a possible implementation, when mapping the PG sequence number corresponding to each of the storage blocks to the storage server of the cluster storage unit, it is performed based on a preset storage policy configuration rule;

[0015] Wherein, the storage policy configuration rules include: a storage server with a preset number of PG sequence numbers mapped thereto.

[0016] In a possible implementation, the index file is created, and each storage block is cached in the distributed cache layer based on an identifier configured for each storage block.

[0017] In a possible implementation, when caching each storage block in the distributed cache layer, it also includes: according to the current storage capacity of each memory in the distributed cache layer and the size of the storage block currently to be written, removing the storage block data stored in the memory in the distributed cache layer.

[0018] According to another aspect of the present application, a remote sensing data reading method is also provided, comprising:

[0019] After receiving the data reading instruction, according to the data information to be read in the data reading instruction, the corresponding storage information is searched in the index file;

[0020] When the storage information is found in the index file, a corresponding storage block is read from the cache recorded in the index file;

[0021] The index file is created when writing remote sensing image data into the distributed cache layer, and is used to record the mapping relationship between each storage block and each cache in the distributed cache layer.

[0022] According to another aspect of the present application, there is also provided a remote sensing data distributed storage device, comprising a data acquisition module, a data separation module, a first storage module and a second storage module;

[0023] The data acquisition module is configured to acquire remote sensing image data to be stored;

[0024] The data segmentation module is configured to segment the remote sensing image data into more than one storage block;

[0025] The first storage module is configured to store each of the storage blocks in a cluster storage unit;

[0026] The second storage module is configured to create an index file and cache each storage block in a distributed cache layer;

[0027] The index file is used to record the mapping relationship between each storage block and each cache in the distributed cache layer.

[0028] According to another aspect of the present application, there is also provided a remote sensing data reading device, comprising a file search module and a data reading module;

[0029] The file search module is configured to search the distributed cache layer for an index file corresponding to the data information to be read according to the data information to be read in the data read instruction after receiving the data read instruction;

[0030] The data reading module is configured to read a corresponding storage block from the cache recorded in the index file when the index file is found.

[0031] By dividing the remote sensing image data to be stored into multiple storage blocks, each storage block obtained by the division is stored in a cluster storage unit, thereby realizing the storage of massive remote sensing data using distributed storage technology. At the same time, by creating an index file, each storage block obtained by the division is cached in a distributed cache layer, so that when storing massive remote sensing data, the combination of distributed cache and distributed object storage technology is adopted, which effectively improves the overall storage performance of remote sensing image data. In addition, by creating an index file, the storage blocks obtained by the division are cached in a distributed cache layer, and when reading the stored data, the distributed cache layer is given priority, which effectively improves the data access efficiency and reduces the data access delay.

[0032] Other features and aspects of the present application will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the present application and, together with the description, serve to explain the principles of the present application.

[0034] Figure 1 A flowchart showing a remote sensing data distributed storage method according to an embodiment of the present application;

[0035] Figure 2 A diagram showing a Ceph storage architecture when Ceph distributed object storage technology is used for data storage in a remote sensing data distributed storage method according to an embodiment of the present application;

[0036] Figure 3 A schematic diagram showing a remote sensing data reading and writing process in a remote sensing data distributed storage method according to an embodiment of the present application;

[0037] Figure 4 A structural block diagram showing a remote sensing data distributed storage device according to an embodiment of the present application;

[0038] Figure 5 A structural block diagram of a remote sensing data distributed storage device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0039] Various exemplary embodiments, features and aspects of the present application will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0040] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0041] In addition, in order to better illustrate the present application, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present application can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present application.

[0042] Figure 1 FIG. 1 is a flow chart showing a method for distributed storage of remote sensing data according to an embodiment of the present application. Figure 1 As shown, the method includes: step S100, obtaining remote sensing image data to be stored, and dividing the remote sensing image data into multiple storage blocks. Step S200, storing each storage block in a cluster storage unit, and creating an index file to cache each storage block in a distributed cache layer. The index file is used to record the mapping relationship between each storage block and each cache in the distributed cache layer.

[0043] Here, it should be pointed out that in the method of the embodiment of the present application, the distributed cache layer is configured with multiple memory caches, each memory cache is used to store different storage blocks. In a possible implementation, a corresponding index file is created for each memory cache in the distributed cache layer, and the storage block data stored in the memory cache is recorded through the created index file.

[0044] Therefore, the distributed storage method of remote sensing data in the embodiment of the present application divides the remote sensing image data to be stored into multiple storage blocks, and stores each storage block obtained by the division into a cluster storage unit, thereby realizing the storage of massive remote sensing data using distributed storage technology. At the same time, each storage block obtained by the division is cached in a distributed cache layer by creating an index file, so that when storing massive remote sensing data, a combination of distributed cache and distributed object storage technology is adopted, which effectively improves the overall storage performance of remote sensing image data. In addition, by creating an index file, the storage blocks obtained by the division are cached in a distributed cache layer, and when reading the stored data, the distributed cache layer is given priority, which effectively improves the data access efficiency and reduces the data access delay.

[0045] It should be noted that when the remote sensing image data to be stored is divided into a plurality of storage blocks, the division can be performed in a manner that the sizes of the storage blocks are the same. That is, the remote sensing image data is divided based on storage blocks of a preset size. It should be noted that the preset size of the divided storage blocks can be set according to the storage system, and can also be flexibly adjusted according to actual conditions. In a possible implementation, the preset size of the storage block can be set to 512KB-4MB, preferably 4MB.

[0046] At the same time, it should also be pointed out that the segmentation of remote sensing image data can be achieved by using conventional data segmentation technical means in the art, which will not be described in detail here.

[0047] Furthermore, in the process of segmenting the remote sensing image data according to the storage blocks of preset sizes, when the total amount of the remote sensing image data to be stored cannot be equally divided by the preset size of the storage blocks, the last remaining part of the data can be used as a storage block.

[0048] For example, when the preset size of the storage block is 40MB and the current remote sensing image data to be stored is 403MB, when segmenting the remote sensing image data, you can choose to segment the first 400MB into 10 40MB storage blocks, and then store the remaining 3MB data block as one storage block.

[0049] In addition, in the method of the embodiment of the present application, when each storage block is stored in the cluster storage unit, Ceph distributed storage technology can be used for storage. Figure 2 The Ceph distributed storage architecture includes: the bottom-level RADOS (distributed object storage system), the basic library LIBRADOS of Ceph storage, and the high-level application interface.

[0050] Among them, the bottom-level RADOS of Ceph storage is mainly composed of three parts, namely Ceph OSD, CephMonitor and Ceph MDS.

[0051] Specifically, OSD is the storage device of Ceph. Ceph OSD stores data in the form of storage blocks on the physical disk of each node in the cluster. Most of the work is implemented by the OSD daemon process, which is mainly used for the storage of all data and storage blocks in the cluster, replication, recovery, replenishment and balanced data distribution of cluster data, and sends heartbeats to other OSD daemons, and then provides some monitoring information to the Monitor.

[0052] CephMonitor is a monitoring module for Ceph storage, which is mainly used to monitor the health status of the entire cluster and ensure the consistency of cluster data. ClusterMAP describes the physical location of storage blocks and a list that aggregates storage devices together. The information is provided by the daemon that maintains cluster members to provide status and cluster configuration information between nodes. Cephmonitor maps mainly include OSD maps, PG maps, MDS maps, and CRUSH, etc. These maps are collectively referred to as cluster maps. Ceph monitor does not store data.

[0053] Ceph MDS refers to the metadata server, whose main function is to save the metadata information of the file system.

[0054] LIBRADOS is the basic library of Ceph storage. Its function is to abstract and encapsulate RADOS and provide an API, which allows applications to interact with the RADOS system by accessing the library. It supports multiple programming languages, such as C, C++, Python, etc.

[0055] There are three high-level application interfaces developed based on the LIBRADOS layer, namely RADOSGW, LIBRBD and MDS. Among them, RADOSGW and RBD provide upper-level interfaces with a higher level of abstraction and are easier to use for applications or clients based on the LIBRADOS library. RADOSGW is a gateway based on the currently popular RESTFUL protocol that supports object storage and is compatible with S3 and Swift. LIBRBD provides a distributed block storage device interface that supports block storage. MDS provides a POSIX-compatible file system that supports file storage. Those skilled in the art can understand that the application layer is the various application methods for each application interface of Ceph in different scenarios.

[0056] Based on the Ceph distributed storage architecture described above, in the method of the embodiment of the present application, when using Ceph distributed storage technology to store each storage block in a cluster storage unit, it can be specifically implemented in the following way.

[0057] First, configure a corresponding identifier for each storage block. Then, calculate the hash value of the identifier of each storage block, and obtain the PG sequence number corresponding to each storage block based on the hash value of the identifier of each storage block. Then, map the PG sequence number corresponding to each storage block to the storage server of the cluster storage unit, and store each storage block in the corresponding storage server.

[0058] For more details, see Figure 3 When the remote sensing image data to be stored is stored in the Ceph cluster, the remote sensing image data will first be divided into multiple storage blocks, and each storage block is correspondingly configured with a unique ID (i.e., identification number). Among them, in a possible implementation method, the size of the identifier configured for each storage block can be determined according to the distributed storage system. That is, the size of the ID configured for each storage block can be limited by RADOS. In the method of the embodiment of the present application, the size of the ID can be set to 4MB. It is understandable that the size of the ID can also be flexibly set by the user according to actual conditions in order to realize the organization and management of the underlying storage. Therefore, when the upper-level application needs to store a large-sized image file, the file needs to be divided into a series of storage blocks of uniform size for storage.

[0059] After the remote sensing image data to be stored is mapped into one or more storage blocks, each storage block is independently mapped to a PG. Specifically, in a possible implementation, the static HASH algorithm in the Ceph system can be used to calculate the hash value of the ID (i.e., object ID, referred to as OID) of each storage block, and the ID of each storage block is mapped to a pseudo-random value with an approximately uniform distribution. Then, the pseudo-random values ​​obtained by mapping are bitwise ANDed with the MASK to obtain the final PG sequence number (PGID).

[0060] It should be noted that the overall result of the hash value calculation and the bitwise AND operation is actually to randomly select one from all m PGs in an approximately uniform manner. Based on this mechanism, when there are a large number of storage blocks and a large number of PGs, RADOS can ensure an approximately uniform mapping between storage blocks and PGs. Because the storage blocks are cut from remote sensing image files, most storage blocks have the same size. Therefore, this mapping ultimately ensures that the total amount of data stored in the storage blocks in each PG is approximately uniform.

[0061] After obtaining the PG serial number corresponding to each storage block through the hash value of the identification of each storage block, the step of mapping the PG serial number corresponding to each storage block to the storage server of the cluster storage unit and storing each storage block in the corresponding storage server can be performed. That is, the PG value of the logical organization unit of the split object is mapped to the actual storage unit OSD of the data.

[0062] In one possible implementation, when mapping the PG serial number corresponding to each storage block to the storage server (i.e., OSD) of the clustered storage unit, the CRUSH algorithm can be used to substitute the pgid value into it, and then a group of n OSDs can be obtained. These n OSDs jointly store and maintain all storage blocks in a PG. The value of n can be configured according to the reliability requirements of the actual application, and is usually set to 3 in a production environment. Specifically for each OSD, the OSD Daemon running on it is responsible for performing operations such as storage, access, and metadata maintenance of the Object mapped to the local file system.

[0063] It should be noted that the results of the CRUSH algorithm are affected by other factors. There are two main influencing factors. One is the current system status. When the status and number of OSDs in the system change, the cluster operation process may also change, and this change will affect the mapping relationship between PG and OSD. The other is the storage policy configuration, which can specify that the three OSDs carrying the same PG are located on different servers or racks in the data center, thereby further improving the reliability of storage.

[0064] Therefore, the mapping relationship between PG and OSD is fixed only when the system status and storage strategy do not change. In actual use, the strategy usually does not change once configured. The change in system status may be due to device damage or the expansion of the storage cluster. However, the Ceph system itself provides automatic support for such changes, so even if the mapping relationship between PG and OSD changes, it will not affect the application. In fact, Ceph uses the dynamic characteristics of the CRUSH algorithm to dynamically migrate a PG to different OSD combinations as needed, thereby automatically achieving high reliability, data distribution rebalancing and other characteristics.

[0065] That is to say, in the method of the embodiment of the present application, when mapping the PG serial number corresponding to each storage block to the storage server of the cluster storage unit, it can be performed based on the preset storage policy configuration rules. Among them, the storage policy configuration rules include: one PG serial number maps a preset number of storage servers. According to the foregoing, in a possible implementation, the number of storage servers corresponding to one PG serial number can be set to 3. That is, the value range of the preset number can be set to 1-3.

[0066] The above steps complete the entire mapping process of remote sensing image data from file to storage block, storage block to PG, and PG to OSD.

[0067] It should also be noted that, see Figure 3 In the method of the embodiment of the present application, after the remote sensing image data is divided into multiple storage blocks and a unique identifier is configured for each storage block, the method further includes caching each divided storage block into a distributed cache layer. That is, each storage block is cached into each memory cache in the distributed cache layer by creating an index file.

[0068] In a possible implementation, each storage block is cached in the distributed cache layer based on an identifier configured for each storage block.

[0069] That is to say, in the method of the embodiment of the present application, each memory cache in the distributed cache layer is configured with a unique label. After the remote sensing image data is divided into one or more storage blocks, when the storage blocks are simultaneously written into the distributed cache layer, in the process of creating the index file, the identification of each storage block is mapped to the label of each memory cache one by one, and the mapping relationship between each storage block and each memory cache is obtained, and the mapping relationship is recorded in the index file. At the same time, the storage block is correspondingly written to the corresponding memory cache.

[0070] It should also be noted that in the method of the embodiment of the present application, the distributed cache is implemented by memory. In a possible implementation, when caching each storage block in the distributed cache layer, it also includes: according to the current storage capacity of each memory in the distributed cache layer and the size of the current storage block to be written, the storage block data stored in the memory in the distributed cache layer is removed. Specifically, the removal of the data stored in each memory cache in the distributed cache layer can be implemented in the following manner.

[0071] That is, by detecting the current storage capacity of each memory cache, when it is detected that the current storage capacity can still meet the storage of the storage block data, the data is not removed, and the storage block data to be stored is directly cached in the memory. When it is detected that the current storage capacity is insufficient to store the current storage block data, data is removed according to the order of the storage block data currently stored in the memory cache.

[0072] The detection of the current storage capacity of each memory cache can be performed by combining the maximum data storage capacity of each memory cache with the number of stored storage blocks. That is, by counting the number of stored storage blocks, the storage capacity of the memory cache is obtained from the number of stored storage blocks and the preset size of the storage blocks, and then the storage capacity of the memory cache is compared with the maximum storage capacity to obtain the current storage capacity of the memory cache.

[0073] At the same time, when sorting the data of each storage block currently stored in the memory cache, the data can be sorted according to the number of accesses of each storage block since it was stored in the memory cache. When sorting the storage blocks, the data can be sorted from most to least according to the number of accesses of each storage block.

[0074] For example, in a possible implementation, each memory in the distributed cache layer is 64GB, and the storage block size is 4MB, so the number of storage blocks is 16384. When each storage block data is stored in the memory cache, it can be cached in the memory in a stacked manner. That is, the writing process of the storage block in the memory is similar to a stack. As long as there is space in the memory cache, the data will be stored in the cache, and no data will be removed until the storage space is saturated. At the same time, in the process of writing data to the memory cache, each time the stored storage block data is repeatedly accessed, the access frequency is increased by 1, and the data will be sorted by access frequency, with frequently accessed data at the top and less frequently accessed data at the bottom. When there is not enough cache space in the memory cache and new storage block data needs to be written, the newly written storage block data will be directly inserted into the cached storage block data sequence, and the data at the bottom of the storage block data sequence in the cache will be removed.

[0075] Furthermore, in the method of the embodiment of the present application, it also includes: counting the access frequency of the storage blocks stored in each cache in the distributed cache. When it is found that the access frequency of the storage blocks stored in each cache in the distributed cache is less than the preset frequency, the storage blocks are removed from the cache. The access frequency of the storage blocks can be obtained by counting the number of accesses within a preset time period.

[0076] That is to say, the data in the storage has hot spots, and the access to the data is not uniform. Some data will be accessed frequently, so it is also called "hot data". In order to improve the data reading and writing access speed, in the method of the embodiment of the present application, on the basis of Ceph object storage, a layer of memory cache layer is added, so that when accessing hot data, the data in the cache will be accessed first, thereby shortening the data access delay.

[0077] That is, when the user needs to read remote sensing image data in distributed storage, the scheduling system will first traverse the remote sensing data index file storage blocks in the memory cache. If the index file is found to exist, the block data in the memory cache will be directly read. If the storage block of the required data file does not exist in the memory cache, the data file will be directly read from the storage and written to the memory cache. Since the memory space capacity is fixed, the data with low access frequency will be moved out of the memory cache, thus completing the complete process of remote sensing image data storage and reading.

[0078] Correspondingly, based on any of the above-mentioned distributed storage methods for remote sensing data, the present application also provides a remote sensing data reading method. It should be noted that the remote sensing data reading method provided in the present application is performed after data is stored based on any of the above-mentioned distributed storage methods for remote sensing data. In other words, the remote sensing data reading method of the embodiment of the present application and the remote sensing data distributed storage method of the embodiment of the present application are both based on the same data storage architecture.

[0079] Specifically, the remote sensing data reading method provided by the present application includes: after receiving a data reading instruction, according to the data information to be read in the data reading instruction, the corresponding storage information is queried from the index file. When the storage information is queried from the index file, the corresponding storage block is read from the cache recorded in the index file. Among them, the index file is created when the remote sensing image data is written to the distributed cache layer, and is used to record the mapping relationship between each storage block and each cache in the distributed cache layer. When the storage information is not queried from the index file, the corresponding data is directly read from the storage server in the cluster storage unit.

[0080] Therefore, the remote sensing data distributed storage method and data reading method of the embodiment of the present application effectively improve the overall storage performance by adopting a combination of distributed cache + Ceph distributed object storage technology, while also achieving the purpose of reducing single point failures, elastic expansion, synchronous linear growth of performance, and easy operation and maintenance, achieving the effect of not requiring data migration for software and hardware upgrades and improving data access speed during high concurrency.

[0081] Accordingly, based on any of the above-mentioned distributed storage methods for remote sensing data, the present application also provides a distributed storage device for remote sensing data. Since the working principle of the distributed storage device for remote sensing data provided by the present application is the same or similar to the principle of the distributed storage method for remote sensing data of the embodiment of the present application, the repeated parts will not be repeated.

[0082] See also Figure 4 The remote sensing data distribution storage device 100 provided in the present application includes a data acquisition module 110, a data segmentation module 120, a first storage module 130 and a second storage module 140. Among them, the data acquisition module 110 is configured to acquire the remote sensing image data to be stored. The data segmentation module is configured to segment the remote sensing image data into more than one storage block. The first storage module 130 is configured to store each storage block in a cluster storage unit. The second storage module 140 is configured to create an index file and cache each storage block in a distributed cache layer. Among them, the index file is used to record the mapping relationship between each storage block and each cache in the distributed cache layer.

[0083] Accordingly, based on any of the above remote sensing data reading methods, the present application also provides a remote sensing data reading device. Since the working principle of the remote sensing data reading device provided by the present application is the same or similar to the principle of the remote sensing data reading method of the embodiment of the present application, the repeated parts will not be repeated.

[0084] See also Figure 5 The remote sensing data reading device 200 provided in the present application includes a file search module 210 and a data reading module 220. The file search module 210 is configured to search for an index file corresponding to the data information to be read in the distributed cache layer according to the data information to be read in the data reading instruction after receiving the data reading instruction. The data reading module 220 is configured to read the corresponding storage block from the cache recorded in the index file when the index file is found.

[0085] The embodiments of the present application have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A remote sensing data distributed storage method, characterized in that: include: Acquire remote sensing image data to be stored, and divide the remote sensing image data into one or more storage blocks; Each storage block is stored in a cluster storage unit, an index file is created, and each storage block is cached in a distributed cache layer; Creating an index file, and caching each storage block in the distributed cache layer based on the identifier configured for each storage block; The index file is used to record the mapping relationship between each storage block and each cache in the distributed cache layer; Wherein, when each of the storage blocks is stored in the cluster storage unit, Ceph distributed storage technology is used for storage; When the Ceph distributed storage technology is used to store each storage block in a cluster storage unit, it includes: A corresponding identifier is configured for each storage block; wherein the size of the identifier configured for each storage block can be determined according to the distributed storage system; that is, the size of the ID configured for each storage block can be limited by RADOS; Calculate the hash value of the identifier of each storage block, and obtain the PG sequence number corresponding to each storage block based on the hash value of the identifier of each storage block; Mapping the PG sequence number corresponding to each storage block to the storage server of the cluster storage unit, and storing each storage block in the corresponding storage server; After the remote sensing image data to be stored is mapped into one or more storage blocks, each storage block is independently mapped to a PG. Specifically, the static HASH algorithm in the Ceph system can be used to calculate the hash value of the ID of each storage block, and the ID of each storage block can be mapped to a pseudo-random value with an approximately uniform distribution. Then, the pseudo-random values ​​obtained by mapping are bitwise ANDed with the MASK to obtain the final PG sequence number. Add a memory cache layer, and cache the segmented storage blocks to the distributed cache layer by creating index files. When users need to read remote sensing image data in distributed storage, the scheduling system will first traverse the remote sensing data index file storage blocks in the memory cache. If the index file is found to exist, the segmented data in the memory cache will be directly read. When the storage block of the required data file does not exist in the memory cache, the storage server in the cluster storage unit will directly read the corresponding data file and write the file to the memory cache. According to the current storage capacity of each memory in the distributed cache layer and the size of the current storage block to be written, the storage block data stored in the memory in the distributed cache layer is removed; The current storage capacity of each memory cache is detected by combining the maximum data storage capacity of each memory cache with the number of stored storage blocks. When it is detected that the current storage capacity can still meet the storage of the storage block data, the data is not removed and the storage block data to be stored is directly cached in the memory. When it is detected that the current storage capacity is insufficient to store the current storage block data, the data is removed according to the order of the storage block data currently stored in the memory cache, wherein the storage blocks can be sorted from most to least according to the number of accesses to each storage block when sorting.

2. The method according to claim 1, characterized in that When the remote sensing image data is divided into more than one storage block, the sizes of the storage blocks are the same.

3. The method according to claim 1, characterized in that When mapping the PG sequence number corresponding to each of the storage blocks to the storage server of the cluster storage unit, it is performed based on a preset storage policy configuration rule; Wherein, the storage policy configuration rules include: a storage server with a preset number of PG sequence numbers mapped thereto.

4. A remote sensing data reading method, characterized in that: The remote sensing data read is stored based on the storage method according to any one of claims 1 to 3, comprising: After receiving the data reading instruction, according to the data information to be read in the data reading instruction, the corresponding storage information is searched in the index file; When the storage information is found in the index file, a corresponding storage block is read from the cache recorded in the index file; The index file is created when writing remote sensing image data into the distributed cache layer, and is used to record the mapping relationship between each storage block and each cache in the distributed cache layer.

5. A remote sensing data distributed storage device, characterized in that: It includes a data acquisition module, a data separation module, a first storage module and a second storage module; The data acquisition module is configured to acquire remote sensing image data to be stored; The data segmentation module is configured to segment the remote sensing image data into more than one storage block; The first storage module is configured to store each of the storage blocks in a cluster storage unit; The second storage module is configured to create an index file and cache each storage block in a distributed cache layer; The index file is used to record the mapping relationship between each storage block and each cache in the distributed cache layer; Wherein, when each of the storage blocks is stored in the cluster storage unit, Ceph distributed storage technology is used for storage; When the Ceph distributed storage technology is used to store each storage block in a cluster storage unit, it includes: A corresponding identifier is configured for each storage block; wherein the size of the identifier configured for each storage block can be determined according to the distributed storage system; that is, the size of the ID configured for each storage block can be limited by RADOS; Calculate the hash value of the identifier of each storage block, and obtain the PG sequence number corresponding to each storage block based on the hash value of the identifier of each storage block; Mapping the PG sequence number corresponding to each storage block to the storage server of the cluster storage unit, and storing each storage block in the corresponding storage server; After the remote sensing image data to be stored is mapped into one or more storage blocks, each storage block is independently mapped to a PG. Specifically, the static HASH algorithm in the Ceph system can be used to calculate the hash value of the ID of each storage block, and the ID of each storage block can be mapped to a pseudo-random value with an approximately uniform distribution. Then, the pseudo-random values ​​obtained by mapping are bitwise ANDed with the MASK to obtain the final PG sequence number. Add a memory cache layer, and cache the segmented storage blocks to the distributed cache layer by creating index files. When users need to read remote sensing image data in distributed storage, the scheduling system will first traverse the remote sensing data index file storage blocks in the memory cache. If the index file is found to exist, the segmented data in the memory cache will be directly read. When the storage block of the required data file does not exist in the memory cache, the storage server in the cluster storage unit will directly read the corresponding data file and write the file to the memory cache. According to the current storage capacity of each memory in the distributed cache layer and the size of the current storage block to be written, the storage block data stored in the memory in the distributed cache layer is removed; The current storage capacity of each memory cache is detected by combining the maximum data storage capacity of each memory cache with the number of stored storage blocks. When it is detected that the current storage capacity can still meet the storage of the storage block data, the data is not removed and the storage block data to be stored is directly cached in the memory. When it is detected that the current storage capacity is insufficient to store the current storage block data, the data is removed according to the order of the storage block data currently stored in the memory cache, wherein the storage blocks can be sorted from most to least according to the number of accesses to each storage block when sorting.

6. A remote sensing data reading device, characterized in that: After the read remote sensing data is stored based on the storage method according to any one of claims 1 to 4, it includes a file search module and a data reading module; The file search module is configured to search the distributed cache layer for an index file corresponding to the data information to be read according to the data information to be read in the data read instruction after receiving the data read instruction; The data reading module is configured to read a corresponding storage block from the cache recorded in the index file when the index file is found.

Images