An integrated and efficient management method for remote sensing image slice storage

Abstract

The present invention discloses an integrated and efficient management method for warehousing remote sensing image slices, which includes the following steps: step 1, warehousing original images; step 2, image preprocessing, obtaining COG images; step 3, map slicing; step 4, warehousing slices; step 5, warehousing metadata; step 6, data publishing; providing data browsing, retrieval and distribution services to users. The present invention provides functions including remote sensing image data storage and management, map slice and database storage management, original image slice-free data format conversion, image metadata storage and management, etc.

Description

Technical Field

[0001] The present invention belongs to the technical field of remote sensing images, and in particular relates to an integrated and efficient management method for warehousing remote sensing image slices. Background Art

[0002] The acquisition of satellite remote sensing and drone image data is becoming more and more convenient, and the amount of data is growing rapidly, but the management of raw image data and image slicing, migration, publishing and management still face certain challenges. For example, patent application 201910552789.4 discloses an oil and gas pipeline supervision system based on drone aerial photography. The system includes drone data access, geographic information system management, real-time monitoring, data analysis, anomaly recognition and analysis based on high-definition images, historical inspection data comparison and analysis, multi-level and multi-user collaborative operations and authority allocation and other functional modules. The system introduces WebGIS, Internet of Things and target detection and recognition technology.

[0003] For example, patent application 202110606706.2 discloses a drone data management method, device, equipment and storage medium, which relates to the field of data management technology. The drone data management method includes: determining a first target data path according to an input selection operation for a target item label; obtaining a target object model corresponding to the target item label stored in the first target data path from a server according to the first target data path; and displaying the target object model.

[0004] However, the existing technology has problems such as original images are stored in the form of file directories, which makes retrieval inconvenient; image slices are mostly stored in the form of hash files, and data migration and copying are inefficient. Summary of the invention

[0005] To solve the above problems, the primary purpose of the present invention is to provide an integrated and efficient management method for remote sensing image slice storage, which includes remote sensing image data storage and management, map slice and database storage management, original image slice-free data format conversion, image metadata storage and management and other functions.

[0006] The purpose of the present invention is to provide an integrated and efficient management method for remote sensing image slicing and warehousing, which simplifies the data management of remote sensing images, improves the efficiency of data sending and receiving, and integrates the processes of original image and metadata storage, slicing and warehousing, online browsing, data retrieval and distribution.

[0007] To achieve the above objectives, the technical solution of the present invention is as follows.

[0008] A remote sensing image slice storage integrated and efficient management method, the method comprising the following steps:

[0009] Step 1: Original image storage:

[0010] According to the original image storage location selected by the user and the distributed storage database address set, all files in the original image storage directory are queried one by one to see whether they exist in the target database. If the file does not exist in the database, the file will be stored in the database; otherwise, the file will be skipped.

[0011] Step 2: Image preprocessing to obtain COG images:

[0012] Perform cloud optimization processing on the original images one by one to obtain COG images;

[0013] Step 3, map slicing: Use the COG image generated by image preprocessing for slicing. Users can set parameters such as maximum zoom level, minimum zoom level, tile generation rules, number of processing threads, and slice storage location as needed;

[0014] Step 4, slicing into the database: the slice file generated in step 3 is stored in the mongoDB database. This step traverses each tile of each single-view image. If all pixels of the tile are transparent, the tile is skipped. Otherwise, it is determined whether the tile contains some transparent pixels. If it does not contain transparent pixels, is_edge is assigned a value of 0, and the tile is converted from PNG format to JPG format. Otherwise, is_edge is assigned a value of 0, and then the tile file and additional attributes are stored in the database.

[0015] Step 5: Metadata storage: Metadata storage supports two formats: XML and shp. By parsing the file, relevant attributes of the image are extracted and stored in the metadata storage database;

[0016] Step 6: Data publishing: Provide data browsing, retrieval and distribution services to users.

[0017] In step 1, the distributed storage database address is the GridFS database address.

[0018] In step 2, the process of obtaining COG images includes determining the band status of the original image (3-band or 4-band), bit depth conversion (converted to 8-bit), invalid value setting (NoData setting) and other parameter settings, and then outputting the COG image through the GDAL tool.

[0019] In step 3, the detailed slicing process is as follows:

[0020] 31) Obtaining images,

[0021] Get the i-th image from a single-view multi-period image

[0022] 32) Obtain user input parameters;

[0023] Input parameters as slicing rules, slice the image according to data parameters;

[0024] 33) GDAL slices;

[0025] Perform GDAL slicing;

[0026] 34) Determine whether the slicing is completed, if yes, end, otherwise loop and execute steps 31)-33).

[0027] Furthermore, in the input parameters, the maximum zoom level is limited to 17, and tiles above this level will be published through real-time slicing of COG images;

[0028] Secondly, the tile generation rule can be either WMTS or WMS;

[0029] Furthermore, the default number of processing threads is the number of CPU cores of the slice server minus 2.

[0030] In step 4, the tile file and additional attributes are as follows:

[0031]

[0032] In step 5, the metadata attribute fields are as follows:

[0033]

[0034] In step 6, during the data browsing process, when the map zoom level is less than 18, the slice storage database provides data support for the remote sensing image preview service to accelerate data browsing efficiency; when the map zoom level is greater than or equal to 18, the data source of the remote sensing image preview service is switched to the COG image real-time slice service, and the real-time slice cache server is queried to see whether the tile requested to be loaded already exists. If it exists, the tile is directly loaded. If not, the real-time slice is loaded before returning and stored in the real-time slice cache database.

[0035] The present invention has the following advantages:

[0036] 1. The database storage management mode of original images and image metadata simplifies the original image retrieval method and provides a portable original image query and download interface;

[0037] 2. Provide cloud optimized processing (Cloud Optimized GeoTiff, COG) of different types of image data (including multi-band selection, bit depth conversion, Nodata value setting), and cooperate with the tile cache database to achieve rapid publishing and efficient preview of high-level map tiles of online maps;

[0038] 3. Provides image slicing and mixed slicing database storage, improves the slicing migration speed, and improves the image preview response speed by providing tile call services. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a schematic diagram of the management tool interface implemented by the present invention.

[0040] Figure 2 It is a schematic diagram of the system logic and data flow implemented by the present invention.

[0041] Figure 3 It is a logic diagram of image slicing and slicing storage implemented by the present invention.

[0042] Figure 4 It is a schematic diagram of the search conditions implemented by the present invention.

[0043] Figure 5 It is a schematic diagram of the search results achieved by the present invention. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0045] The present invention implements an integrated and efficient management method for remote sensing image slicing and storage. In order to simplify the data management of remote sensing images and improve the efficiency of data transmission and reception, the method integrates the processes of original image and metadata storage, slicing and storage, online browsing, data retrieval and distribution. The tool interface used is as follows: Figure 1 As shown, the principle is as Figure 2 shown.

[0046] The specific steps include:

[0047] Step (1), original image storage:

[0048] According to the original image storage location selected by the user and the distributed storage database address (i.e., GridFS database address) set, all files in the original image storage directory are queried one by one to see whether they exist in the target database. If the file does not exist in the database, the file is stored in the database; otherwise, the file is skipped.

[0049] Step (2), image preprocessing (i.e., COG image conversion): The original images are optimized one by one in the cloud to obtain COGTiff images. This process includes determining the band status of the original image (3 bands or 4 bands), bit depth conversion (converted to 8 bits), invalid value setting (NoData setting) and other parameter settings, and then the COG image is output through the GDAL tool.

[0050] Step (3) Map slicing (image slicing): This step uses the COG image generated by image preprocessing. Users can set parameters such as the maximum zoom level, minimum zoom level, tile generation rules, number of processing threads, and slice storage location as needed. The detailed process is as follows: Figure 3 As shown,

[0051] 31) Obtaining images,

[0052] Get the i-th image from a single-view multi-period image

[0053] 32) Obtain user input parameters;

[0054] Input parameters as slicing rules, slice the image according to data parameters;

[0055] 33) GDAL slices;

[0056] Perform GDAL slicing;

[0057] 34) Determine whether the slicing is completed, if yes, end, otherwise loop and execute steps 31)-33).

[0058] in:

[0059] 1) The maximum zoom level is limited to 17. Tiles above this level will be published through real-time slicing of COG images;

[0060] 2) The tile generation rule can be either WMTS or WMS;

[0061] 3) The default number of processing threads is the number of CPU cores of the slice server minus 2.

[0062] Step (4) Slice storage: The slice file generated in step 3 is stored in the mongoDB database. This step traverses each tile of each single-view image. If all pixels of the tile are transparent, the tile is skipped. Otherwise, it is determined whether the tile contains some transparent pixels. If it does not contain transparent pixels, is_edge is assigned a value of 0, and the tile is converted from PNG format to JPG format. Otherwise, is_edge is assigned a value of 0, and then the tile file and additional attributes are stored in the database. The additional attribute fields are shown in Table 1.

[0063] Step (5) Metadata storage: Metadata storage supports two formats: XML and shp. By parsing the file, the relevant attributes of the image are extracted and stored in the metadata storage database. The metadata attribute fields are shown in Table 2.

[0064] Step (6) Data publishing: Provide users with data browsing, retrieval and distribution services, and interactive retrieval conditions such as Figure 4 As shown, the search results are as follows Figure 5 As shown. During data browsing, when the map zoom level is less than 18, the slice storage database provides data support for the remote sensing image preview service to accelerate data browsing efficiency; when the map zoom level is greater than or equal to 18, the data source of the remote sensing image preview service is switched to the COG image real-time slice service, and the real-time slice cache server is queried to see whether the requested tile already exists. If it does, the tile is directly loaded. If not, the real-time slice is loaded before returning and stored in the real-time slice cache database. That is, according to the user's browsing situation, the area with a zoom level greater than or equal to 18 for the first browsing is sliced ​​in real time, and the area with a zoom level greater than or equal to 18 for repeated browsing is directly called from the real-time slice cache server. This strategy can reduce the time consumption of the early slicing process and the amount of tile data storage, and can also accelerate the data loading speed of hot spots (areas that are browsed repeatedly).

[0065] Table 1 Tile attribute comparison table

[0066]

[0067] Table 2 Metadata attribute comparison table

[0068]

[0069]

[0070] The present invention utilizes the database storage management mode of original images and image metadata, simplifies the original image retrieval method, and provides a portable original image query and download interface; at the same time, it provides cloud optimized processing (Cloud Optimized GeoTiff, COG) of image data of different types (including multi-band selection, bit depth conversion, Nodata value setting), and cooperates with the slice cache database to realize the rapid release and efficient preview of high-level map tiles of online maps; it provides image slice and mixed slice database storage, improves the slice migration speed, and improves the image preview response speed by providing tile call service.

[0071] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An integrated and efficient management method for storing remote sensing image slices, characterized in that The method comprises the following steps: Step 1: Original image storage: According to the original image storage location selected by the user and the distributed storage database address set, all files in the original image storage directory are queried one by one to see whether they exist in the target database. If the file does not exist in the database, the file is stored in the database; otherwise, the file is skipped; Step 2: Image preprocessing to obtain COG images: Perform cloud optimization processing on the original images one by one to obtain COG images; Step 3, map slicing: Use the COG image generated by image preprocessing for slicing. The user sets the maximum zoom level, minimum zoom level, tile generation rule, number of processing threads, and slice storage location parameters as needed; Step 4, slicing into the database: the slice file generated in step 3 is stored in the mongoDB database. This step traverses each tile of each single-view image. If all pixels of the tile are transparent, the tile is skipped. Otherwise, it is determined whether the tile contains some transparent pixels. If it does not contain transparent pixels, is_edge is assigned a value of 0, and the tile is converted from PNG format to JPG format. Otherwise, is_edge is assigned a value of 0, and then the tile file and additional attributes are stored in the database. Step 5: Metadata storage: Metadata storage supports two formats: XML and shp. By parsing the file, relevant attributes of the image are extracted and stored in the metadata storage database; Step 6: Data publishing: providing data browsing, retrieval and distribution services to users; During data browsing, when the map zoom level is less than 18, the slice storage database provides data support for the remote sensing image preview service to accelerate data browsing efficiency; When the map zoom level is greater than or equal to 18, the data source of the remote sensing image preview service is switched to the COG image real-time tile service, and the real-time tile cache server is queried to see whether the tile requested to be loaded already exists. If it does, the tile is loaded directly. If it does not exist, the real-time tile is loaded before returning and stored in the real-time tile cache database.

2. The integrated and efficient management method for storing remote sensing image slices according to claim 1, characterized in that In step 1, the distributed storage database address is the GridFS database address.

3. The integrated and efficient management method for storage of remote sensing image slices according to claim 1 is characterized in that In step 2, the process of obtaining the COG image includes determining the band status of the original image, bit depth conversion, invalid value setting parameter setting, and then outputting the COG image through the GDAL tool; the above band is 3 bands or 4 bands, the above bit depth is converted to 8 bits, and the above invalid value is set to NoData setting.

4. The integrated and efficient management method for storing remote sensing image slices according to claim 1, characterized in that In step 3, the detailed slicing process is as follows: 31) Obtaining images, Get the i-th image from a single-view multi-period image 32) Obtain user input parameters; Input parameters as slicing rules, slice the image according to data parameters; 33) GDAL slices; Perform GDAL slicing; 34) Determine whether the slicing is completed, if yes, end, otherwise loop and execute steps 31)-33).

5. The integrated and efficient management method for storage of remote sensing image slices according to claim 4 is characterized in that In the input parameters, the maximum zoom level is limited to 17, and tiles above this level will be published through real-time slicing of COG images; Secondly, the tile generation rule can be either WMTS or WMS; Furthermore, the default number of processing threads is the number of CPU cores of the slice server minus 2.

6. The integrated and efficient management method for storing remote sensing image slices according to claim 1, characterized in that In step 4, the tile file and additional attributes are as follows:

7. The integrated and efficient management method for storage of remote sensing image slices according to claim 1, characterized in that In step 5, the metadata attribute fields are as follows:

Images