System and method for automatically collecting multi-type remote sensing satellite image data into cloud

Abstract

The invention discloses a system and a method for automatically collecting multi-type remote sensing satellite image data into a cloud, and relates to the technical field of remote sensing satellite image data collection. The system comprises a data acquisition module, a cluster DB module, a data processing module, a COG folder module, a data cloud storage module and the like, metadata information of image data is pulled, metadata is analyzed by Agent, a metadatabase table is obtained, the image data at a specified position is read according to the metadatabase table, and the image data is stored in the cloud after being processed and subjected to COG conversion. The system can automatically collect and store multi-type remote sensing satellite image data in the cloud in a timed or real-time manner, and can realize dynamic capacity expansion of a storage cluster according to the real-time data size to meet the collection rate requirement.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing satellite image data acquisition, and particularly to a multi-type remote sensing satellite image data automatic acquisition into-cloud system and method. Background Art

[0002] In recent years, with the increasing prominence of global environmental problems and the advancement of the sustainable development strategy, remote sensing satellite image data has played an increasingly important role in fields such as environmental monitoring, meteorological prediction, and urban planning. Remote sensing technology captures various information of the Earth through high-precision sensors carried by on-orbit satellites. It can not only observe the surface and atmosphere conditions all-weather, over a large range, and multi-dimensionally, but also provide a long time series of historical data to support in-depth research on issues such as climate change, natural disasters, and ecological environment.

[0003] In the field of environmental monitoring, remote sensing satellite image data can accurately capture forest coverage, ocean pollution, desertification process, etc., providing important information for environmental protection and disaster prevention. At the same time, in meteorological prediction, satellite remote sensing technology can observe the movement of clouds, temperature distribution, precipitation conditions, etc., providing a scientific basis for weather forecasting and early warning of extreme climate events.

[0004] Urban planning can utilize remote sensing data to analyze the urbanization process, understand the changes in land use patterns, and evaluate the impact of the urban heat island effect. In addition, in many aspects such as traffic planning, population distribution surveys, and the layout of urban greening and public facilities, valuable perspectives and references can be obtained from remote sensing data.

[0005] However, in practical applications, in the face of the massive amount of remote sensing satellite image data generated daily, how to achieve effective data acquisition, management, and processing is undoubtedly a huge challenge. Traditional data processing methods often struggle to meet the real-time processing and analysis requirements of such high-density, high-speed, and highly diverse data. Therefore, the introduction of big data governance technology, especially the development of a multi-type remote sensing satellite image data automatic acquisition into-cloud system combined with a cloud computing platform, will be one of the key technical routes to solve this problem.

[0006] Existing acquisition systems, such as the Earth Observing System Data and Information System (EOSDIS) of NASA; the Earth Observation Data Center of ESA using grid and cloud technologies; global remote sensing data exchange networks (such as CEOS), etc., have established some cross-domain data sharing and backups, partially ensuring data security and smoothness. Some domestic data service networks can provide some professional data storage and management services. For example: the High-Resolution Earth Observation System, etc.

[0007] However, the existing acquisition systems lack the automation of collecting and converting various types of remote sensing data and lack a unified service interface to provide services externally.

[0008] For example, the invention application with the application number 202411170870.3 discloses a satellite remote sensing image acquisition system and a satellite remote sensing image acquisition method. Through the method, fiber optic transmission can be achieved between the signal conversion device and the signal acquisition device, and the data transmission rate is greatly increased. However, its solution is only applicable to the acquisition of single-type remote sensing data and cannot collect and process multi-type remote sensing satellite image data. Summary of the Invention

[0009] The purpose of the present invention is to provide a multi-type remote sensing satellite image data automatic acquisition and cloud entry system and method, which can automatically collect and process various types of remote sensing satellite image data.

[0010] The embodiments of the present invention provide a multi-type remote sensing satellite image data automatic acquisition and cloud entry system and method.

[0011] First aspect: A multi-type remote sensing satellite image data automatic acquisition and cloud entry system, comprising:

[0012] A data acquisition module, which uses Agent to parse metadata and obtain a metadata database table;

[0013] A cluster DB module, which stores the obtained metadata database table;

[0014] A data processing module, which is scheduled by the data cloud entry module, pulls the metadata database table in the cluster DB, reads the image data at the specified location according to the metadata database table, and performs processing and COG conversion;

[0015] A COG folder module, which temporarily stores the converted COG files;

[0016] A data cloud entry module, which performs cloud storage on the converted COG files.

[0017] Second aspect: An automatic acquisition and cloud entry method based on the system in the first aspect, comprising:

[0018] An automatic acquisition and cloud entry method for multi-type remote sensing satellite image data based on the system according to claim 1, comprising:

[0019] S1. Pull the metadata information of the image data, use Agent to parse the metadata, obtain the metadata database table, and store it;

[0020] S2. Pull the stored metadata database table, read the image data at the specified location according to the metadata database table, and perform processing and COG conversion;

[0021] S3. Store the converted COG file in cloud storage.

[0022] Further, S1 includes the steps of:

[0023] S11. Use the data acquisition module to obtain the metadata protocol of the specified type of image data;

[0024] S12. Select the corresponding adapter according to the metadata protocol to form an Agent parsing combination;

[0025] S13. Parse the metadata of the specified type of image data according to the Agent to obtain a uniformly formatted metadata database table, and store it in the cluster DB.

[0026] Among them, the uniformly formatted metadata database table is an SQL field, and the SQL field includes the self-owned data fields and retrieval common fields of each image data metadata.

[0027] Further, the data acquisition module is built-in with an image data exception data filtering rule, and when obtaining metadata, data cleaning is performed on each metadata protocol data.

[0028] Further, S2 includes the steps of:

[0029] S21. The data processing module is scheduled to pull the metadata database table corresponding to the image data to be uploaded;

[0030] S22. Select the corresponding adapter, parse and read the image data according to the metadata database table, and process it;

[0031] S23. Perform COG conversion on the processed image data to obtain a COG file;

[0032] S24. Temporarily save the COG file to the COG folder, and delete the temporarily saved file after the COG file is stored in cloud storage;

[0033] Among them, when the data processing module is scheduled to pull the metadata database table, a memory paging cache strategy is adopted.

[0034] Further, the processing of the image data in S22 includes:

[0035] Use an automated data cleaning algorithm to process image data with format and information anomalies; perform rasterization processing, atmospheric correction, and projection conversion on the image data.

[0036] Further, add a thumbnail to the COG file and save it in the cloud.

[0037] Further, after the COG file is stored in cloud storage, generate a globally unique identifier (userfileid), and return and save it in the cluster DB associated with the corresponding metadata database table.

[0038] Further, when the data cloud - entry module stores the COG file in the cloud, it adopts a distributed node server solution.

[0039] Further, when the data cloud - entry module stores the COG file in the cloud, it automatically expands according to the data volume. By using the database modulo algorithm, the data is evenly distributed on different node servers.

[0040] In a third aspect: An electronic device includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the steps of the method provided in the second aspect.

[0041] In a fourth aspect: A non - transitory computer - readable storage medium stores a computer program thereon. When the computer program is executed by a processor, it implements the steps of the method provided in the second aspect.

[0042] Advantages of the present invention:

[0043] 1. The system of the present invention can process various types of remote - sensing satellite image data, including different image data such as optical, radar, and infrared, enabling it to serve a wider range of application scenarios. It supports the automatic expansion of acquisition nodes. The system can dynamically adjust resources according to the data volume and access requirements to achieve elastic scaling. This not only improves the reliability of the system but also optimizes the use of storage and computing resources and reduces the operating cost. The implementation of this system based on big - data governance technology provides an efficient, reliable, and intelligent solution for the automatic acquisition, management, and application of remote - sensing data.

[0044] 2. On the basis of optimizing and integrating in combination with industry standards and existing technologies, the present invention, based on big - data governance technology, can significantly improve the efficiency of image data processing, reduce manual intervention, and achieve fast and real - time data collection and storage. The application of big - data governance technology enables the system to effectively manage and maintain large - scale remote - sensing data. Through a unified data management strategy, the system can ensure the quality and consistency of data and at the same time provide efficient data retrieval and access. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a schematic flow chart of a method for automatically collecting and cloud - entering multi - type remote - sensing satellite image data of the present invention;

[0046] Figure 2 It is a schematic structural diagram of a carbon - emission meta - database table of the present invention;

[0047] Figure 3 It is a schematic structural diagram of a globally unique identifier (userfileid) of the present invention;

[0048] Figure 4 Schematic diagram of the master-slave storage structure adopted by the present invention;

[0049] Figure 5 Statistical chart of the present invention for timing monitoring of collected image data;

[0050] Figure 6 Node diagram of the automated control strategy configuration for task management of the present invention;

[0051] Figure 7 Schematic diagram of the modulo algorithm of the present invention;

[0052] Figure 8 Schematic diagram of the structure of the electronic device of the present invention. Detailed implementation manners

[0053] The embodiments of the present invention will be described in detail below. The examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as a limitation to the present invention.

[0054] Remote sensing satellite image data is the information of the Earth's surface collected by satellite sensors, covering a variety of data types, and each type can provide valuable information for applications in different fields. Common types of satellite remote sensing data include:

[0055] Optical remote sensing data: Visible light band data: including data in bands such as red, green, and blue light, used for monitoring vegetation, land cover, urban development, etc. Infrared band data: including near-infrared and thermal infrared data, used for monitoring vegetation health, land temperature distribution, etc. Multispectral data: optical data covering multiple bands, which can be used for environmental monitoring, agricultural management, etc. Radar remote sensing data: X-band, C-band, L-band, etc.: used in fields such as topographic mapping, land use monitoring, and ocean monitoring, with the advantage of penetrating clouds. Synthetic aperture radar data (SAR): provides information such as surface elevation, vegetation cover, and terrain, and is not affected by weather. Thermal infrared remote sensing data: Thermal infrared data: used for measuring surface temperature, urban heat island effect, environmental changes, etc. Hyperspectral remote sensing data: Hyperspectral data: contains dozens to hundreds of band data, which can be used for advanced applications such as vegetation species classification and land use type identification. Meteorological remote sensing data: Cloud map data: can be used for weather forecasting, disaster monitoring, etc. Atmospheric composition data: such as atmospheric temperature, humidity, gas concentration, etc., used for climate research. Temporal remote sensing data: Time series data: can track changes on the Earth's surface, such as vegetation growth, urban expansion, etc.

[0056] Through continuous collection of various types of remote sensing data by various satellites, in practical applications, in the face of the massive amount of remote sensing satellite image data generated daily, how to achieve effective data collection, management, and processing is undoubtedly a huge challenge. Traditional data processing methods often struggle to meet the real-time processing and analysis requirements of multi-type, high-density, high-speed, and highly diverse data.

[0057] To address the above problems, the present invention provides a multi-type remote sensing satellite image data automatic acquisition and cloud entry system. Figure 1 The following is the structural flowchart of the multi-type remote sensing satellite image data automatic acquisition and cloud entry system provided by the embodiment of the present invention. The system includes:

[0058] A data acquisition module, a data processing module, a data cloud entry module, a COG folder module, a cluster DB module, etc.

[0059] Each module jointly performs timed or real-time acquisition and storage of various remote sensing satellite image data, and realizes dynamic storage expansion according to the size of the real-time image data volume, which can meet the high acquisition rate requirements.

[0060] The system uses a supporting task scheduling center to combine different adapters for unified deployment to form an Agent, and sets corresponding automatic execution logic for automatic acquisition. Through customized adapter combination, the Agent realizes the parsing of remote sensing satellite image data of different protocols, achieving the purpose of once customization, permanent acquisition, and permanent update.

[0061] Among them, the data acquisition module uses the Agent to parse the metadata, obtains the metadata database table, and stores it in the cluster DB module; the data processing module pulls the metadata database table in the cluster DB, reads the image data at the specified location according to the metadata database table, and performs processing and COG conversion; after temporarily storing in the COG folder module; uses the data cloud entry module to perform cloud storage of the converted COG file.

[0062] Through the above module settings, the multi-type remote sensing satellite image data automatic acquisition and cloud entry system can be based on a distributed architecture and through modular design, maximizing the deployment flexibility of the system and the simplicity of docking. Through the automatic acquisition and cloud entry system, the image data collected by various remote sensing satellite sensors can be automatically acquired and stored, and then provided with services through the unified service interface of the cloud server, greatly reducing the difficulty of using remote sensing satellite image data.

[0063] Based on the above automatic acquisition and cloud entry system of the system, the present invention discloses a multi-type remote sensing satellite image data automatic acquisition and cloud entry method, including:

[0064] S1. Pull the metadata information of the image data, use the Agent to parse the metadata, obtain the metadata database table, and store it.

[0065] As Figure 1 shown, the Agent in the data acquisition module is customized according to the image data metadata protocol, and parses the metadata according to the specified protocol and stores it in the cluster DB. Specifically:

[0066] First, use the data acquisition module to obtain the image data metadata protocol of the specified type; then, according to the metadata protocol, select the corresponding adapter to form an Agent parsing combination; and then, according to the Agent to parse the metadata of the specified type of image data, obtain a uniformly formatted metadata database table and store it in the cluster DB.

[0067] First, clarify the types of remote sensing satellites and data characteristics of the image data that need to be identified and collected, such as: optical data, SAR data, etc. Obtain the image data metadata protocol of the specified type, customize the corresponding Agent through the metadata protocol to form an Agent parsing combination (such as parsing data addresses, fluctuation information, etc.); through the Agent parsing logic and the data structure generation tool in the data acquisition module, through the customized multiple Agent parsing logics, by parsing these metadata protocols and storing them in the database, reverse generate a uniformly formatted SQL field metadata database table, and obtain a set of metadata database tables for this type of image data. The metadata database table corresponds to the actual storage address of the image data.

[0068] The metadata database table uses uniformly formatted SQL fields, and the SQL fields include the own data fields of each image data metadata and the retrieval general fields.

[0069] For the automatic identification and collection of multi-type remote sensing satellite image data, because the image data metadata from different remote sensing satellites have different formats and standards, it is necessary to standardize the image data metadata to meet the subsequent usage requirements. The uniformly formatted SQL fields include the own data fields of each image data metadata and the retrieval general fields.

[0070] As Figure 2 shown, the own data fields of the carbon emission data include the image acquisition time, band value, file name, file type, dataset name, and bbox spatial data, etc.; others such as: storage location, creation time, user file id, etc. can be saved as retrieval general fields. Generally, due to the generality of their subsequent usage, the retrieval general fields will be stored in the database together with the fields of each type of image data metadata protocol. The own data fields and the retrieval general fields together form the general data protocol standard for image data to be uploaded to the cloud.

[0071] The data acquisition module also has built-in common image data abnormal data filtering rules. During real-time acquisition, it will perform data cleaning on each metadata protocol data, process missing data, eliminate useless information, and decode and convert metadata formats for further analysis.

[0072] The management of metadata database tables adopts the establishment of a dedicated cluster DB to handle the storage and retrieval of metadata database tables. These servers are configured with fast IO and sufficient memory to support high-speed metadata access. The separate storage of metadata and file data allows the system to independently optimize and manage metadata.

[0073] In order to further improve performance, the metadata database table reading operation will be cached on the client or server side to reduce the number of visits to the metadata server; when the data processing module is scheduled to pull the metadata database table, a memory paging cache strategy is adopted to perform memory paging cache on the metadata database table with large amounts of data, which can improve the speed of pulling the metadata database table, and can also realize the page classification of the metadata database table, thereby improving the access speed to the image data.

[0074] The management of metadata database tables is highly available and fault-tolerant, and high availability and persistence of metadata database table storage are ensured through technologies such as multiple copies, cluster settings, and heartbeat detection.

[0075] At the same time, replication techniques such as Figure 4 As shown, a slave database that is exactly the same as the master database can be copied. The master database can read and write data, and the slave database can only read data. This solution can ensure data backup and redundancy in the master database cluster and ensure data security.

[0076] S2. Pull the stored metadata database table, read the image data of the specified position according to the metadata database table, and perform processing and COG conversion.

[0077] The data processing module is a data processing tool library that can be called from the data cloud module in the dispatch center to perform customized operations on the image data cloud process, such as Figure 1 As shown, the data processing module is scheduled to pull the metadata database table corresponding to the image data to be uploaded; select the corresponding adapter, read the image data according to the metadata database table and the adapter analysis, and process it; perform COG conversion on the processed image data to obtain the COG file.

[0078] The data processing module can provide a variety of data processing operations, such as:

[0079] Use an automated data cleaning algorithm to process files with format and information anomalies in image data, eliminate damaged image data files, and decode and convert the image data format for further analysis. Effectively integrating image data from different sources and types can increase the information content and quality of the data, providing more comprehensive and accurate information for various applications.

[0080] Perform rasterization on the image data. Rasterization is to convert different data types into a unified raster format for subsequent processing and integration. Perform atmospheric correction on the image data. Atmospheric correction is to perform atmospheric correction on optical image data to eliminate the influence of the atmosphere, clouds, etc. and improve the data quality; perform projection conversion on the image data. Projection conversion is to perform operations such as projecting a remote sensing satellite file into a specified coordinate system, sinusoidal projection, and affine transformation.

[0081] The data processing module can use the GDAL toolkit to convert TIF format image data into COG format image data, obtain the COG file, and then temporarily save the COG file to the COG folder. After the COG file is stored in the cloud, the temporarily saved file is deleted, which can speed up the rapid processing of the COG file.

[0082] At the same time, add a thumbnail to the COG file and save it in the cloud to facilitate preview when using the COG file.

[0083] Furthermore, after the COG file is stored in the cloud, the system assigns a globally unique identifier to each COG file and the corresponding image data, such as Figure 3 shown. Use userfileid to establish the association between them, and return the cluster DB to associate with the corresponding meta-database table and save it. After it is stored in the cloud, the download path corresponds to the record saved in the meta-database table to establish the corresponding association relationship. It is convenient for subsequent retrieval and positioning of the meta-database table. A field index can be established in the meta-database table set, and an additional index service is used to support complex query requirements and improve retrieval performance.

[0084] S3. Use the data cloud storage module to store the converted COG file in the cloud.

[0085] The data cloud storage module uses a multi-node server. Through the meta-database table and the specified data processing mode, the scheduling center is used to schedule the data processing module to perform cloud storage operations on the image data processed by the data processing module. The data cloud storage module will extract the data in the table, upload the specified converted data to the cloud storage, and write the returned cloud storage result back to the corresponding record in the cluster DB table in real time, so that the download path after it is stored in the cloud corresponds to the image data record to establish the corresponding association relationship for convenient subsequent retrieval and call.

[0086] The data cloud storage module realizes dynamic expansion through the scheduling center to meet the rapid cloud storage demand under a large amount of data.

[0087] The data cloud input module can adopt the storage structure of NameNode, DataNode, and client library.

[0088] Among them, the NameNode is used to manage the metadata database table, including the directory tree structure of the image data file system, and the metadata of files and directories, such as permissions, sizes, and block mappings of files. By default, the cluster DB is used to store the image data protocol metadata database table, and other storage engines such as HDFS of Hadoop and ES can be replaced.

[0089] The DataNodes are distributed on multi-node servers and are used to store actual image data. Each node server manages the storage of data blocks in the local file system and can support object storage systems such as Qiniu Cloud or SGCloud.

[0090] The read and write access to the image data files can be carried out through the client library. The client will first exchange with the NameNode to obtain the metadata database table information, and then obtain the actually stored image data from the DataNode according to this information.

[0091] Furthermore, the data acquisition module and the data cloud input module can be connected through WeSocket. By configuring the corresponding nodes to customize the Corn expression, the scheduling center can call the specified nodes according to the Corn expression logic to execute tasks to achieve automated control of the node operation, such as Figure 6 shown:

[0092] Furthermore, the system can use a custom script to perform timed monitoring on the collected image data and monitor the processing progress of the image data, such as Figure 5 shown.

[0093] Furthermore, the data cloud input module adopts a database modulo algorithm, which supports dynamic expansion of nodes and automatic distribution of data, realizing automatic expansion, so that the data can be evenly distributed on multiple data node servers. The specific principle is:

[0094] The database modulo algorithm is a data sharding strategy. As Figure 7 shown, if the data cloud input module adopts multi-node servers, direct centralized operation on a single database often encounters performance bottlenecks. For example, scalability and concurrent processing capabilities will be limited. Through the database modulo algorithm, the system can evenly distribute the data on different node servers, improving the overall scalability and performance.

[0095] The database modulo algorithm uses the modulo operation (taking the remainder) in mathematics to distribute data to different shards. Specifically:

[0096] First, set an N value to determine the number of shards (N), which represents the total number of shards in the system. Then, select one or more fields as shard keys. Common shard keys include fields with good hashing properties such as user ID and order number in the metadata database table, etc. The value of this or these fields will be used to calculate the shard location. Generally, the shard key should ensure uniform distribution of data. Then, apply the modulo function to perform a modulo operation on the value of the shard key. The calculation formula is:

[0097] Shard location = (shard key value) mod (N)

[0098] The result of the operation will be an integer from 0 to N - 1, and this integer represents the shard number where the data should be stored.

[0099] Taking the user ID as an example of the shard key, if there are 10 shards, the data with user ID 1001 will be stored on the shard numbered 1001 mod 10 = 1.

[0100] When it is necessary to add data to the cloud module node server to expand the system, only the N value needs to be adjusted. The cloud module can then re - obtain its own number and query the data corresponding to its own number through the modulo algorithm. The incoming cloud data will be redistributed to all nodes to achieve the purpose of automatic expansion.

[0101] The system of the present invention can process various types of remote - sensing satellite data, including data from different sensors such as optical, radar, and infrared, enabling it to serve a wider range of application scenarios. It supports automatic expansion of the acquisition nodes. The system can dynamically adjust resources according to the data volume and access requirements to achieve elastic scaling. This ability not only improves the reliability of the system but also optimizes the use of storage and computing resources and reduces the operating cost. The implementation of this system based on big - data governance technology provides an efficient, reliable, and intelligent solution for the automated acquisition, management, and application of remote - sensing data.

[0102] The present invention also provides an electronic device, Figure 8 which is a schematic structural diagram of the electronic device provided in the embodiment of the present invention. As Figure 8 shown, the electronic device may include: a processor, a communication interface, a memory, and a communication bus. Among them, the processor, the communication interface, and the memory complete communication with each other through the communication bus. The processor can call the logical instructions in the memory, for example, to execute the following method:

[0103] S1. Pull the metadata information of the image data, parse the metadata using the Agent to obtain the metadata database table, and store it;

[0104] S2. Pull the stored metadata database table, read the image data at the specified location based on the metadata database table, and perform processing and COG conversion;

[0105] S3. Store the converted COG file in cloud storage.

[0106] In addition, when the logical instructions in the above-mentioned memory are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs that can store program codes.

[0107] The embodiments of the present invention also provide a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it is configured to execute the methods provided in the above-mentioned various embodiments, for example, including:

[0108] S1. Pull the metadata information of the image data, use Agent to parse the metadata, obtain the metadata database table, and store it;

[0109] S2. Pull the stored metadata database table, read the image data at the specified location based on the metadata database table, and perform processing and COG conversion;

[0110] S3. Store the converted COG file in cloud storage.

[0111] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0112] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the essence of the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-type remote sensing satellite image data automatic collection and cloud system, characterized in that: include: The data collection module uses Agent to parse metadata and obtain metadata database tables; Cluster DB module, used to store the obtained metadata database table; The data processing module is dispatched by the data cloud module to pull the metadata database table in the cluster DB, read the image data of the specified location according to the metadata database table, and perform processing and COG conversion; COG folder module, temporarily storing converted COG files; Data cloud module, which stores the converted COG files in the cloud.

2. The method for automatically collecting and entering data into the cloud based on the system of claim 1, characterized in that: include: S1, pull metadata information of image data, use Agent to parse metadata, obtain metadata database table, and store it; S2, pull the stored metadata database table, read the image data of the specified position according to the metadata database table, and perform processing and COG conversion; S3. Store the converted COG file in the cloud.

3. The method according to claim 2, characterized in that The S1 comprises the steps of: S11, using the data acquisition module to obtain a specified type of image data metadata protocol; S12, according to the metadata protocol, select the corresponding adapter to form an Agent parsing combination; S13, parsing metadata of the specified type of image data according to the Agent, obtaining a metadata database table with a unified format, and storing it in the cluster DB; The metadata database table with a unified format is an SQL field, and the SQL field includes the metadata own data field of each image data and a general search field.

4. The method according to claim 2, characterized in that: The data acquisition module has built-in abnormal data filtering rules for image data, and when acquiring metadata, each metadata protocol data is cleaned.

5. The method according to claim 2, characterized in that: The S2 comprises the steps of: S21, the data processing module is scheduled to pull the metadata database table corresponding to the image data to be uploaded; S22, selecting a corresponding adapter, parsing and reading the image data according to the metadata database table, and processing it; S23, performing COG conversion on the processed image data to obtain a COG file; S24, temporarily saving the COG file to the COG folder, and deleting the temporarily saved file after the COG file is stored in the cloud; Among them, when the data processing module is scheduled to pull the metadata database table, a memory paging cache strategy is adopted.

6. The method according to claim 5, characterized in that The processing of the image data in S22 includes: Use automated data cleaning algorithms to process image data with abnormal formats and information; perform rasterization, atmospheric correction and projection conversion on image data.

7. The method according to claim 2, characterized in that Add thumbnails to COG files and save them to the cloud.

8. The method according to claim 2, characterized in that: After the COG file is stored in the cloud, a globally unique identifier (userfileid) is generated, and the corresponding metadata database table associated with the cluster DB is returned and saved.

9. The method according to claim 2, characterized in that: The data cloud module uses a distributed node server solution when storing COG files in the cloud.

10. The method according to claim 9, characterized in that When the data cloud module stores COG files in the cloud, it automatically expands the capacity according to the amount of data and uses the database modulus algorithm to evenly distribute the data on different node servers.

Images