Distributed multi-cloud-node remote sensing data synchronous transmission method, system and equipment

Abstract

The invention provides a distributed multi-cloud-node remote sensing data synchronous transmission method, system and device, and the method comprises the steps: receiving a data synchronization request of a user, the data synchronization request taking a data set defined by business logic as a synchronization unit, and the logic of the data set being defined by at least one piece of business dimension information; in response to the data synchronization request, extracting all data entities corresponding to the data set logic from the distributed data source according to the service dimension information; decomposing a synchronization task for all the data entities into sub-tasks corresponding to different data types; and on the basis of the transmission strategy, after to-be-transmitted data in the subtasks is preprocessed, transmission is performed through the dynamically selected network link, so that the destination cloud node executes post-processing operation corresponding to preprocessing on the received data, and a data set is restored.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of remote sensing data processing, in particular to a distributed multi-cloud node remote sensing data synchronous transmission method, system and device. BACKGROUND

[0002] In recent years, China's space remote sensing satellite field has developed rapidly. High-resolution, ocean, mapping and cloud series satellites have been launched and operated in orbit, and have been widely used in land observation, ocean monitoring, geographic mapping, meteorological and hydrological professional fields.

[0003] Satellite remote sensing data in the field of remote sensing satellites is typical multi-modal data, and remote sensing data transmission needs to cover various multi-modal data. At present, for the synchronization and transmission of various types of data, general transmission tools are used respectively to synchronize and transmit, but these general tools have scattered functions, are not strong in pertinence, have low transmission efficiency, lack unified management, are difficult to control, and have low data utilization efficiency. SUMMARY

[0004] In view of the above problems, the present application provides a distributed multi-cloud node remote sensing data synchronous transmission method, system and device, which at least solves one of the above problems.

[0005] According to a first aspect of the present application, a distributed multi-cloud node remote sensing data synchronous transmission method is provided, the method comprising:

[0006] receiving a data synchronization request of a user, the data synchronization request taking a data set defined by business logic as a synchronization unit, and the logic of the data set being defined by at least one business dimension information;

[0007] in response to the data synchronization request, extracting all data entities corresponding to the data set logic from a distributed data source according to the business dimension information;

[0008] decomposing the synchronization task for the all data entities into sub-tasks corresponding to different data types;

[0009] After pre-processing the data to be transmitted in the sub-tasks based on a transmission strategy, transmitting through a dynamically selected network link, so that the destination cloud node performs post-processing operations corresponding to the pre-processing on the received data to restore the data set.

[0010] According to an embodiment of the present application, the data set defined by the business logic corresponds to a business logic organization unit in the field of remote sensing data processing, and the business dimension information includes at least one of satellite model, load type, shooting time and spatial position.

[0011] The business logic organization unit includes track observation data, data segments, data scenes, or standard products.

[0012] According to an embodiment of the present application, the response to the data synchronization request, according to the business dimension information, extracting all data entities corresponding to the data collection logic from the distributed data source includes:

[0013] Receive the user's input in natural language form synchronization demand description;

[0014] The remote sensing large language model is used for semantic analysis of the synchronization demand description, and the business dimension information defining the data collection logic is identified and extracted;

[0015] According to the business dimension information, a structured query statement is generated;

[0016] Based on the structured query statement, the structured data related to the data collection logic is extracted from the registered relational database, in-memory database or space-time database, and the structured data includes metadata, attribute data and feature parameter data;

[0017] According to the entity data association relationship recorded in the metadata, the unstructured entity data file stored in the memory cache, high-speed file storage system or distributed object storage is extracted;

[0018] The structured data and the unstructured entity data file are logically associated to jointly constitute all data entities of the data collection defined by the business logic.

[0019] According to an embodiment of the present application, the synchronization task for the all data entities is decomposed into subtasks corresponding to different data types, including:

[0020] Identify the form of data in the all data entities;

[0021] The synchronization task is decomposed into at least two different nature subtasks, including database synchronization subtask and data file synchronization subtask:

[0022] For the data file synchronization subtask, according to the logical organization structure of the unstructured entity data file, the subtask of parallel transmission is divided and created with data scene as the basic unit.

[0023] According to an embodiment of the present application, the database synchronization subtask is used to synchronize the structured data stored in the relational database, in-memory database or space-time database;

[0024] The data file synchronization subtask is used for synchronizing unstructured entity data files stored in a memory cache, a high-speed file storage system or a distributed object storage.

[0025] According to an embodiment of the present application, the preprocessing operation comprises a compression operation, and the preprocessing of the to-be-transmitted data in the subtask based on the transmission strategy comprises:

[0026] obtaining file attributes and service characteristics of the to-be-transmitted data, wherein the file attributes comprise a file extension and an internal format signature, and the service characteristics are defined by service dimension information contained in the data synchronization request;

[0027] matching the file attributes and the service characteristics with a predefined compression strategy selection table, wherein the compression strategy selection table records a mapping relationship between file types and compression algorithms and compression parameters;

[0028] according to a matching result, automatically selecting and invoking a corresponding compression algorithm and compression parameters to perform a compression operation on the to-be-transmitted data.

[0029] According to an embodiment of the present application, the preprocessing operation comprises a block operation, and the preprocessing of the to-be-transmitted data in the subtask based on the transmission strategy comprises:

[0030] identifying file types and network environment states of the to-be-transmitted data;

[0031] selecting a corresponding block mode from a plurality of block strategies according to the file types of the to-be-transmitted data;

[0032] dynamically adjusting sizes of data blocks of the to-be-transmitted data according to the network environment states;

[0033] The block strategy at least comprises at least one of the following:

[0034] for a file type with a fixed frame format, performing block according to a fixed number of complete data frames;

[0035] for a raster image file type, performing block according to a gridding mode;

[0036] for a video file type, performing block according to a fixed time length;

[0037] for a file type with an unclear format, performing block according to a fixed byte size.

[0038] According to an embodiment of the present application, the preprocessing operation comprises a memory cache operation, and the preprocessing of the to-be-transmitted data in the subtask based on the transmission strategy comprises:

[0039] constructing a memory virtualization logical cache pool by using a memory of a server node;

[0040] Load the data to be transmitted into the memory virtualization logical cache pool for caching to perform read and write access.

[0041] According to an embodiment of the present application, after the pre-processing of the data to be transmitted in the sub-task is performed based on the transmission strategy, the data is transmitted through a dynamically selected network link, which includes:

[0042] Based on the transmission strategy configured independently for each sub-task, corresponding pre-processing operations are performed on the data to be transmitted at the sending end;

[0043] According to the attributes of the sub-task and the real-time state of multiple network links, a target link is dynamically selected from the multiple network links;

[0044] The pre-processed data to be transmitted is transmitted through the target link.

[0045] According to an embodiment of the present application, the destination cloud node performs post-processing operations corresponding to the pre-processing on the received data to restore the data set, which includes:

[0046] Identify the type of pre-processing operation experienced by the received data;

[0047] According to the pre-processing operation type, a corresponding post-processing service is called to perform an inverse operation;

[0048] Perform integrity and correctness checks on the data subjected to the inverse operation;

[0049] Store the structured data and unstructured entity data files that pass the check according to the corresponding business logic association relationship to generate a data set consistent with the business logic definition at the sending end.

[0050] According to an embodiment of the present application, the transmission through a dynamically selected network link includes:

[0051] A protocol abstraction interface layer is provided, which provides a unified data transmission interface upwardly and encapsulates the underlying implementation of multiple transmission protocols downwardly;

[0052] At the time of transmission initialization, based on real-time network state, transmission task characteristics and destination cloud node support capabilities, a target transmission protocol is dynamically selected from the multiple transmission protocols;

[0053] During the transmission process, transmission performance is monitored in real time, and the parameters of the target transmission protocol are dynamically adjusted for optimization;

[0054] When the target transmission protocol connection fails, a protocol rollback mechanism is started to automatically switch to an alternative transmission protocol for data transmission.

[0055] According to a second aspect of the present application, a distributed multi-cloud node remote sensing data synchronization transmission system is provided, the system comprising:

[0056] A remote sensing data source configuration and synchronization customization module is configured to receive a user's data synchronization request and extract all data entities corresponding to the logical data set from the distributed data source according to the business dimension information contained in the data synchronization request, the data synchronization request taking the data set defined by the business logic as the synchronization unit, and the logic of the data set being defined by at least one business dimension information;

[0057] A data synchronization task management and scheduling module is connected to the remote sensing data source configuration and synchronization customization module and configured to receive the data entities and decompose the synchronization tasks for the data entities into subtasks corresponding to different data types and schedule the subtasks;

[0058] A to-be-synchronized data preprocessing module is connected to the data synchronization task management and scheduling module and configured to perform preprocessing operations on the to-be-transmitted data in the scheduled subtasks based on a transmission strategy;

[0059] A database synchronization and transmission service module is connected to the data synchronization task management and scheduling module and configured to execute the database synchronization subtasks;

[0060] A data file synchronization and transmission service module is connected to the data synchronization task management and scheduling module and configured to execute the data file synchronization subtasks and transmit through dynamically selected network links;

[0061] A synchronization data cache management module is connected to the to-be-synchronized data preprocessing module, the database synchronization and transmission service module, and the data file synchronization and transmission service module, respectively, and is configured to construct and manage a memory virtualization logical cache pool at the sending end and the destination cloud node to provide high-speed read-write access support for the preprocessing, transmission, and post-processing of the to-be-transmitted data.

[0062] According to a third aspect of the present application, an electronic device is provided, comprising:

[0063] One or more processors;

[0064] A memory for storing one or more computer programs,

[0065] The one or more processors execute the one or more computer programs to implement the steps of the method of the first aspect.

[0066] According to a fourth aspect of the present application, a computer readable storage medium is provided, which stores a computer program or instructions, which, when executed by a processor, implements the steps of the method according to the first aspect.

[0067] According to the embodiments of the present application, at least one of the following technical effects can be achieved:

[0068] (1) The method of the present application synchronizes and transmits the business logic organization unit of multi-modal remote sensing data as the basic unit, which enables the business user to truly propose synchronization and transmission requirements from the perspective of remote sensing data processing business, rather than individually specifying specific data files, directories or database records as in the traditional method. At the same time, according to the remote sensing data set to be synchronized, a multi-modal data synchronization sub-task is automatically created to realize unified management, state monitoring and centralized access of multi-modal data synchronization and transmission.

[0069] (2) Through the memory caching service, the single-point read-write bandwidth of the remote sensing data of the present application can reach 8GB / S, which can significantly improve the speed of remote sensing data encryption and decryption, compression and decompression, block merging, network transmission and subsequent processing, loading and display, and realize efficient and flexible synchronization and transmission.

[0070] (3) The link monitoring and optimization service proposed by the present application can select the most suitable transmission link according to the synchronization task needs, and realize cross-domain and cross-network segment data transmission.

[0071] (4) The multiple compression methods designed by the present application for multi-modal remote sensing data can use targeted compression algorithms for different data compared to single general compression methods, which can significantly improve compression efficiency and quality and enhance the efficiency of compression transmission.

[0072] (5) The multiple data block methods designed by the present application for multi-modal remote sensing data can ensure that each data block can be transmitted, processed and utilized independently without waiting for other data blocks to form a complete data frame, support reading partially transmitted files, support viewing partially transmitted images, and support viewing partially transmitted formatted files, thereby effectively supporting edge transmission and edge use.

[0073] (6) The multiple transmission protocols adapted by the present application can better adapt to remote sensing data processing scenarios and more flexibly adapt to various network environments, and have better adaptability.

[0074] (7) The application can propose remote sensing data synchronization and transmission requirements in a natural language interactive question and answer manner, without giving technical details such as data file paths or database links as in traditional methods, thus greatly reducing the difficulty of use and improving the ease of use. For example: in the system, the user can propose synchronization requirements in a similar manner: “Please synchronize all remote sensing images of today's activities to cloud center A for the production of special topic products.” BRIEF DESCRIPTION OF DRAWINGS

[0075] The above and other objects, features and advantages of the present application will become more apparent from the following description of embodiments of the present application, taken in conjunction with the accompanying drawings, in which:

[0076] Figure 1 A distributed multi-cloud node remote sensing data synchronization transmission system architecture diagram according to an embodiment of the present application is schematically shown;

[0077] Figure 2 A distributed multi-cloud node remote sensing data synchronization transmission system diagram according to an embodiment of the present application is schematically shown;

[0078] Figure 3 A data source configuration and synchronization customization module architecture diagram according to an embodiment of the present application is schematically shown;

[0079] Figure 4 A data synchronization task management and scheduling module architecture diagram according to an embodiment of the present application is schematically shown;

[0080] Figure 5 A data source configuration and synchronization customization module architecture diagram according to an embodiment of the present application is schematically shown;

[0081] Figure 6 A database synchronization and transmission service module architecture diagram according to an embodiment of the present application is schematically shown;

[0082] Figure 7 A data file synchronization and transmission service module architecture diagram according to an embodiment of the present application is schematically shown;

[0083] Figure 8 A distributed cloud node network link diagram according to an embodiment of the present application is schematically shown;

[0084] Figure 9 A synchronization data cache management module architecture diagram according to an embodiment of the present application is schematically shown;

[0085] Figure 10 A transmission cluster memory cache management architecture diagram according to an embodiment of the present application is schematically shown;

[0086] Figure 11A flowchart illustrating a distributed multi-cloud node remote sensing data synchronization and transmission method according to an embodiment of this application is shown in the schematic diagram.

[0087] Figure 12 A flowchart illustrating another distributed multi-cloud node remote sensing data synchronization and transmission method according to an embodiment of this application is shown schematically.

[0088] Figure 13 This illustration schematically depicts a usage pattern diagram of efficient remote sensing data processing and transmission based on memory caching according to an embodiment of this application;

[0089] Figure 14 A block diagram of an electronic device suitable for implementing a distributed multi-cloud node remote sensing data synchronization transmission method according to an embodiment of this application is illustrated. Detailed Implementation

[0090] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0091] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0092] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0093] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0094] Figure 1 The diagram illustrates the architecture of a distributed multi-cloud node remote sensing data synchronization transmission system according to an embodiment of this application.

[0095] As Figure 1 shown, the remote sensing data efficient synchronization and transmission system architecture of the application includes a cloud infrastructure layer, a data storage cloud service layer, a synchronization and transmission service layer, and a business user interaction layer. Among them, the cloud infrastructure layer mainly provides management control and synchronization transmission computing server cluster for the upper layer service, and storage, network, security and other device resources, and realizes virtualization, pooling and cloud management of resources through cloud platform service.

[0096] The data storage cloud service layer mainly provides storage middleware services for multi-modal remote sensing data based on the cloud platform, including: memory cache management service, high-speed file storage system based on high-speed storage device, distributed file storage system based on large-capacity storage device and OSS object storage service, as well as relational database MySql and PolarDB, in-memory database Redis, and spatiotemporal database PostgreSQL and other database services.

[0097] The synchronization and transmission service layer is the core component of the application, which provides data encryption / decryption, data compression / decompression, link monitoring and optimization, database synchronization and transmission, data file synchronization and transmission, and other core basic services, as well as data source management, synchronization task decomposition and scheduling, synchronization task monitoring and control, task load balancing, cache management and data verification and other management control services.

[0098] The business user interaction layer mainly provides a front-end interaction interface, provides synchronization data set selection, synchronization strategy configuration and task customization, synchronization task monitoring management and synchronization data cache management for remote sensing data processing system users, and calls the services of the synchronization and transmission service layer, to realize efficient synchronization and transmission of remote sensing data.

[0099] Figure 2 The distributed multi-cloud node remote sensing data synchronization transmission system according to the embodiment of the application is schematically shown.

[0100] As Figure 2 shown, the remote sensing data synchronization transmission system of the application mainly completes the synchronization and transmission of multi-modal remote sensing data between distributed multi-cloud nodes of space remote sensing ground systems, including remote sensing data source configuration and synchronization customization, data synchronization task management and scheduling, pre-processing of data to be synchronized, database synchronization and transmission service, data file synchronization and transmission service, and synchronization data cache management modules.

[0101] As Figure 3 shown, the data source configuration and synchronization customization module is mainly responsible for the registration, configuration and management of various types of remote sensing data sources, supports users to access according to permissions, and selects the data set to be synchronized and transmission, customizes and initiates the synchronization task.

[0102] Database instance registration and configuration is responsible for registering and configuring the database instances such as relational databases, in-memory databases, and spatio-temporal databases storing metadata, attribute data, and feature data of remote sensing data in various cloud nodes into the synchronization and transmission software, and managing user access permissions to various database instances. It supports one-click import of database instance information through the interface of the cloud platform.

[0103] Data storage configuration and mounting is responsible for configuring and mounting various storage system data sources such as in-memory cache, high-speed file storage, distributed file storage, and OSS object storage storing remote sensing data entity files in various cloud nodes into the synchronization and transmission software, and managing user access permissions to directories and files of various storage systems. It supports one-click import of information of various storage systems through the interface of the cloud platform.

[0104] Remote sensing data query and selection supports users with permissions to query and select remote sensing data to be synchronized from the registered data sources according to satellite models, payload types, shooting times, spatial positions, data types, data themes, and data tags, forming a remote sensing data set to be synchronized. This enables the initiation of synchronization and transmission tasks in units of remote sensing data sets. By calling the interface of the remote sensing large language model, users can propose remote sensing data synchronization requirements in a natural language interaction question-and-answer manner, realizing intelligent data synchronization.

[0105] Synchronization task strategy configuration and customization configures synchronization and transmission strategies and customizes the initiation of synchronization tasks according to the synchronization and transmission requirements of remote sensing data sets. The synchronization task supports selecting one or more destination cloud nodes, configuring task priority, urgency, timeliness requirements, encryption, compression, transmission protocol, and other strategies.

[0106] As shown in Figure 4 , the data synchronization task management and scheduling module is mainly responsible for analyzing multi-modal remote sensing data sets, decomposing them into multiple synchronization sub-tasks according to data modalities, and scheduling and executing each sub-task according to load balancing strategies.

[0107] Synchronization task analysis and decomposition: Since the remote sensing data set to be transmitted in the synchronization task usually includes multi-modal data and may include numerous data in terms of quantity, it needs to be analyzed from two dimensions of data modality and data composition. First, according to data modality, data in databases and file storage are sorted out, and data subsets are constructed for relational, in-memory, and spatio-temporal data respectively; for data in file storage, according to the logical organization structure of remote sensing data, the smallest logical unit of remote sensing data, i.e., data scene, is taken as the basic unit to divide sub-data sets.

[0108] Subtask creation and configuration: According to the analysis and decomposition of the data subsets of the synchronization task, create subtasks for data synchronization and transmission respectively, and configure the subtask strategy according to the synchronization task strategy and subtask data characteristics, and submit the scheduling execution.

[0109] Task parallel scheduling: According to the task queue and the resource state of the transmission node, the subtasks are scheduled to the transmission node cluster according to the priority, short task priority and other strategies, and then the execution nodes are allocated in the transmission node cluster according to the load balancing strategy, realizing the parallel execution of multiple tasks and multiple nodes.

[0110] Task monitoring and management: Real-time acquisition of task execution state and transmission progress, support for tracking and monitoring through visual interface; and support for user manual priority adjustment of tasks, support for task pause, cancellation, retry, speed limit and other operations.

[0111] As shown in Figure 5 , the pre-synchronization data preprocessing module is mainly responsible for performing data caching, encryption, compression, blocking, and feature calculation on demand at the sending end before synchronization and transmission, and performing decryption, decompression, block merging, integrity and correctness verification at the destination cloud node after transmission is completed.

[0112] Data cache loading: For data files stored in high-speed file storage or distributed file storage, in order to speed up the preprocessing and transmission speed and improve the timeliness, the data files are pre-read and cached into the memory cache, and the storage state and hotness of the data are updated to facilitate reuse.

[0113] Data encryption and decryption: According to the task strategy, the data is encrypted before transmission to generate encrypted data for transmission; after transmission is completed, the encrypted data is decrypted at the destination cloud node to generate clear data. Supports mainstream symmetric encryption algorithms such as DES and AES.

[0114] Data compression and decompression: According to the task strategy, suitable compression algorithms are selected for multi-modal remote sensing data. For binary files such as original code streams and formatted data, and small files such as text, XML and DOC documents, software modules such as RAR and ZIP are used for packaging and compression to improve transmission efficiency. For high-resolution, wide-area remote sensing images, videos and other large data files, lossless or micro-loss compression can be selected for overall compression. The module supports multiple image compression algorithms to ensure the best compression effect in different scenarios, including JPEG-LS, JPEG 2000, WebP, etc.; supports custom micro-loss compression parameters such as compression quality level and compression ratio. Supports automatic selection of compression algorithms, establishes a compression default strategy selection table, and automatically selects compression algorithms according to file extension, internal format signature and business characteristics. A fragment of the compression strategy selection table is shown in Table 1:

[0115] Table

[0116] File extension Data type Compression tool and algorithm Expected compression ratio *.dat; *.FRED Raw code stream, formatted data, etc. binary data RAR (split volume compression, LZ algorithm) ~6:1 or 8:1 *.tif; *.tiff; *.img; *.hdf5 Remote sensing image / thematic map JPEG2000, WebP, etc. (lossless or micro-loss) ~3:1 or 4:1 *.shp; *.geojson Vector data / high-precision planning GZIP (Deflate algorithm), Geobuf ~4:1 or 6:1 *.xml; *.json; *.txt Metadata / log / auxiliary parameters BZIP2 (Burrows-Wheeler transform) ~5:1

[0117] Data chunking and merging: For a single data file that is too large, support splitting into multiple data chunks for multi-threaded parallel transmission, and after transmission is completed, merging in order at the destination cloud node. A large file is cut into numerous data chunks, each data chunk is assigned a file chunk number, and through the chunk number, it can be accurately and correctly written into the temporary file in the cache of the destination cloud node. When the last data chunk arrives at the destination cloud node and is successfully written into the cache file, the temporary file name is changed, and a confirmation message is returned to the sender, notifying the sender that all file chunks have been received. During transmission, each data chunk can be independently encrypted, compressed, and checked. In addition to supporting the general strategy of cutting data chunks by a fixed byte size, according to the data frame format of the remote sensing data file to be cut, the application cuts data chunks according to a fixed number of data frames without damaging the complete data frame, ensuring that each data chunk contains complete data frames, facilitating transmission and processing. For large-width and large-area raster images, support image chunking in the form of a "nine-square grid" or a grid, and then use parallel transmission to achieve image loading and display while transmitting. For the size of the chunk, adjust according to the network status. In a high-bandwidth, low-latency network environment (such as within a cloud node), larger chunks (such as 8MB or 16MB) can be used to reduce protocol overhead. In a bandwidth-limited or unstable network environment (such as across cloud nodes or across regions), smaller chunks (such as 256KB or 512KB) can be used to improve transmission resilience and preemptive ability. Various types of data chunking strategies are shown in Table 2 below:

[0118] Table

[0119] Data category File extension Data description Chunking method Advantages Raw code stream *.dat AOS frame format data with fixed frame length of 1024 bytes Chunking in units of 1024-byte AOS frames Each chunk of data can be independently parallel processed Formatted data *.FRED Decrypted and decompressed formatted data with fixed frame length Chunking in units of formatted data frames of each satellite payload Each chunk of data can be parallel transmitted and processed according to data frames Remote sensing image *.tif; *.tiff; *.img; *.hdf5 Remote sensing image / thematic map, etc. raster image According to row and column numbers or latitude and longitude information, the data is divided into "checkered patterns", "nine-square patterns", etc. It can also be divided according to grid subdivision standards Each chunk of data can be independently parallel processed and loaded and displayed Remote sensing video *.mp4; *.avi Remote sensing video stream According to fixed time length, the video data is divided Edge transmission and edge broadcast Other large files Other extensions Large data volume files with unclear format According to fixed byte size, separate Parallel transmission

[0120] Data check calculation: To ensure the integrity and correctness of data synchronization and transmission, support calculating the checksum of the data to be transmitted through MD5, SHA1, and other message digest algorithms and CRC cyclic redundancy check algorithms. Calculate the checksum of the data before transmission and transmit it with the data to the destination cloud node, and then calculate the checksum again at the destination cloud node to complete the data check. In addition to checking based on message digest algorithms, the application also supports sampling checking according to the remote sensing data frame format. According to the synchronization word and feature code in the data frame format, sample the feature information at a specific location before transmission, and check the feature information at the same location after transmission to confirm whether they are consistent.

[0121] As Figure 6As shown, the database synchronization and transmission service module is mainly responsible for performing synchronization and transmission tasks of relational, memory and spatiotemporal databases, and completing synchronization and transmission of metadata and attribute data of the to-be-synchronized remote sensing data set. By using the related API interface of the cloud platform, the database instance of the destination node can be automatically created, the database structure can be created and the data can be initialized; the data records to be synchronized can be automatically queried, customized and packaged according to the association organization and dependency relationship of remote sensing data, and the integrity and correctness of the data records after synchronization and transmission are verified to ensure that the data is real and available.

[0122] Database instance creation: responsible for calling the API interface of the relational, memory and spatiotemporal database products of the cloud platform at the synchronization destination, creating the required specifications of MySQL, PolarDB, Redis and PostgreSQL database instances, and configuring login accounts, security groups, access white lists and other parameters to ensure that the database instance at the destination is available before synchronization.

[0123] Database structure synchronization and initialization: after the database instance is created, the creation scripts of the database table structure, view, custom data type, function, stored procedure, trigger and other database objects are generated at the source end, and the initialization data generation script is generated, the dependency relationship and execution order of the scripts of each database object are arranged. By copying the DDL statement, the database connection tool is called to execute the script in the destination database instance to complete the synchronization and initialization of the database structure.

[0124] Synchronization data customization and packaging: according to the association organization, dependency relationship and other information of remote sensing data, the metadata and attribute data of the to-be-synchronized remote sensing data set are automatically queried, customized and packaged to generate a set of data records to be synchronized, support three-level object name mapping of library table columns, support custom synchronization data filtering, and realize fine-grained control and customized accurate synchronization of to-be-synchronized data.

[0125] Full and incremental data synchronization: for database instance, database or data table level synchronization, the data transmission service DTS interface of the cloud platform is called to create a database synchronization link to realize full and incremental data synchronization, and the incremental log of the capture database is used to ensure real-time data transmission. Support data encryption and multi-concurrent compression transmission, support data breakpoint transmission capability, support full verification of source and target data during synchronization, and ensure the consistency of source and target data during non-stop synchronization.

[0126] As Figure 7As shown, the data file synchronization and transmission service module is mainly responsible for performing the synchronization and transmission tasks of data entity files, constructing efficient transmission links according to the storage location of remote sensing data, the strategy and source of the synchronization task, and the network status of the destination node, using appropriate network protocols, and completing the efficient transmission of data files through multi-path parallel transmission.

[0127] Link monitoring and optimization: the multiple distributed cloud nodes of the remote sensing data processing system are connected through wide-area or regional business private networks, and the network bandwidth, load status and availability between each node are different; there are even cases where some nodes cannot be directly connected due to management, security and other reasons, and need to be transferred through the central cloud node, such as Figure 8 As shown.

[0128] Therefore, the link monitoring and optimization unit is responsible for tracking, monitoring and maintaining the link status between the source node and the destination node of the synchronization and transmission task, evaluating each link through connectivity, bandwidth, load, packet loss rate, delay and other indicators, and optimizing the most matched transmission link in combination with the priority, urgency and timeliness requirements of the task and other strategies. For cross-domain and cross-segment transmission tasks without direct connectivity, the central cloud node can be used for forwarding to realize cross-domain and cross-segment data synchronization and transmission.

[0129] In some implementations, the connectivity, bandwidth, round-trip time (RTT) and packet loss rate link state indicators of the alternative link can be evaluated by periodically sending lightweight probe packets (such as ICMP Echo Request or specific small TCP / UDP data packets). These state information can also be collected by periodic heartbeat detection, broadcast information or SDN controller. Then, the real-time state of each link is recorded in the system, and a global cloud node link topology graph is maintained, which records all possible connection relationships between ground system cloud nodes and their state attributes at a specific time. Finally, the most matched transmission link is selected according to the link topology graph and task strategy.

[0130] Transmission protocol adaptation: manage and adapt the transmission protocols required by the access transmission network, which can adapt TCP, UDP, FTP, PDXP, FEP, KCP and QUIC transmission protocols, and can use appropriate network protocols for different data types and network conditions.

[0131] In some implementations, a protocol abstract interface layer can be first constructed, which provides a unified data sending / receiving interface upwards and encapsulates the underlying implementation details of different protocols downwards. When the system is initialized, the implementation class library of each protocol is loaded. Secondly, protocol selection can be dynamically determined according to real-time network status (such as delay, packet loss rate, bandwidth), transmission task characteristics (such as size, priority, real-time requirement), and support conditions of target cloud nodes. Thirdly, during the execution of the task, protocol parameters can be dynamically adjusted for optimization, such as TCP window size, retransmission times, UDP buffer size, etc. Finally, to ensure the robustness of communication, a protocol fallback mechanism is supported, for example, when QUIC connection fails multiple times, automatic downgrade to HTTP / 2 or HTTP / 1.1 over TCP is implemented by using ALPN extension for protocol negotiation during TLS handshake.

[0132] Resume: Support the resume of network data transmission, automatically record the current transmission task execution state when software exception occurs, and automatically restore the transmission task after the troubleshooting software returns to normal, and automatically resume the transmission from the breakpoint. Resume combined with large file block transmission can bring more protection for large file transmission, because resume has the ability to retransmit from the breakpoint, and block transmission brings convenience for breakpoint positioning and resume. For example, when the Nth block of data is sent, the connection is disconnected or the session is abnormal, and after the connection and session retry, the sender can continue to transmit from the last interrupted file block, without the need to start over.

[0133] Flow control: The bandwidth used by the transmission service can be set as needed to control the transmission task, such as speed limiting, recovery, etc., for supporting the implementation of transmission task priority adjustment. Through the configuration of maximum connection number, maximum parallel task number, parallel thread number, etc., dynamic adjustment of flow control is completed. By limiting the link bandwidth used by low-level tasks, more network bandwidth can be allocated to high-priority tasks. When high-bandwidth parallel transmission is needed, the characteristics of remote sensing data fragmentation, spectral segment, polarization mode, camera, etc. can be fully utilized to automatically and efficiently carry out parallel transmission. The number of parallel threads is consistent with the number of data files constituting the remote sensing data synchronization basic unit, which can maximize parallel transmission and improve overall efficiency without cutting data files.

[0134] As shown in Figure 9 , the synchronization data cache management module mainly manages the multi-level data cache at both ends of the synchronization source node and the destination node, completes the life cycle management of the cache data, and ensures the efficient execution of data transmission and processing and effective use of data.

[0135] Transmission cluster memory cache management: this unit mainly realizes the registration management and capacity adjustment management of the available memory of the transmission cluster server node, and supports memory disk information query, memory disk logout, data read-write and deletion functions. The principle diagram of the transmission cluster memory cache management is shown in Figure 10 The memory of each server node in the transmission cluster is used to make a file system, and a memory virtualization logical cache pool is constructed, which provides a unified data access interface to the software services such as synchronous data preprocessing and data transmission, realizes fast and efficient access of remote sensing data, improves the data read-write efficiency, and makes the remote sensing data preprocessing and transmission efficient and flexible.

[0136] Memory cache data management: at the sending end, the data loading and caching of the data to be synchronized from high-speed file storage, distributed file storage, etc. are realized; at the destination cloud node, the received data is cached in the memory cache disk, and the ability of writing back the data cache to high-speed file storage and distributed file storage is provided, realizing the persistence of memory cache data. At the same time, according to the memory cache usage waterline of each cloud node, the remote sensing data in the cache can be cleaned up according to the LRU (Least Recently Used) and LFU (Least Frequently Used) algorithms.

[0137] Figure 11 A flowchart of a distributed multi-cloud node remote sensing data synchronization transmission method according to an embodiment of the application is schematically shown.

[0138] As shown in Figure 11 , the distributed multi-cloud node remote sensing data synchronization transmission method includes operation S110-operation S140.

[0139] In operation S110, a data synchronization request of a user is received.

[0140] The data synchronization request takes a data set defined by business logic as the synchronization unit, and the logic of the data set is defined by at least one business dimension information.

[0141] The data set defined by business logic refers to a data unit organized according to the internal business relationship of data processing and application in the field of remote sensing data processing. The data set defined by business logic can include whole-track observation data, data segments, data scenes and standard products. The whole-track observation data can be all data obtained by a remote sensing satellite continuously observing a whole track. The data segment is a segment data file and its supporting metadata formed by dividing the whole-track data according to a certain time or geographical interval. The data scene is the smallest logical unit of remote sensing data, which refers to an image obtained by one imaging operation and its complete metadata, browsing map, auxiliary parameters, etc. The standard product is a usable data product generated by standardizing processing such as radiation calibration and geometric correction on the original observation data.

[0142] The business dimension information is a business attribute parameter used to define and query the above data set. It can include satellite model, payload type, shooting time and spatial position. The satellite model is the specific satellite model in the satellite model such as GF special project, marine series, mapping series, etc. The payload type is, for example, a panchromatic camera, a multispectral camera, a hyperspectral camera, a synthetic aperture radar (SAR), an infrared sensor, etc. The shooting time is the specific time point or time range of data collection. The spatial position is the geographical area or latitude and longitude coordinate range corresponding to the data.

[0143] In some embodiments, a data synchronization request described by a user in natural language can be received through a natural language interaction interface; a remote sensing large language model is called through the natural language interaction interface to analyze the data synchronization request, and at least one dimension information of satellite model, payload type, shooting time, spatial position and data type is extracted; based on the dimension information, the synchronization task request is generated.

[0144] In operation S120, in response to the data synchronization request, all data entities logically corresponding to the data set are extracted from the distributed data source according to the business dimension information.

[0145] The distributed data source refers to various heterogeneous databases and storage systems that store multi-modal remote sensing data in a distributed multi-cloud node environment. The distributed data source can include a database for storing structured data. The database can be a relational database, an in-memory database or a spatio-temporal database. The relational database, such as MySQL, PolarDB, is used to store metadata and attribute data. The in-memory database, such as Redis, is used to store high-frequency access feature parameters and key-value pair data. The spatio-temporal database, such as PostgreSQL (with PostGIS extension), is used to store data with spatio-temporal attributes.

[0146] The storage system is a system for storing unstructured entity data files. The storage system can be a memory cache, a high-speed file storage system and a distributed file storage / object storage, etc. The memory cache is a high-speed cache based on server memory, used for temporary storage of hot data. The high-speed file storage system is a file system based on high-speed storage devices such as SSD. The distributed file storage / object storage is, for example, NAS, OSS, HDFS, used for persistent storage of massive entity files (images, videos, etc.).

[0147] All data is all data components constituting a complete business logic data set. It can include structured data and unstructured entity data files. The structured data includes metadata, attribute data and feature parameter data. The unstructured entity data files include original code stream data files (AOS frame format), payload formatted data files, image files (such as TIFF, IMG format), video files, thumb maps, browsing map files, DOC document reports

[0148] Among them, the metadata is the information describing the data attributes, sources, and processing procedures (such as the XML format metadata file content stored in the database). The attribute data is various characteristic attributes of the observation target. The characteristic parameter data is various parameters stored in the form of key-value pairs.

[0149] In operation S130, the synchronization task for the entire data entity is decomposed into sub-tasks corresponding to different data types.

[0150] The sub-tasks corresponding to different data types can be two parallel sub-tasks decomposed from the synchronization task according to the differences in data form and storage location. The sub-tasks corresponding to different data types can include database synchronization sub-tasks and data file synchronization sub-tasks.

[0151] The database synchronization sub-task is responsible for synchronizing structured data stored in various databases to the destination node.

[0152] The data file synchronization sub-task is responsible for synchronizing unstructured entity data files stored in various storage systems to the destination node.

[0153] In operation S140, based on the transmission strategy, the pre-processing of the data to be transmitted in the sub-task is performed, and then transmitted through the dynamically selected network link, so that the destination cloud node performs the post-processing operation corresponding to the pre-processing on the received data to restore the data set.

[0154] The pre-processing / post-processing operation refers to the reversible operation performed on the data before and after transmission, aiming to improve efficiency, security, and reliability. The pre-processing / post-processing operation can include encryption / decryption, compression / decompression, blocking / merging, etc.

[0155] Encryption / decryption: using DES, AES, etc. algorithm to encrypt and decrypt the data.

[0156] Compression / decompression: selecting algorithms according to data types (such as ZIP / RAR for packaging, JPEG 2000 / WebP for image compression) to reduce data volume, and decompressing after transmission.

[0157] Blocking / merging: dividing large files into multiple data blocks (such as complete data frames or nine-square grid) for parallel transmission, and merging and restoring at the destination cloud node.

[0158] Verification: calculating and verifying the checksums such as MD5, SHA1, and CRC cyclic redundancy check, or checking the data frame synchronization header to ensure data integrity.

[0159] The dynamically selected network link refers to an optimal path selected from multiple optional paths between a source node and a destination node according to real-time network conditions. The dynamically selected network link can include: link state indicators: connectivity, bandwidth, delay, packet loss rate, load; task policy attributes: priority, urgency, timeliness requirement of the task; for cross-domain scenarios without direct links, the link can include a path through a central cloud node.

[0160] In some embodiments, a structured query statement can be generated according to the business dimension information; based on the structured query statement, structured data logically related to the data set is extracted from a registered relational database, an in-memory database, or a space-time database, the structured data including metadata, attribute data, and feature parameter data; according to the entity data association relationship recorded in the metadata, unstructured entity data files stored in an in-memory cache, a high-speed file storage system, or a distributed object storage are extracted; the structured data and the unstructured entity data files are logically associated to jointly constitute all data entities of the data set defined by the business logic.

[0161] In some embodiments, the form of the data in the all data entities can be identified; the synchronization task can be divided into at least two subtasks of different natures, including a database synchronization subtask and a data file synchronization subtask; for the data file synchronization subtask, the unstructured entity data files are divided and parallel transmission subtasks are created according to the logical organization structure of the unstructured entity data files, taking a data scene as a basic unit.

[0162] In some embodiments, the database synchronization subtask is used to synchronize structured data stored in a relational database, an in-memory database, or a space-time database; the data file synchronization subtask is used to synchronize unstructured entity data files stored in an in-memory cache, a high-speed file storage system, or a distributed object storage.

[0163] In some embodiments, a corresponding preprocessing operation can be performed on the to-be-transmitted data at the sending end based on a transmission strategy independently configured for each subtask; a target link is dynamically selected from a plurality of network links according to the attributes of the subtask and the real-time state of the plurality of network links; and the preprocessed to-be-transmitted data is transmitted through the target link.

[0164] In some embodiments, the type of preprocessing operation experienced by the received data can be identified; a corresponding post-processing service is called to perform an inverse operation according to the preprocessing operation type; the data subjected to the inverse operation is subjected to integrity and correctness checks; and the structured data and the unstructured entity data files that pass the checks are stored according to the corresponding business logic association relationship to generate the data set defined by the business logic consistent with the sending end.

[0165] In some embodiments, the preprocessing operation includes a compression operation, and the performing preprocessing on the to-be-transmitted data in the subtask based on the transmission strategy includes: obtaining file attributes and service characteristics of the to-be-transmitted data, the file attributes including a file extension and an internal format signature, and the service characteristics being defined by service dimension information contained in the data synchronization request; matching the file attributes and the service characteristics with a predefined compression strategy selection table, the compression strategy selection table recording a mapping relationship between file types and compression algorithms and compression parameters; and automatically selecting and invoking corresponding compression algorithms and compression parameters to perform a compression operation on the to-be-transmitted data according to a matching result.

[0166] In some embodiments, the preprocessing operation includes a block operation, and the performing preprocessing on the to-be-transmitted data in the subtask based on the transmission strategy includes: identifying a file type of the to-be-transmitted data and a network environment state; selecting a corresponding block method from multiple block strategies according to the file type of the to-be-transmitted data; and dynamically adjusting a size of a data block of the to-be-transmitted data according to the network environment state; wherein the block strategy at least includes at least one of the following: for a file type with a fixed frame format, performing block according to a fixed number of complete data frames; for a raster image file type, performing block according to a gridding method; for a video file type, performing block according to a fixed time length; and for a file type with an unclear format, performing block according to a fixed byte size.

[0167] In some embodiments, the preprocessing operation includes a memory caching operation, and the performing preprocessing on the to-be-transmitted data in the subtask based on the transmission strategy includes: constructing a memory virtualization logical cache pool by using a memory of a server node; and loading the to-be-transmitted data into the memory virtualization logical cache pool for caching to perform read-write access.

[0168] Figure 12 A flowchart of another distributed multi-cloud node remote sensing data synchronization transmission method according to an embodiment of the application is schematically shown.

[0169] As Figure 12 shown, the distributed multi-cloud node remote sensing data synchronization transmission method includes operations S1-S7 applied to a sending end.

[0170] S1, a business user registers and configures various database instances such as relational, memory and space-time types for source nodes and destination nodes, and registers and mounts storage resources such as memory caches, high-speed file storages and distributed storages.

[0171] S2. Based on registered and configured data sources, and according to multi-dimensional information such as satellite model, payload type, imaging time, spatial location, data type, theme, and tags, it queries and selects remote sensing datasets to be synchronized, configures synchronization and transmission strategies, and customizes and initiates synchronization and transmission tasks. Alternatively, it can intelligently initiate synchronization and transmission tasks through interactive question-and-answer sessions with remote sensing data query and selection units using natural language.

[0172] S3, the synchronization task management and scheduling module, analyzes, decomposes, and creates sub-tasks for remote sensing data synchronization tasks, and schedules each sub-task to be executed according to the task load balancing strategy and job scheduling strategy. Simultaneously, it sends task instructions to the task management and scheduling module of the destination cloud node.

[0173] S4. Based on the task strategy and the characteristics of the remote sensing data to be transmitted, such as storage location, composition, and size, perform preprocessing operations such as data caching, encryption, compression, and segmentation as needed.

[0174] S5. Assess the network status between the source and destination nodes of the synchronization task, select appropriate transmission links and transmission protocols for each subtask based on the task strategy, and call the database or data file synchronization and transmission service to send data to the destination cloud node.

[0175] S6. Users can monitor the status of synchronization tasks and perform management and control operations such as pausing, canceling, continuing, and limiting the speed as needed until the task is completed.

[0176] S7. After the data is sent, release the relevant connection resources, manage the data cached in memory according to the caching strategy, and perform lifecycle management of the cached data.

[0177] like Figure 12 As shown, the distributed multi-cloud node remote sensing data synchronization transmission method includes operations S1'-S4' applied to the target cloud node.

[0178] S1': Based on the task scheduling instructions and load balancing strategy, schedule a suitable receiving node, check the buffer space, and complete the receiving preparation.

[0179] S2': Receive transmission requests, establish transmission links, cache received data, and perform decryption, decompression, merging, and verification as needed to ensure data integrity and correctness.

[0180] S3' During the receiving process, according to the task strategy, the backend data processing software is notified to process and utilize the cached data, achieving simultaneous transmission and use.

[0181] S4' After the data is received and processed, manage the data cached in memory according to the caching strategy and perform lifecycle management of the cached data.

[0182] According to the embodiment of the present application, the method is used for synchronization and transmission in the basic unit of the business logic organization unit of remote sensing data. Different from the synchronization and transmission in the object of a single file or a database table, the method takes the logical organization unit of remote sensing data such as the whole-orbit observation data, single observation data, data segment, data scene or standard product of a remote sensing satellite as the object, extracts the multi-modal data set to be synchronized and transmitted according to the internal correlation and composition logic of remote sensing data, and then respectively calls the synchronization service of multi-modal data for synchronization and transmission.

[0183] According to the embodiment of the present application, the efficient synchronization and transmission system of remote sensing data is designed, which can realize efficient synchronization and transmission of multi-modal remote sensing data between distributed multi-cloud nodes of a space remote sensing ground system, including remote sensing data source configuration and synchronization customization, data synchronization task management and scheduling, pre-processing of data to be synchronized, database synchronization and transmission service, data file synchronization and transmission service and synchronization data cache management and other modules.

[0184] According to the embodiment of the present application, under the support of a remote sensing large language model, the user can interact with a query and selection unit in natural language and remote sensing data to query and select, propose a remote sensing data synchronization demand, so that the system can intelligently extract multi-dimensional information such as a remote sensing satellite model, a load type, a shooting time, a spatial position, a data type and a data theme, a data label and the like, and then automatically query and extract remote sensing data to be synchronized to form a remote sensing data set to be synchronized. Thus, the time and complexity of the user to find, filter and determine the data file or record to be synchronized can be greatly reduced.

[0185] According to the embodiment of the present application, the transmission cluster memory cache service is designed, a file system is made using the memory of each server node in the transmission cluster, a memory virtualization logical cache pool is constructed, a unified data access interface is provided to software services such as synchronization data pre-processing and data transmission, fast and efficient access of remote sensing data is realized, data read-write efficiency is improved, and remote sensing data pre-processing and transmission are efficient and flexible. The use mode is as shown in Figure 13 .

[0186] According to the embodiment of the present application, the link monitoring and optimization service is designed, which can track, monitor and maintain the link state between the source node and the destination node of the synchronization and transmission task, evaluate each link through indicators such as connectivity, bandwidth, load, packet loss rate and delay, optimize the most matched transmission link in combination with strategies such as the priority, urgency and timeliness requirement of the task, and realize data synchronization and transmission across domains and network segments through a transit node.

[0187] According to the embodiments of this application, unlike general methods that mainly support traditional transmission protocols such as TCP and UDP, this application supports multiple transmission protocols. In addition to TCP and UDP, it also supports PDXP and FEP protocols commonly used in the remote sensing field, as well as KCP and QUIC protocols for complex network environments.

[0188] According to the embodiments of this application, unlike traditional methods that uniformly adopt a single compression method such as RAR, this application designs multiple compression methods for multimodal remote sensing data, enabling the selection of appropriate compression methods for different types of data. For binary files such as raw bitstreams and formatted data, as well as small files such as text, XML, and DOC documents, software modules such as RAR and ZIP are used for packaging and compression. For large data files such as high-resolution, wide-swath remote sensing images and videos, algorithms such as JPEG-LS, JPEG 2000, and WebP can be selected for lossless or minimally lossy compression, greatly improving the compression effect.

[0189] According to the embodiments of this application, for data block transmission, in addition to supporting the general strategy of dividing data blocks into fixed byte sizes, this application also designs targeted segmentation methods for different types of remote sensing data, facilitating simultaneous transmission and use. For remote sensing data files with fixed frame formats, such as raw bitstreams and formatted data, data blocks are divided according to a fixed number of data frames, ensuring that each data block contains a complete data frame, allowing each data block to be processed directly, facilitating simultaneous transmission and processing. For wide-format, large-area remote sensing raster images, image blocks are supported using grid-like, nine-square, or other grid-based segmentation methods, and then parallel transmission is used to achieve parallel transmission and loading display of the image grid, fully realizing simultaneous transmission and use.

[0190] According to the embodiments of this application, based on the support of the cloud platform, it is possible to realize customized fine-grained synchronization of remote sensing data in relational, in-memory, and spatiotemporal databases. It supports one-click completion of database instance creation, database structure synchronization and initialization, and full and incremental data synchronization. It can automatically complete the synchronization of application-oriented remote sensing business data according to the association and dependency relationships of remote sensing data.

[0191] Any of the modules, sub-modules, units, sub-units according to the embodiments of the present application, or at least part of functions of any of them, can be implemented in one module. Any of the modules, sub-modules, units, sub-units according to the embodiments of the present application can be split into multiple modules for implementation. Any of the modules, sub-modules, units, sub-units according to the embodiments of the present application can be at least partially implemented as a hardware circuit, for example, a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on board, a system on package, an application specific integrated circuit (ASIC), or any other reasonable way of hardware or firmware through integration or packaging of circuits, or in any one of software, hardware and firmware, or in a proper combination of any of them. Alternatively, one or more of the modules, sub-modules, units, sub-units according to the embodiments of the present application can be at least partially implemented as computer program modules, which can perform corresponding functions when running.

[0192] For example, any of the remote sensing data source configuration and synchronization customization module, data synchronization task management and scheduling module, to-be-synchronized data preprocessing module, database synchronization and transmission service module, data file synchronization and transmission service module and synchronization data cache management module can be combined in one module for implementation, or any of them can be split into multiple modules. Alternatively, at least part of the functions of one or more of the modules can be combined with at least part of the functions of other modules and implemented in one module. According to the embodiments of the present application, at least one of the remote sensing data source configuration and synchronization customization module, data synchronization task management and scheduling module, to-be-synchronized data preprocessing module, database synchronization and transmission service module, data file synchronization and transmission service module and synchronization data cache management module can be at least partially implemented as a hardware circuit, for example, a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on board, a system on package, an application specific integrated circuit (ASIC), or any other reasonable way of hardware or firmware through integration or packaging of circuits, or in any one of software, hardware and firmware, or in a proper combination of any of them. Alternatively, at least one of the remote sensing data source configuration and synchronization customization module, data synchronization task management and scheduling module, to-be-synchronized data preprocessing module, database synchronization and transmission service module, data file synchronization and transmission service module and synchronization data cache management module can be at least partially implemented as computer program modules, which can perform corresponding functions when running.

[0193] According to the embodiments of the present application, the present application also provides a readable storage medium and a computer program product.

[0194] According to an embodiment of the present application, a non-transitory computer-readable storage medium storing computer instructions is provided, where the computer instructions are used to cause a computer to perform the method as above.

[0195] According to an embodiment of the present application, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the method as above.

[0196] It should be noted that the display control device part in the embodiments of the present application corresponds to the display control method part in the embodiments of the present application, and the description of the display control system part is specifically referred to the display control method part, which will not be repeated here.

[0197] Figure 14 A block diagram of an electronic device suitable for implementing the method described above according to an embodiment of the present application is schematically shown. Figure 14 The electronic device shown is merely an example and should not bring any limitation to the function and use range of the embodiments of the present application.

[0198] As shown, the electronic device according to an embodiment of the present application includes a processor 601 which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 602 or loaded from a storage part 608 into a random access memory (RAM) 603. The processor 601 may, for example, include a general-purpose microprocessor (e.g., a CPU), an instruction set processor, and / or a related chipset, and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 can also include an on-board memory for cache use. The processor 601 can include a single processing unit or multiple processing units for performing different actions of the method processes according to the embodiments of the present application. Figure 14 In the RAM 603, various programs and data required for the operation of the electronic device 600 are stored. The processor 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. The processor 601 performs various operations of the method processes according to the embodiments of the present application by executing the programs in the ROM 602 and / or the RAM 603. It should be noted that the programs can also be stored in one or more memories other than the ROM 602 and the RAM 603. The processor 601 can also perform various operations of the method processes according to the embodiments of the present application by executing the programs stored in the one or more memories.

[0199]

[0200] ​According to an embodiment of the present application, the electronic device can further include an input / output (I / O) interface 605 that is also connected to the bus 604. The system 600 can further include one or more of the following components connected to the input / output (I / O) interface 605: an input part 606 including a keyboard, a mouse, etc.; an output part 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage part 608 including a hard disk, etc.; and a communication part 609 including a network interface card such as a LAN card, a modem, etc. The communication part 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the input / output (I / O) interface 605 as necessary. A removable medium 611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is mounted on the drive 610 as necessary, so that a computer program read therefrom is installed in the storage part 608 as necessary.

[0201] According to an embodiment of the present application, the method flow according to the embodiments of the present application can be implemented as a computer software program. For example, the embodiments of the present application include a computer program product comprising a computer program carrying out the method shown in the flowchart, which is embodied on a computer readable storage medium. In such an embodiment, the computer program can be downloaded and installed from a network by the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the processor 601, the above-described functions defined in the system implementing the embodiments of the present application are performed. According to an embodiment of the present application, the system, device, apparatus, module, unit, etc. described above can be implemented by computer program modules.

[0202] The present application also provides a computer readable storage medium, which can be included in the device / apparatus / system described in the above embodiments, or can exist separately without being assembled into the device / apparatus / system. The above computer readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present application.

[0203] According to an embodiment of the present application, the computer readable storage medium can be a non-transitory computer readable storage medium. For example, it can include, but is not limited to, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage, magnetic storage, or any suitable combination of the foregoing. In this application, a computer readable storage medium can be any tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device.

[0204] For example, according to an embodiment of the present application, the computer readable storage medium can include one or more memories of ROM 602 and / or RAM 603 described above and / or other than ROM 602 and RAM 603.

[0205] Embodiments of the present application also include a computer program product, which includes a computer program containing program codes for executing the methods provided by the embodiments of the present application, and when the computer program product is run on an electronic device, the program codes are used to make the electronic device implement the methods provided by the embodiments of the present application.

[0206] When the computer program is executed by the processor 601, the above-mentioned functions defined in the system / apparatus of the embodiments of the present application are executed. According to an embodiment of the present application, the system, apparatus, module, unit, etc. described above can be implemented by computer program modules.

[0207] In one embodiment, the computer program can rely on tangible storage media such as optical storage media, magnetic storage media, etc. In another embodiment, the computer program can also be transmitted, distributed, downloaded and installed in the form of signals on network media. The program codes contained in the computer program can be transmitted by any suitable network media, including but not limited to wireless, wired, etc., or any suitable combination of the foregoing.

[0208] According to an embodiment of the present application, program code for implementing the computer programs provided by the embodiments of the present application can be written in any combination of one or more programming languages, and specifically, can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. The programming language includes, but is not limited to, such as Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on a user computing device, partially on a user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case involving a remote computing device, the remote computing device can be connected to the user computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, connected to the Internet through an Internet service provider).

[0209] The flow diagrams and block diagrams in the drawings are illustrations of possible architectures, functions, and operations for systems, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which includes one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or in the reverse order, depending on the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or operations, or combinations of special purpose hardware and computer instructions.

[0210] The embodiments of the present application are described above. However, these embodiments are only for illustrative purposes, and are not intended to limit the scope of the present application. Although each embodiment is described above separately, this does not mean that the measures in each embodiment cannot be used advantageously in combination. The scope of the present application is defined by the appended claims and their equivalents. Without departing from the scope of the present application, those skilled in the art can make various substitutions and modifications, which should all fall within the scope of the present application.

Claims

1. A distributed multi-cloud node remote sensing data synchronization transmission method, characterized in that, The method comprises: receiving a data synchronization request of a user, the data synchronization request taking a data set defined by business logic as a synchronization unit, the logic of the data set being defined by at least one business dimension information; in response to the data synchronization request, extracting all data entities corresponding to the data set logic from a distributed data source according to the business dimension information; decomposing a synchronization task for the all data entities into subtasks corresponding to different data types; after performing preprocessing on the data to be transmitted in the subtasks based on a transmission strategy, transmitting through a dynamically selected network link, so that a destination cloud node performs post-processing operations corresponding to the preprocessing on the received data to restore the data set.

2. The method of claim 1, wherein, The data set defined by the business logic corresponds to a business logic organization unit in the field of remote sensing data processing, and the business dimension information includes at least one of satellite model, load type, shooting time, and spatial position. The business logic organization unit includes whole-track observation data, data segments, data scenes, or standard products.

3. The method of claim 1, wherein, The response to the data synchronization request includes: receiving a synchronization requirement description input by a user in natural language form; performing semantic analysis on the synchronization requirement description using a remote sensing large language model to identify and extract business dimension information defining the data set logic; generating a structured query statement according to the business dimension information; based on the structured query statement, extracting structured data related to the data set logic from a registered relational database, in-memory database, or spatio-temporal database, the structured data including metadata, attribute data, and feature parameter data; extracting unstructured entity data files stored in an in-memory cache, high-speed file storage system, or distributed object storage according to the entity data association relationship recorded in the metadata; logically associating the structured data and unstructured entity data files to collectively constitute all data entities of the data set defined by the business logic.

4. The method of claim 1, wherein, The decomposition of the synchronization task for the all data entities into subtasks corresponding to different data types includes: identifying the form of data in the all data entities; decomposing the synchronization task into at least two subtasks of different properties, including a database synchronization subtask and a data file synchronization subtask; for the data file synchronization subtask, dividing and creating parallel transmission subtasks based on the logical organization structure of the unstructured entity data files, taking a data scene as a basic unit.

5. The method of claim 1, wherein, The preprocessing operation includes a compression operation, and the preprocessing based on the transmission strategy includes: obtaining file attributes and business features of the data to be transmitted, the file attributes including file extensions and internal format signatures, and the business features being defined by the business dimension information included in the data synchronization request; The file attribute and the business feature are matched with a predefined compression strategy selection table, which records a mapping relationship between file types and compression algorithms and compression parameters; According to the matching result, a corresponding compression algorithm and compression parameters are automatically selected and called to perform compression operation on the to-be-transmitted data.

6. The method of claim 1, wherein, The preprocessing operation includes a block operation, and the preprocessing of the to-be-transmitted data in the subtask based on the transmission strategy includes: identifying the file type of the to-be-transmitted data and the network environment state; According to the file type of the to-be-transmitted data, a corresponding block method is selected from a plurality of block strategies; According to the network environment state, the size of the data block of the to-be-transmitted data is dynamically adjusted; The block strategy at least includes at least one of the following: For file types with fixed frame format, block by a fixed number of complete data frames; For raster image file types, block by gridding method; For video file types, block by fixed time length; For file types with unclear format, block by fixed byte size.

7. The method of claim 1, wherein, The preprocessing operation includes a memory cache operation, and the preprocessing of the to-be-transmitted data in the subtask based on the transmission strategy includes: A memory virtualization logical cache pool is constructed by using the memory of the server node; The to-be-transmitted data is loaded into the memory virtualization logical cache pool for caching to perform read-write access.

8. The method of claim 1, wherein, The destination cloud node performs post-processing operation corresponding to the preprocessing on the received data, and restores the data set, which includes: Identify the type of preprocessing operation experienced by the received data; According to the preprocessing operation type, a corresponding post-processing service is called to perform inverse operation; Perform integrity check and correctness check on the data after the inverse operation; The structured data and unstructured entity data files that pass the check are stored according to the corresponding business logic association relationship, generating the data set consistent with the sending end according to the business logic definition.

9. A distributed multi-cloud node remote sensing data synchronization transmission system, characterized in that, The system includes: Remote sensing data source configuration and synchronization customization module, for receiving user's data synchronization request, and extracting all data entities corresponding to the data set logic from distributed data source according to the business dimension information contained in the data synchronization request, the data synchronization request taking the data set defined by business logic as the synchronization unit, and the logic of the data set being defined by at least one business dimension information; Data synchronization task management and scheduling module connected to the remote sensing data source configuration and synchronization customization module, for receiving the data entities and decomposing the synchronization tasks for the data entities into subtasks corresponding to different data types and scheduling the subtasks; To-be-synchronized data preprocessing module connected to the data synchronization task management and scheduling module, for performing preprocessing operation on the to-be-transmitted data in the scheduled subtask based on the transmission strategy; Database synchronization and transmission service module connected to the data synchronization task management and scheduling module, for executing database synchronization subtask; The data file synchronization and transmission service module is connected to the data synchronization task management and scheduling module, and is configured to execute a data file synchronization subtask and transmit data through a dynamically selected network link. The synchronization data cache management module is connected to the to-be-synchronized data preprocessing module, the database synchronization and transmission service module, and the data file synchronization and transmission service module, respectively, and is configured to construct and manage a memory virtualization logical cache pool at a sending end and a target cloud node, and provide high-speed read-write access support for preprocessing, transmission, and post-processing of the to-be-transmitted data. 10.An electronic device, comprising: one or more processors; memory storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1-8.

Images