Distributed multi-cloud node remote sensing data synchronization transmission method, system and device

Abstract

The application provides a distributed multi-cloud node remote sensing data synchronization transmission method, system and device, the method comprising: receiving a data synchronization request of a user, the data synchronization request taking a data set defined by business logic as a synchronization unit, the logic of the data set being defined by at least one business dimension information; in response to the data synchronization request, extracting all data entities corresponding to the data set logic from a distributed data source according to the business dimension information; decomposing the synchronization task for all data entities into subtasks corresponding to different data types; based on a transmission strategy, performing preprocessing on the to-be-transmitted data in the subtasks, and then transmitting through a dynamically selected network link, so that the destination cloud node performs post-processing operations corresponding to the preprocessing on the received data to restore the data set.

Description

Technical Field

[0001] This application relates to the field of remote sensing data processing, and specifically to a method, system, and device for synchronous transmission of distributed multi-cloud node remote sensing data. Background Technology

[0002] In recent years, my country's aerospace remote sensing satellite field has developed rapidly. A series of satellites, such as Gaofen, Haiyang, Tianhui, and Yunhai, have been launched and are in orbit, and have been widely used in professional fields such as land observation, marine monitoring, geographic mapping, meteorology and hydrology.

[0003] Satellite remote sensing data in the field of remote sensing satellites is a typical multimodal data. Remote sensing data transmission needs to cover various types of multimodal data. At present, the synchronization and transmission of various types of data are mainly carried out by using their respective general transmission tools. However, these general tools have scattered functions, are not very targeted, have low transmission efficiency, lack unified management, are difficult to control, and have low data utilization efficiency. Summary of the Invention

[0004] In view of the above problems, this application provides a method, system and device for distributed multi-cloud node remote sensing data synchronization and transmission, which solves at least one of the above problems.

[0005] According to a first aspect of this application, a method for synchronous transmission of distributed multi-cloud node remote sensing data is provided, the method comprising:

[0006] Receive user data synchronization requests, wherein the data synchronization requests are based on a data set defined by business logic as the synchronization unit, and the logic of the data set is defined by at least one business dimension information;

[0007] In response to the data synchronization request, all data entities logically corresponding to the data set are extracted from the distributed data source based on the business dimension information.

[0008] The synchronization task for all data entities is decomposed into sub-tasks corresponding to different data types;

[0009] Based on the transmission strategy, the data to be transmitted in the subtask is preprocessed and then transmitted through a dynamically selected network link, so that the destination cloud node can perform post-processing operations corresponding to the preprocessing on the received data to restore the data set.

[0010] According to an embodiment of this application, the data set defined by the business logic corresponds to a business logic organization unit in the field of remote sensing data processing, and the business dimension information includes at least one of satellite model, payload type, shooting time, and spatial location.

[0011] The business logic organization unit includes whole-track observation data, data segments, data scenes, or standard products.

[0012] According to an embodiment of this application, the step of extracting all data entities logically corresponding to the data set from the distributed data source in response to the data synchronization request, based on the business dimension information, includes:

[0013] Receive the user's synchronization request description in natural language.

[0014] The remote sensing big language model is used to perform semantic parsing on the synchronization requirement description, and to identify and extract the business dimension information that defines the logic of the data set;

[0015] Based on the aforementioned business dimension information, generate a structured query statement;

[0016] Based on the structured query statement, structured data logically related to the data set is extracted from the registered relational database, in-memory database, or spatiotemporal database. The structured data includes metadata, attribute data, and feature parameter data.

[0017] Based on the entity data relationships recorded in the metadata, extract unstructured entity data files stored in memory cache, high-speed file storage system or distributed object storage;

[0018] The structured data and unstructured entity data files are logically associated to form all the data entities of the data set defined by the business logic.

[0019] According to an embodiment of this application, the step of decomposing the synchronization task for all data entities into sub-tasks corresponding to different data types includes:

[0020] Identify the format of data in all the data entities;

[0021] The synchronization task is decomposed into at least two types of subtasks of different natures, including a database synchronization subtask and a data file synchronization subtask:

[0022] For the data file synchronization subtask, based on the logical organization structure of the unstructured entity data file, subtasks for parallel transmission are divided and created using data scenes as the basic unit.

[0023] According to an embodiment of this application, the database synchronization subtask is used to synchronize structured data stored in a relational database, an in-memory database, or a spatiotemporal database;

[0024] The data file synchronization subtask is used to synchronize unstructured entity data files stored in memory cache, high-speed file storage system or distributed object storage.

[0025] According to an embodiment of this application, the preprocessing operation includes a compression operation, and the preprocessing of the data to be transmitted in the subtask based on the transmission strategy includes:

[0026] Obtain the file attributes and business characteristics of the data to be transmitted. The file attributes include the file extension and internal format signature, and the business characteristics are defined by the business dimension information contained in the data synchronization request.

[0027] The file attributes and business characteristics are matched with a predefined compression strategy selection table, which records the mapping relationship between file type, compression algorithm and compression parameters;

[0028] Based on the matching results, the corresponding compression algorithm and compression parameters are automatically selected and invoked to perform compression operations on the data to be transmitted.

[0029] According to an embodiment of this application, the preprocessing operation includes a block operation, and the preprocessing of the data to be transmitted in the subtask based on the transmission strategy includes:

[0030] Identify the file type and network environment status of the data to be transmitted;

[0031] Based on the file type of the data to be transmitted, select the corresponding chunking method from multiple chunking strategies;

[0032] The size of the data blocks into which the data to be transmitted is divided is dynamically adjusted according to the network environment status.

[0033] The block partitioning strategy includes at least one of the following:

[0034] For file types with a fixed frame format, the data is divided into blocks based on a fixed number of complete data frames.

[0035] For raster image file types, divide them into blocks according to the gridding method;

[0036] For video file types, divide them into blocks of fixed duration;

[0037] For file types with ambiguous formats, divide them into blocks of fixed byte size.

[0038] According to an embodiment of this application, the preprocessing operation includes a memory caching operation, and the preprocessing of the data to be transmitted in the subtask based on the transmission strategy includes:

[0039] Utilize the memory of server nodes to build a memory virtualization logical cache pool;

[0040] The data to be transmitted is loaded into the memory virtualization logical cache pool for caching, so that it can be accessed for reading and writing.

[0041] According to an embodiment of this application, the step of preprocessing the data to be transmitted in the subtask based on a transmission strategy and then transmitting it through a dynamically selected network link includes:

[0042] Based on the transmission strategy configured independently for each subtask, the corresponding preprocessing operation is performed on the data to be transmitted at the sending end;

[0043] Based on the attributes of the subtask and the real-time status of multiple network links, a target link is dynamically selected from the multiple network links;

[0044] The preprocessed data to be transmitted is transmitted through the target link.

[0045] According to an embodiment of this application, the destination cloud node performs a post-processing operation corresponding to the preprocessing on the received data to restore the data set, including:

[0046] Identify the type of preprocessing operation that the received data has undergone;

[0047] Based on the preprocessing operation type, the corresponding postprocessing service is invoked to perform the reverse operation;

[0048] For the data that has undergone the inverse operation, perform integrity and correctness checks.

[0049] The validated structured data and unstructured entity data files are stored according to their corresponding business logic relationships to generate a data set consistent with the business logic definition of the sending end.

[0050] According to an embodiment of this application, the transmission via dynamically selected network links includes:

[0051] A protocol abstraction interface layer is provided, which provides a unified data transmission interface at the top and encapsulates the underlying implementations of various transmission protocols at the bottom.

[0052] During transmission initialization, a target transmission protocol is dynamically selected from the various transmission protocols based on real-time network status, transmission task characteristics, and the support capabilities of the destination cloud node.

[0053] During transmission, transmission performance is monitored in real time, and the parameters of the target transmission protocol are dynamically adjusted for optimization.

[0054] When the target transmission protocol connection fails, a protocol fallback mechanism is activated to automatically switch to an alternative transmission protocol for data transmission.

[0055] According to a second aspect of this application, a distributed multi-cloud node remote sensing data synchronization and transmission system is provided, the system comprising:

[0056] The remote sensing data source configuration and synchronization customization module is used to receive data synchronization requests from users and extract all data entities corresponding to the data set logic from the distributed data source according to the business dimension information contained in the data synchronization request. The data synchronization request takes the data set defined by the business logic as the synchronization unit, and the logic of the data set is defined by at least one business dimension information.

[0057] The data synchronization task management and scheduling module is connected to the remote sensing data source configuration and synchronization customization module. It is used to receive the data entity, decompose the synchronization task for the data entity into sub-tasks corresponding to different data types, and schedule the sub-tasks.

[0058] The data preprocessing module to be synchronized is connected to the data synchronization task management and scheduling module, and is used to perform preprocessing operations on the data to be transmitted in the scheduled subtasks based on the transmission strategy.

[0059] The database synchronization and transmission service module is connected to the data synchronization task management and scheduling module and is used to execute database synchronization subtasks;

[0060] The data file synchronization and transmission service module is connected to the data synchronization task management and scheduling module, and is used to execute the data file synchronization sub-task and transmit it through a dynamically selected network link;

[0061] The synchronous data cache management module is connected to the data to be synchronized preprocessing module, the database synchronization and transmission service module, and the data file synchronization and transmission service module, respectively. It is used to build and manage the memory virtualization logical cache pool at the sending end and the destination cloud node, and provide high-speed read and write access support for the preprocessing, transmission and postprocessing of the data to be transmitted.

[0062] According to a third aspect of this application, an electronic device is provided, comprising:

[0063] One or more processors;

[0064] Memory, used to store one or more computer programs.

[0065] The one or more processors execute the one or more computer programs to implement the steps of the method described in the first aspect.

[0066] According to a fourth aspect of this application, a computer-readable storage medium is provided having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the method according to the first aspect.

[0067] According to the embodiments of this application, at least one of the following technical effects can be achieved:

[0068] (1) The method proposed in this application, which uses the business logic organization unit of multimodal remote sensing data as the basic unit for synchronization and transmission, enables business users to truly propose synchronization and transmission requirements from the perspective of remote sensing data processing business, without having to specify specific data files, directories, or database records one by one as in traditional methods. At the same time, based on the remote sensing dataset to be synchronized, multimodal data synchronization subtasks are automatically created, realizing unified management, status monitoring, and centralized access to multimodal data synchronization and transmission.

[0069] (2) Through the memory caching service, the single-point read and write bandwidth of the remote sensing data in this application can reach 8GB / s, which can significantly improve the speed of remote sensing data encryption and decryption, compression and decompression, block merging, network transmission and subsequent processing, loading and display, and realize efficient and flexible synchronization and transmission.

[0070] (3) The link monitoring and optimization service proposed in this application can select the most suitable transmission link according to the needs of the synchronization task, so as to realize cross-domain and cross-network segment data transmission.

[0071] (4) The various compression methods designed for multimodal remote sensing data in this application, compared with a single general compression method, can adopt targeted compression algorithms for different data, significantly improving compression efficiency and quality, and enhancing the efficiency of compressed transmission.

[0072] (5) The various data block methods designed for multimodal remote sensing data in this application, compared with the single fixed byte size block method, can ensure that each data block can be transmitted, processed and utilized independently without waiting for other data blocks to form a complete data frame. It supports reading partially transmitted files, viewing partially transmitted images, and viewing partially transmitted formatted files, thereby effectively supporting simultaneous transmission and use.

[0073] (6) The multiple transmission protocols adapted to this application are better suited to remote sensing data processing scenarios, can adapt more flexibly to various network environments, and have better adaptability.

[0074] (7) This application can make requests for remote sensing data synchronization and transmission in a natural language interactive question-and-answer manner, without having to provide technical details such as data file paths or database links as in traditional methods. This greatly reduces the difficulty of use and improves ease of use. For example, in this system, users can make synchronization requests in a similar way: "Please synchronize all remote sensing images of today's activities to cloud center A for the production of special products." Attached Figure Description

[0075] The above-mentioned contents, other objects, features and advantages of this application will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0076] Figure 1 This illustration schematically shows an architecture diagram of a distributed multi-cloud node remote sensing data synchronization and transmission system according to an embodiment of this application;

[0077] Figure 2 A schematic diagram of a distributed multi-cloud node remote sensing data synchronization and transmission system according to an embodiment of this application is shown.

[0078] Figure 3 The diagram illustrates the architecture of the data source configuration and synchronization customization module according to an embodiment of this application.

[0079] Figure 4 The diagram illustrates the architecture of the data synchronization task management and scheduling module according to an embodiment of this application.

[0080] Figure 5 The diagram illustrates the architecture of the data source configuration and synchronization customization module according to an embodiment of this application.

[0081] Figure 6 The diagram illustrates the architecture of a database synchronization and transmission service module according to an embodiment of this application.

[0082] Figure 7 The diagram illustrates the architecture of a data file synchronization and transmission service module according to an embodiment of this application.

[0083] Figure 8 This illustration schematically shows a network link diagram between distributed cloud nodes according to an embodiment of this application;

[0084] Figure 9 The diagram illustrates the architecture of a synchronous data cache management module according to an embodiment of this application.

[0085] Figure 10 This diagram schematically illustrates a transport cluster memory cache management architecture according to an embodiment of this application.

[0086] Figure 11A flowchart illustrating a distributed multi-cloud node remote sensing data synchronization and transmission method according to an embodiment of this application is shown in the schematic diagram.

[0087] Figure 12 A flowchart illustrating another distributed multi-cloud node remote sensing data synchronization and transmission method according to an embodiment of this application is shown schematically.

[0088] Figure 13 This illustration schematically depicts a usage pattern diagram of efficient remote sensing data processing and transmission based on memory caching according to an embodiment of this application;

[0089] Figure 14 A block diagram of an electronic device suitable for implementing a distributed multi-cloud node remote sensing data synchronization transmission method according to an embodiment of this application is illustrated. Detailed Implementation

[0090] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0091] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0092] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0093] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0094] Figure 1 The diagram illustrates the architecture of a distributed multi-cloud node remote sensing data synchronization transmission system according to an embodiment of this application.

[0095] like Figure 1 As shown, the remote sensing data high-efficiency synchronization and transmission system architecture of this application includes a cloud infrastructure layer, a data storage cloud service layer, a synchronization and transmission service layer, and a business user interaction layer. Among them, the cloud infrastructure layer mainly provides computing server clusters for management, control, and synchronization transmission for upper-layer services, as well as storage, network, security and other equipment resources, and realizes the virtualization, pooling and cloud management of resources through cloud platform services.

[0096] The data storage cloud service layer mainly provides storage middleware services for multimodal remote sensing data based on the cloud platform, including: memory cache management services, high-speed file storage systems based on high-speed storage devices, distributed file storage systems based on large-capacity storage devices and OSS object storage services, as well as database services such as relational databases MySQL and PolarDB, in-memory database Redis, and spatiotemporal database PostgreSQL.

[0097] The synchronization and transmission service layer is the core component of this application, providing core basic services such as data encryption / decryption, data compression / decompression, link monitoring and optimization, database synchronization and transmission, and data file synchronization and transmission, as well as management and control services such as data source management, synchronization task decomposition and scheduling, synchronization task monitoring and control, task load balancing, cache management and data verification.

[0098] The business user interaction layer mainly provides a front-end interactive interface, offering interactive services such as synchronous dataset selection, synchronous strategy configuration and task customization, synchronous task monitoring and management, and synchronous data cache management to users of the remote sensing data processing system. It also calls various services of the synchronization and transmission service layer to achieve efficient synchronization and transmission of remote sensing data.

[0099] Figure 2 A schematic diagram of a distributed multi-cloud node remote sensing data synchronization transmission system according to an embodiment of this application is shown.

[0100] like Figure 2 As shown, the remote sensing data synchronization and transmission system of this application mainly completes the synchronization and transmission of multimodal remote sensing data between distributed multi-cloud nodes of the aerospace remote sensing ground system, including modules such as remote sensing data source configuration and synchronization customization, data synchronization task management and scheduling, data preprocessing to be synchronized, database synchronization and transmission service, data file synchronization and transmission service, and synchronization data cache management.

[0101] like Figure 3 As shown, the data source configuration and synchronization customization module is mainly responsible for the registration, configuration and management of various remote sensing data sources. It supports users to obtain access according to their permissions, select the datasets that need to be synchronized and transmitted, and customize and initiate synchronization tasks.

[0102] The database instance registration and configuration function is responsible for registering and configuring database instances (relational databases, in-memory databases, and spatiotemporal databases) that store metadata, attribute data, and feature data of remote sensing data in various cloud nodes into the synchronization and transmission software. It also manages user access permissions for various database instances. One-click import of database instance information is supported via the cloud platform's interface.

[0103] The data storage configuration and mounting function is responsible for configuring and mounting various storage system data sources, such as memory caches, high-speed file storage, distributed file storage, and OSS object storage, that store remote sensing data entity files in each cloud node to the synchronization and transmission software. It also manages user access permissions to directories and files in various storage systems. It supports one-click import of information from various storage systems via the cloud platform's interface.

[0104] The remote sensing data query and selection function allows authorized users to query and select remote sensing data to be synchronized based on multi-dimensional information such as satellite model, payload type, imaging time, spatial location, data type, data theme, and data tags, forming a remote sensing dataset to be synchronized. This enables synchronization and transmission tasks to be initiated on a dataset-by-dataset basis. By calling the interface of the remote sensing large language model, users can submit remote sensing data synchronization requests in a manner similar to natural language interactive question-and-answer, achieving intelligent data synchronization.

[0105] Synchronization task strategy configuration and customization: Based on the synchronization and transmission requirements of the remote sensing dataset, configure synchronization and transmission strategies and customize the initiation of synchronization tasks. Synchronization tasks support selecting one or more destination cloud nodes; support configuring task priority, urgency, timeliness requirements, whether to encrypt, whether to compress, transmission protocol, and other strategies.

[0106] like Figure 4 As shown, the data synchronization task management and scheduling module is mainly responsible for analyzing multimodal remote sensing datasets, decomposing them into multiple synchronization sub-tasks according to data type, and then scheduling and executing each sub-task according to a load balancing strategy.

[0107] Synchronization Task Analysis and Decomposition: Since the remote sensing datasets to be transmitted in synchronization tasks typically include multimodal data and may contain a large number of data points, analysis is required from two dimensions: data type and data composition. First, according to data type, the data in the database and file storage are organized, and subsets are constructed for relational, in-memory, and spatiotemporal data respectively. For the data in file storage, according to the logical organization structure of remote sensing data, the smallest logical unit of remote sensing data—the data scene—is used as the basic unit to divide the dataset into subsets.

[0108] Subtask creation and configuration: Create data synchronization and transmission subtasks based on the data subsets analyzed and decomposed from the synchronization task, configure the subtask strategy according to the synchronization task strategy and the subtask data characteristics, and submit the task for scheduling and execution.

[0109] Parallel task scheduling: Based on the task queue and the resource status of the transmission nodes, subtasks are scheduled to the transmission node cluster according to priority, short task priority and other strategies. Then, execution nodes are allocated in the transmission node cluster according to the load balancing strategy to achieve parallel execution of multiple tasks and multiple nodes.

[0110] Task monitoring and management: Real-time acquisition of task execution status and transmission progress, supporting tracking and monitoring through a visual interface; and allowing users to manually adjust task priorities, as well as perform operations such as pausing, canceling, retrying, and rate limiting.

[0111] like Figure 5 As shown, the preprocessing module for data to be synchronized is mainly responsible for performing data caching, encryption, compression, segmentation, and verification feature calculation on the sending end as needed before synchronization and transmission. After transmission is completed, the module performs data decryption, decompression, segmentation and merging, integrity and correctness verification on the destination cloud node as needed.

[0112] Data cache loading: For data files stored in high-speed file storage or distributed file storage, in order to speed up preprocessing and transmission and improve timeliness, the data files are pre-read and cached in memory, and the storage status and popularity of the data are updated for easy reuse.

[0113] Data encryption / decryption: Based on the task strategy, data is encrypted before transmission to generate encrypted data for transmission; after transmission, the encrypted data is decrypted at the destination cloud node to generate plain data. Supports mainstream symmetric encryption algorithms such as DES and AES.

[0114] Data Compression and Decompression: Based on the task strategy, appropriate compression algorithms are selected for multimodal remote sensing data. For raw bitstreams, formatted data, and other binary files, as well as small files such as text, XML, and DOC documents, software modules such as RAR and ZIP are used for packaging and compression to improve transmission efficiency. For large data files such as high-resolution, wide-swath remote sensing images and videos, lossless or minimal-loss compression can be selected for overall compression. The module supports multiple image compression algorithms to ensure optimal compression results in different scenarios, including JPEG-LS, JPEG 2000, and WebP; it supports custom parameters for minimal-loss compression, such as compression quality level and compression ratio. Automatic selection of compression algorithms is supported, establishing a default compression strategy selection table that automatically selects the compression algorithm based on file extension, internal format signature, and business characteristics. A fragment of the compression strategy selection table is shown in Table 1 below:

[0115] surface

[0116] file extension Data types Compression tools and algorithms Expected compression ratio *.dat;*.FRED Raw bitstream, formatted data, and other binary data RAR (multi-part compression, LZ algorithm) ~6:1 or 8:1 *.tif;*.tiff; *.img;*.hdf5 Remote sensing images / thematic maps JPEG2000, WebP, etc. (lossless or minimally lossy) ~3:1 or 4:1 *.shp; *.geojson Vector data / high-precision planning GZIP (Deflate algorithm), Geobuf ~4:1 or 6:1 *.xml;*.json; *.txt Metadata / Logs / Auxiliary Parameters BZIP2 (Burrows-Wheeler Transform) ~5:1

[0117] Data Segmentation and Merging: For extremely large single data files, the system supports segmenting into multiple data blocks for multi-threaded parallel transmission. After transmission, the blocks are merged sequentially at the destination cloud node. A large file is divided into numerous data blocks, each assigned a file block number. This block number allows for accurate writing to the temporary cache file at the destination cloud node. Once the last data block arrives at the destination cloud node and is successfully written to the cache file, the temporary filename is changed, and an acknowledgment message is sent back to the sender, notifying them that all data blocks have been received. During transmission, each data block can undergo independent encryption, compression, and verification. In addition to supporting the general strategy of segmenting data blocks by fixed byte size, this application segments data blocks according to the data frame format of the remote sensing data file to be segmented, ensuring that the complete data frame is not destroyed. This guarantees that each data block contains a complete data frame, facilitating processing while transmitting. For wide-format, large-area raster images, the system supports image segmentation using a "nine-grid" or gridded method, followed by parallel transmission to achieve image loading and display while transmitting. The size of the data chunks is adjusted based on network conditions. In high-bandwidth, low-latency network environments (such as within a cloud node), larger chunks (e.g., 8MB or 16MB) can be used to reduce protocol overhead. In bandwidth-constrained or unstable network environments (e.g., across cloud nodes or regions), smaller chunks (e.g., 256KB or 512KB) are used to improve transmission resilience and preemption capabilities. The chunking strategies for various data types are shown in Table 2 below.

[0118] surface

[0119] Data types file extension Data Description Blocking method advantage raw bitstream *.dat AOS frame format data with a fixed frame length of 1024 bytes Divide the AOS frame into blocks of 1024 bytes each. Each piece of data can be processed independently in parallel. Formatted data *.FRED Decrypting formatted data with a fixed frame length after decompression Divide the data into blocks based on the formatted data frames of each satellite payload. Each data block can be transmitted and processed in parallel according to data frames. Remote sensing images *.tif;*.tiff; *.img;*.hdf5 Raster images such as remote sensing images and thematic maps Data can be segmented based on row and column numbers or latitude and longitude information, using methods such as "grid" or "nine-square grid"; it can also be segmented according to grid division standards. Each piece of data can be processed and loaded for display independently in parallel. Remote sensing video *.mp4;*.avi Remote sensing video stream Video data is segmented based on a fixed duration. Transmit and broadcast simultaneously Other large files Other extensions Large data files with unclear format Separate according to fixed byte size Parallel transmission

[0120] Data verification calculation: To ensure the integrity and correctness of data synchronization and transmission, checksums are calculated for the data to be transmitted using message digest algorithms such as MD5 and SHA1, and CRC cyclic redundancy check algorithms. The checksum is calculated before transmission and transmitted with the data to the destination cloud node. The destination cloud node then calculates the checksum and compares it to complete the data verification. In addition to message digest-based verification, this application also supports sampling verification according to the remote sensing data frame format. Based on the synchronization word and feature code in the data frame format, feature information at specific locations is sampled and generated before transmission. After transmission, the feature information at the same locations is checked to confirm consistency.

[0121] like Figure 6As shown, the database synchronization and transmission service module is primarily responsible for performing synchronization and transmission tasks for relational, in-memory, and spatiotemporal databases, completing the synchronization and transmission of metadata and attribute data of the remote sensing dataset to be synchronized. Utilizing the relevant API interfaces of the cloud platform, it can automatically create database instances for the target node, create the database structure, and initialize data; it can automatically query, customize, and package data records to be synchronized based on the association and dependencies of remote sensing data, and perform integrity and correctness verification on the synchronized and transmitted data records to ensure data authenticity and usability.

[0122] Database instance creation: Responsible for calling the API interfaces of relational, in-memory, and time-space database products on the cloud platform at the synchronization destination to create database instances of the required specifications, such as MySQL, PolarDB, Redis, and PostgreSQL, and configuring parameters such as login account, security group, and access whitelist to ensure that the database instance at the destination is available before synchronization.

[0123] Database structure synchronization and initialization: After the database instance is created, the creation scripts for database table structures, views, custom data types, functions, stored procedures, triggers, and other database objects, as well as the scripts for initializing data, are generated on the source side. The dependencies and execution order of the scripts for each database object are arranged. Through the copying of DDL statements, the database connection tool is invoked to execute the scripts on the destination database instance, completing the database structure synchronization and initialization.

[0124] Synchronization data customization and packaging: Based on the association and dependency relationships of remote sensing data, automatically query, customize and package the metadata and attribute data of the remote sensing dataset to be synchronized, generate a set of data records to be synchronized, support three-level object name mapping of library, table and column, support custom filtering of synchronized data and columns, and realize fine-grained control and customized precise synchronization of the data to be synchronized.

[0125] Full and incremental data synchronization: For synchronization at the database instance, database, or table level, the cloud platform's Data Transmission Service (DTS) interface is invoked to create a database synchronization link, enabling full and incremental data synchronization. Real-time data transmission is ensured by capturing incremental database logs. Data encryption and multi-concurrency compressed transmission are supported, as well as breakpoint transmission capabilities. Full verification of data on both the source and target ends during synchronization is supported to guarantee data consistency between the source and target ends during non-disruptive synchronization.

[0126] like Figure 7As shown, the data file synchronization and transmission service module is mainly responsible for performing the synchronization and transmission tasks of data entity files. Based on the storage location of remote sensing data, the synchronization task strategy, and the network conditions of source and destination nodes, it constructs an efficient transmission link, adopts a suitable network protocol, and completes the efficient transmission of data files through multi-path parallel transmission.

[0127] Link monitoring and optimization: Multiple distributed cloud nodes in a remote sensing data processing system are connected via wide area or regional private networks. The network bandwidth, load, and availability vary among these nodes; some nodes may even be unable to connect directly due to management or security reasons, requiring relay through a central cloud node. Figure 8 As shown.

[0128] Therefore, the link monitoring and optimization unit is responsible for tracking, monitoring, and maintaining the link status between the source and destination nodes of synchronization and transmission tasks. It comprehensively evaluates each link based on indicators such as connectivity, bandwidth, load, packet loss rate, and latency, and selects the most suitable transmission link based on task priority, urgency, and timeliness requirements. For cross-domain and cross-network segment transmission tasks without direct links, forwarding can be performed through the central cloud node to achieve cross-domain and cross-network segment data synchronization and transmission.

[0129] In some implementations, the connectivity, bandwidth, round-trip time (RTT), and packet loss rate of candidate links can be evaluated by periodically sending lightweight probe packets (such as ICMP Echo Requests or specific small TCP / UDP packets). Alternatively, this status information can be collected centrally via periodic heartbeat detection, broadcast messages, or an SDN controller. Then, the real-time status of each link is recorded in the system, maintaining a global cloud node link topology map that records all possible connections and their status attributes between ground system cloud nodes at a specific time. Finally, based on the link topology map and mission strategy, the best-matching transmission link is selected.

[0130] Transport protocol adaptation: Manages and adapts the transport protocols required for accessing the transport network. It can adapt to transport protocols such as TCP, UDP, FTP, PDXP, FEP, KCP, and QUIC, and can use appropriate network protocols for different data types and network conditions.

[0131] In some implementations, a protocol abstraction interface layer can be built first. This layer provides a unified data sending / receiving interface upwards and encapsulates the underlying implementation details of different protocols downwards. During system initialization, the implementation libraries of each protocol are loaded. Secondly, protocol selection can be dynamically determined based on real-time network conditions (such as latency, packet loss rate, and bandwidth), transmission task characteristics (such as size, priority, and real-time requirements), and the support capabilities of the target cloud nodes. Thirdly, protocol parameters can be dynamically adjusted and optimized during task execution, such as TCP window size, retransmission count, and UDP buffer size. Finally, to ensure robust communication, a protocol fallback mechanism is supported. For example, when a QUIC connection fails multiple times, automatic degradation to HTTP / 2 or HTTP / 1.1 over TCP can be achieved by using the ALPN extension for protocol negotiation during the TLS handshake.

[0132] Resume interrupted downloads: Supports resuming interrupted network data transmissions. When a software exception occurs, the current transmission task execution status is automatically recorded. After troubleshooting and software recovery, the transmission task is automatically resumed from the interrupted point. Combined with chunked large file transmission, resumed downloads provide greater assurance for large file transfers because they have the ability to resume transmission from the point of interruption, while chunked transmission facilitates breakpoint location and resumption. For example, if the connection is broken or the session fails when sending the Nth block of data, after the connection is restored and the session is retried, the sender can continue transmission from the last interrupted block without restarting.

[0133] Flow control: The bandwidth used by the transmission service can be configured as needed, allowing for rate limiting and recovery of transmission tasks, supporting the adjustment of transmission task priorities. Dynamic flow control is achieved by configuring the maximum number of connections, the maximum number of parallel tasks, and the number of parallel threads. By limiting the link bandwidth used by lower-priority tasks, more network bandwidth is allocated to higher-priority tasks. When high-bandwidth parallel transmission is required, the characteristics of data files generated by remote sensing data fragmentation, spectral segmentation, polarization, and camera segmentation are fully utilized to automatically and efficiently carry out parallel transmission. The number of parallel threads is consistent with the number of data files constituting the basic unit of remote sensing data synchronization. This maximizes parallel transmission and improves overall efficiency without splitting data files.

[0134] like Figure 9 As shown, the synchronization data cache management module mainly manages the multi-level data cache at both ends of the synchronization source node and the destination node, completes the lifecycle management of cached data, and ensures efficient execution of data transmission and processing and effective utilization of data.

[0135] Transmission Cluster Memory Cache Management: This unit primarily implements capabilities such as registration and capacity adjustment management of available memory on transmission cluster server nodes. It also supports memory disk information query, memory disk deregistration, data read / write, and deletion functions. The principle diagram of transmission cluster memory cache management is shown below. Figure 10 As shown, a file system is created using the memory of each server node in the transmission cluster, and a memory virtualization logical cache pool is constructed. A unified data access interface is provided to software services such as synchronous data preprocessing and data transmission, enabling fast and efficient access to remote sensing data, improving data read and write efficiency, and making remote sensing data preprocessing and transmission efficient and flexible.

[0136] Memory-cached data management: On the sending end, it loads and caches data to be synchronized from high-speed file storage, distributed file storage, etc.; on the destination cloud node, it is responsible for caching the received data in the memory cache disk and providing the ability to write the cached data back to high-speed file storage and distributed file storage, thus achieving memory-cached data persistence. Simultaneously, it can use a waterline to evict and clean up the remote sensing data in the memory cache of each cloud node according to algorithms such as LRU (Least Recently Used) and LFU (Least Frequently Used).

[0137] Figure 11 A flowchart illustrating a distributed multi-cloud node remote sensing data synchronization transmission method according to an embodiment of this application is shown.

[0138] like Figure 11 As shown, the distributed multi-cloud node remote sensing data synchronization and transmission method includes operations S110-S140.

[0139] In operation S110, a user's data synchronization request is received.

[0140] Data synchronization requests are synchronized on a data set defined by business logic, and the logic of this data set is defined by at least one business dimension information.

[0141] In the field of remote sensing data processing, a business logic-defined data set refers to data units organized according to the inherent business relationships between data processing and application. A business logic-defined data set can include whole-orbit observation data, data segments, data scenes, and standard products. Whole-orbit observation data can be all the data acquired by a remote sensing satellite continuously observing an entire orbit. A data segment is a segment data file formed by dividing whole-orbit data into certain time or geographical intervals, along with its accompanying metadata. A data scene is the smallest logical unit of remote sensing data, referring to an image acquired in a single imaging operation, along with its complete metadata, preview map, auxiliary parameters, etc. Standard products are usable data products generated after standardization processing such as radiometric calibration and geometric correction of the raw observation data.

[0142] Business dimension information refers to the business attribute parameters used to define and query the aforementioned data set. This can include satellite model, payload type, acquisition time, and spatial location. Satellite model information includes specific satellite models from projects such as the High Resolution Imaging System, the Oceanographic Series, and the Tianhui Series. Payload type information includes panchromatic cameras, multispectral cameras, hyperspectral cameras, synthetic aperture radar (SAR), and infrared sensors. Acquisition time refers to the specific point in time or time range of data collection. Spatial location refers to the geographical region or latitude and longitude coordinate range corresponding to the data.

[0143] In some embodiments, a data synchronization request described by a user in natural language can be received through a natural language interaction interface; the remote sensing big language model can be invoked through the natural language interaction interface to parse the data synchronization request and extract at least one dimension of information from satellite model, payload type, shooting time, spatial location, and data type; and the synchronization task request can be generated based on the dimension information.

[0144] In operation S120, in response to the data synchronization request, all data entities logically corresponding to the data set are extracted from the distributed data source based on the business dimension information.

[0145] Distributed data sources refer to various heterogeneous databases and storage systems that store multimodal remote sensing data in a distributed multi-cloud node environment. Distributed data sources can include databases for storing structured data. These databases can be relational databases, in-memory databases, or spatiotemporal databases. Relational databases, such as MySQL and PolarDB, are used to store metadata and attribute data. In-memory databases, such as Redis, are used to store frequently accessed feature parameters and key-value pairs. Spatiotemporal databases, such as PostgreSQL (with PostGIS extensions), are used to store data with spatiotemporal attributes.

[0146] A storage system is a system used to store unstructured physical data files. Storage systems can be memory caches, high-speed file storage systems, and distributed file / object storage systems, etc. Memory caches are high-speed caches based on server memory, used for temporarily storing frequently accessed data. High-speed file storage systems are file systems based on high-speed storage devices such as SSDs. Distributed file / object storage systems, such as NAS, OSS, and HDFS, are used for persistent storage of massive amounts of physical files (images, videos, etc.).

[0147] All data refers to all data components that constitute a complete business logic data set. This can include structured data and unstructured entity data files. Structured data includes metadata, attribute data, and feature parameter data. Unstructured entity data files include raw bitstream data files (AOS frame format), payload formatted data files, image files (such as TIFF and IMG formats), video files, thumb maps, view maps, and DOC document reports.

[0148] Metadata consists of information describing data attributes, sources, and processing procedures (such as XML-formatted metadata files stored in a database). Attribute data comprises various characteristic attributes of the observed target. Feature parameter data consists of various parameters stored in key-value pairs.

[0149] In operation S130, the synchronization task for all data entities is decomposed into subtasks corresponding to different data types.

[0150] Subtasks corresponding to different data types can be divided into two categories of parallel subtasks based on the data format and storage location. These subtasks can include database synchronization subtasks and data file synchronization subtasks.

[0151] The database synchronization subtask is responsible for synchronizing structured data stored in various databases to the destination node.

[0152] The data file synchronization subtask is responsible for synchronizing unstructured entity data files stored in various storage systems to the destination node.

[0153] In operation S140, based on the transmission strategy, the data to be transmitted in the subtask is preprocessed and then transmitted through a dynamically selected network link, so that the destination cloud node can perform post-processing operations corresponding to the preprocessing on the received data to restore the data set.

[0154] Preprocessing / postprocessing operations refer to reversible operations performed on data before and after transmission, designed to improve efficiency, security, and reliability. Preprocessing / postprocessing operations can include encryption / decryption, compression / decompression, and data splitting / merging.

[0155] Encryption / Decryption: Using algorithms such as DES and AES to encrypt and decrypt data during transmission.

[0156] Compression / Decompression: Select an algorithm based on the data type (e.g., ZIP / RAR for packaging, JPEG 2000 / WebP for image compression) to reduce data size, and decompress after transmission.

[0157] Chunking / Merging: Divide a large file into multiple data blocks (such as by complete data frames or a nine-grid) for parallel transmission, and merge and restore them at the destination cloud node.

[0158] Verification: Calculate and verify checksums such as MD5, SHA1, and CRC cyclic redundancy check, or check the data frame synchronization header to ensure data integrity.

[0159] Dynamically selected network links refer to the optimal path chosen from multiple available paths between the source node and the destination node based on real-time network conditions. Dynamically selected network links can include: link status metrics such as connectivity, bandwidth, latency, packet loss rate, and load; task policy attributes such as task priority, urgency, and timeliness requirements; and for cross-domain scenarios without direct links, the link may include paths that relay through a central cloud node.

[0160] In some embodiments, a structured query statement can be generated based on the business dimension information; based on the structured query statement, structured data logically related to the data set can be extracted from the registered relational database, in-memory database, or spatiotemporal database. The structured data includes metadata, attribute data, and feature parameter data; based on the entity data association relationships recorded in the metadata, unstructured entity data files stored in memory cache, high-speed file storage system, or distributed object storage can be extracted; the structured data and unstructured entity data files can be logically associated to jointly constitute all data entities of the data set defined by the business logic.

[0161] In some embodiments, the form of data in all data entities can be identified; the synchronization task can be decomposed into at least two different types of subtasks, including a database synchronization subtask and a data file synchronization subtask; for the data file synchronization subtask, based on the logical organization structure of the unstructured entity data file, subtasks for parallel transmission can be divided and created using data scenes as the basic unit.

[0162] In some embodiments, the database synchronization subtask is used to synchronize structured data stored in a relational database, an in-memory database, or a spatiotemporal database; the data file synchronization subtask is used to synchronize unstructured entity data files stored in a memory cache, a high-speed file storage system, or a distributed object storage.

[0163] In some embodiments, based on a transmission strategy configured independently for each subtask, a corresponding preprocessing operation can be performed on the data to be transmitted at the sending end; according to the attributes of the subtask and the real-time status of multiple network links, a target link can be dynamically selected from the multiple network links; and the preprocessed data to be transmitted can be transmitted through the target link.

[0164] In some embodiments, the type of preprocessing operation that the received data has undergone can be identified; based on the type of preprocessing operation, the corresponding postprocessing service is invoked to perform the reverse operation; for the data that has undergone the reverse operation, integrity and correctness checks are performed; the structured data and unstructured entity data files that have passed the checks are stored according to the corresponding business logic association relationship to generate a data set consistent with the business logic definition of the sending end.

[0165] In some embodiments, the preprocessing operation includes a compression operation. The preprocessing of the data to be transmitted in the subtask based on the transmission strategy includes: obtaining the file attributes and business characteristics of the data to be transmitted; the file attributes include the file extension and internal format signature; the business characteristics are defined by business dimension information included in the data synchronization request; matching the file attributes and business characteristics with a predefined compression strategy selection table; the compression strategy selection table records the mapping relationship between file types, compression algorithms, and compression parameters; and automatically selecting and calling the corresponding compression algorithm and compression parameters to perform a compression operation on the data to be transmitted based on the matching result.

[0166] In some embodiments, the preprocessing operation includes a chunking operation. The preprocessing of the data to be transmitted in the subtask based on the transmission strategy includes: identifying the file type and network environment status of the data to be transmitted; selecting a corresponding chunking method from multiple chunking strategies according to the file type of the data to be transmitted; and dynamically adjusting the size of the data blocks into which the data to be transmitted is divided according to the network environment status. The chunking strategy includes at least one of the following: for file types with a fixed frame format, chunking is performed according to a fixed number of complete data frames; for raster image file types, chunking is performed according to a gridding method; for video file types, chunking is performed according to a fixed duration; and for file types with an ambiguous format, chunking is performed according to a fixed byte size.

[0167] In some embodiments, the preprocessing operation includes a memory caching operation. The preprocessing of the data to be transmitted in the subtask based on the transmission strategy includes: constructing a memory virtualization logical cache pool using the memory of the server node; and loading the data to be transmitted into the memory virtualization logical cache pool for caching to enable read and write access.

[0168] Figure 12 A flowchart illustrating another distributed multi-cloud node remote sensing data synchronization and transmission method according to an embodiment of this application is shown.

[0169] like Figure 12 As shown, the distributed multi-cloud node remote sensing data synchronization transmission method includes operations S1-S7 applied to the sending end.

[0170] S1. Business users register and configure various database instances, such as relational, in-memory, and spatiotemporal databases, for source and destination nodes, and register and mount storage resources such as memory cache, high-speed file storage, and distributed storage.

[0171] S2. Based on registered and configured data sources, and according to multi-dimensional information such as satellite model, payload type, imaging time, spatial location, data type, theme, and tags, it queries and selects remote sensing datasets to be synchronized, configures synchronization and transmission strategies, and customizes and initiates synchronization and transmission tasks. Alternatively, it can intelligently initiate synchronization and transmission tasks through interactive question-and-answer sessions with remote sensing data query and selection units using natural language.

[0172] S3, the synchronization task management and scheduling module, analyzes, decomposes, and creates sub-tasks for remote sensing data synchronization tasks, and schedules each sub-task to be executed according to the task load balancing strategy and job scheduling strategy. Simultaneously, it sends task instructions to the task management and scheduling module of the destination cloud node.

[0173] S4. Based on the task strategy and the characteristics of the remote sensing data to be transmitted, such as storage location, composition, and size, perform preprocessing operations such as data caching, encryption, compression, and segmentation as needed.

[0174] S5. Assess the network status between the source and destination nodes of the synchronization task, select appropriate transmission links and transmission protocols for each subtask based on the task strategy, and call the database or data file synchronization and transmission service to send data to the destination cloud node.

[0175] S6. Users can monitor the status of synchronization tasks and perform management and control operations such as pausing, canceling, continuing, and limiting the speed as needed until the task is completed.

[0176] S7. After the data is sent, release the relevant connection resources, manage the data cached in memory according to the caching strategy, and perform lifecycle management of the cached data.

[0177] like Figure 12 As shown, the distributed multi-cloud node remote sensing data synchronization transmission method includes operations S1'-S4' applied to the target cloud node.

[0178] S1': Based on the task scheduling instructions and load balancing strategy, schedule a suitable receiving node, check the buffer space, and complete the receiving preparation.

[0179] S2': Receive transmission requests, establish transmission links, cache received data, and perform decryption, decompression, merging, and verification as needed to ensure data integrity and correctness.

[0180] S3' During the receiving process, according to the task strategy, the backend data processing software is notified to process and utilize the cached data, achieving simultaneous transmission and use.

[0181] S4' After the data is received and processed, manage the data cached in memory according to the caching strategy and perform lifecycle management of the cached data.

[0182] According to embodiments of this application, a method for synchronizing and transmitting remote sensing data using its operational logic organizational units as the basic unit is described. Unlike synchronization and transmission based on individual files or database tables, this method uses logical organizational units of remote sensing data, such as whole-orbit observation data, single-segment observation data, data segments, data scenes, or standard products from remote sensing satellites. It extracts the multimodal datasets to be synchronized and transmitted according to the inherent relationships and structural logic of the remote sensing data, and then calls the synchronization service for each multimodal data to perform synchronization and transmission.

[0183] According to the embodiments of this application, a high-efficiency remote sensing data synchronization and transmission system is designed, which can realize the high-efficiency synchronization and transmission of multimodal remote sensing data between distributed multi-cloud nodes of aerospace remote sensing ground system. The system includes modules such as remote sensing data source configuration and synchronization customization, data synchronization task management and scheduling, data preprocessing to be synchronized, database synchronization and transmission service, data file synchronization and transmission service, and synchronization data cache management.

[0184] According to embodiments of this application, with the support of a remote sensing large language model, users can interact with remote sensing data query and selection units using natural language to raise remote sensing data synchronization requests. The system can intelligently extract multi-dimensional information such as remote sensing satellite model, payload type, imaging time, spatial location, data type, data theme, and data tags. Then, it automatically queries and extracts the remote sensing data to be synchronized, forming a remote sensing dataset to be synchronized. This significantly reduces the time and complexity for users searching, filtering, and determining the data files or records to be synchronized.

[0185] According to the embodiments of this application, a transmission cluster memory caching service is designed. A file system is created using the memory of each server node in the transmission cluster, constructing a memory virtualization logical cache pool. A unified data access interface is provided to software services such as synchronous data preprocessing and data transmission, enabling fast and efficient access to remote sensing data, improving data read and write efficiency, and making remote sensing data preprocessing and transmission efficient and flexible. Its usage mode is as follows: Figure 13 As shown.

[0186] According to the embodiments of this application, a link monitoring and optimization service is designed, which can track, monitor and maintain the link status between the source node and the destination node of the synchronization and transmission task. It comprehensively evaluates each link through indicators such as connectivity, bandwidth, load, packet loss rate, and latency, and optimizes the most suitable transmission link by combining the priority, urgency and timeliness requirements of the task. It can also realize cross-domain and cross-network segment data synchronization and transmission through relay nodes.

[0187] According to the embodiments of this application, unlike general methods that mainly support traditional transmission protocols such as TCP and UDP, this application supports multiple transmission protocols. In addition to TCP and UDP, it also supports PDXP and FEP protocols commonly used in the remote sensing field, as well as KCP and QUIC protocols for complex network environments.

[0188] According to the embodiments of this application, unlike traditional methods that uniformly adopt a single compression method such as RAR, this application designs multiple compression methods for multimodal remote sensing data, enabling the selection of appropriate compression methods for different types of data. For binary files such as raw bitstreams and formatted data, as well as small files such as text, XML, and DOC documents, software modules such as RAR and ZIP are used for packaging and compression. For large data files such as high-resolution, wide-swath remote sensing images and videos, algorithms such as JPEG-LS, JPEG 2000, and WebP can be selected for lossless or minimally lossy compression, greatly improving the compression effect.

[0189] According to the embodiments of this application, for data block transmission, in addition to supporting the general strategy of dividing data blocks into fixed byte sizes, this application also designs targeted segmentation methods for different types of remote sensing data, facilitating simultaneous transmission and use. For remote sensing data files with fixed frame formats, such as raw bitstreams and formatted data, data blocks are divided according to a fixed number of data frames, ensuring that each data block contains a complete data frame, allowing each data block to be processed directly, facilitating simultaneous transmission and processing. For wide-format, large-area remote sensing raster images, image blocks are supported using grid-like, nine-square, or other grid-based segmentation methods, and then parallel transmission is used to achieve parallel transmission and loading display of the image grid, fully realizing simultaneous transmission and use.

[0190] According to the embodiments of this application, based on the support of the cloud platform, it is possible to realize customized fine-grained synchronization of remote sensing data in relational, in-memory, and spatiotemporal databases. It supports one-click completion of database instance creation, database structure synchronization and initialization, and full and incremental data synchronization. It can automatically complete the synchronization of application-oriented remote sensing business data according to the association and dependency relationships of remote sensing data.

[0191] Any one or more of the modules, submodules, units, and subunits according to the embodiments of this application, or at least part of the functions of any one or more of them, can be implemented in one module. Any one or more of the modules, submodules, units, and subunits according to the embodiments of this application can be implemented by dividing them into multiple modules. Any one or more of the modules, submodules, units, and subunits according to the embodiments of this application can be at least partially implemented as hardware circuits, such as field-programmable gate arrays (FPGAs), programmable logic arrays (PLAs), systems-on-a-chip, systems-on-a-substrate, systems-on-package, application-specific integrated circuits (ASICs), or implemented by hardware or firmware in any other reasonable manner by integrating or packaging circuits, or implemented in any one of software, hardware, and firmware, or in a suitable combination of any of these. Alternatively, one or more of the modules, submodules, units, and subunits according to the embodiments of this application can be at least partially implemented as computer program modules, which, when run, can perform corresponding functions.

[0192] For example, any and more of the following modules can be implemented in one module: remote sensing data source configuration and synchronization customization module, data synchronization task management and scheduling module, data preprocessing module, database synchronization and transmission service module, data file synchronization and transmission service module, and synchronization data cache management module. Alternatively, any one of these modules can be split into multiple modules. Or, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module. According to embodiments of this application, at least one of the following modules can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable method of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the following modules can be implemented, at least partially, as a computer program module: remote sensing data source configuration and synchronization customization module, data synchronization task management and scheduling module, data preprocessing module to be synchronized module, database synchronization and transmission service module, data file synchronization and transmission service module, and synchronization data cache management module. When the computer program module is run, it can perform the corresponding functions.

[0193] According to embodiments of this application, this application also provides a readable storage medium and a computer program product.

[0194] According to an embodiment of this application, a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to perform the method described above.

[0195] According to an embodiment of this application, a computer program product includes a computer program that, when executed by a processor, implements the method described above.

[0196] It should be noted that the display control device part in the embodiments of this application corresponds to the display control method part in the embodiments of this application. For a detailed description of the display control system part, please refer to the display control method part, which will not be repeated here.

[0197] Figure 14 A block diagram of an electronic device suitable for implementing the methods described above, according to an embodiment of this application, is illustrated schematically. Figure 14 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0198] like Figure 14 As shown, an electronic device according to an embodiment of this application includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage portion 608 into a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this application.

[0199] RAM 603 stores various programs and data required for the operation of electronic device 600. Processor 601, ROM 602, and RAM 603 are interconnected via bus 604. Processor 601 executes various operations of the method flow according to embodiments of this application by executing programs in ROM 602 and / or RAM 603. It should be noted that the programs may also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 may also execute various operations of the method flow according to embodiments of this application by executing programs stored in said one or more memories.

[0200] According to embodiments of this application, the electronic device may further include an input / output (I / O) interface 605, which is also connected to a bus 604. The system 600 may also include one or more of the following components connected to the input / output (I / O) interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the input / output (I / O) interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 610 as needed so that computer programs read from it can be installed into the storage section 608 as needed.

[0201] According to embodiments of this application, the method flow according to embodiments of this application can be implemented as a computer software program. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611. When the computer program is executed by processor 601, it performs the functions defined in the system of embodiments of this application. According to embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0202] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0203] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium. Examples include, but are not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0204] For example, according to embodiments of this application, a computer-readable storage medium may include the ROM 602 and / or RAM 603 described above and / or one or more memories other than ROM 602 and RAM 603.

[0205] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods provided in the embodiments of this application. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the methods provided in the embodiments of this application.

[0206] When the computer program is executed by the processor 601, it performs the functions defined in the system / apparatus of this application embodiment. According to the embodiments of this application, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0207] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via the communication section 609, and / or installed from the removable medium 611. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0208] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0209] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Those skilled in the art will understand that the features recited in the various embodiments and / or claims of this application can be combined and / or combined in various ways, even if such combinations or combinations are not expressly stated in this application. In particular, the various embodiments and / or features described in the claims of this application may be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.

[0210] The embodiments of this application have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of this application. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this application is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this application, and all such substitutions and modifications should fall within the scope of this application.

Claims

1. A method for synchronous transmission of distributed multi-cloud node remote sensing data, characterized in that, The method includes: Receive user data synchronization requests, wherein the data synchronization requests are based on a data set defined by business logic as the synchronization unit, and the logic of the data set is defined by at least one business dimension information; In response to the data synchronization request, all data entities logically corresponding to the data set are extracted from the distributed data source based on the business dimension information. The synchronization task for all data entities is decomposed into sub-tasks corresponding to different data types; Based on the transmission strategy, the data to be transmitted in the subtask is preprocessed and then transmitted through a dynamically selected network link, so that the destination cloud node can perform post-processing operations corresponding to the preprocessing on the received data to restore the data set. Among them, dynamically selected network links refer to the optimal path selected from multiple available paths between the source node and the destination node based on real-time network conditions; In response to the data synchronization request, extracting all data entities logically corresponding to the data set from the distributed data source based on the business dimension information includes: Receive the user's synchronization request description in natural language. The remote sensing big language model is used to perform semantic parsing on the synchronization requirement description, and to identify and extract the business dimension information that defines the logic of the data set; Based on the aforementioned business dimension information, generate a structured query statement; Based on the structured query statement, structured data logically related to the data set is extracted from the registered relational database, in-memory database, or spatiotemporal database. The structured data includes metadata, attribute data, and feature parameter data. Based on the entity data relationships recorded in the metadata, extract unstructured entity data files stored in memory cache, high-speed file storage system or distributed object storage; The structured data and unstructured entity data files are logically associated to form all the data entities of the data set defined by the business logic. The step of decomposing the synchronization task for all data entities into sub-tasks corresponding to different data types includes: Identify the format of data in all the data entities; The synchronization task is decomposed into at least two types of subtasks of different natures, including a database synchronization subtask and a data file synchronization subtask. For the data file synchronization subtask, based on the logical organization structure of the unstructured entity data file, subtasks for parallel transmission are divided and created using data scenes as the basic unit. The database synchronization subtask is used to synchronize structured data stored in relational databases, in-memory databases, or spatiotemporal databases; The preprocessing operation includes a block operation, and the preprocessing of the data to be transmitted in the subtask based on the transmission strategy includes: Identify the file type and network environment status of the data to be transmitted; Based on the file type of the data to be transmitted, select the corresponding chunking method from multiple chunking strategies; The size of the data blocks into which the data to be transmitted is divided is dynamically adjusted according to the network environment status.

2. The method according to claim 1, characterized in that, The data set defined by the business logic corresponds to the business logic organization unit in the field of remote sensing data processing, and the business dimension information includes at least one of satellite model, payload type, shooting time, and spatial location; The business logic organization unit includes whole-track observation data, data segments, data scenes, or standard products; Among them, whole-orbit observation data is all the data obtained by a remote sensing satellite continuously observing an entire orbit, and data segment is a segment data file and its accompanying metadata formed by dividing the whole-orbit data according to time or geographical intervals.

3. The method according to claim 1, characterized in that, The preprocessing operation includes a compression operation, and the preprocessing of the data to be transmitted in the subtask based on the transmission strategy includes: Obtain the file attributes and business characteristics of the data to be transmitted. The file attributes include the file extension and internal format signature, and the business characteristics are defined by the business dimension information contained in the data synchronization request. The file attributes and business characteristics are matched with a predefined compression strategy selection table, which records the mapping relationship between file type, compression algorithm and compression parameters; Based on the matching results, the corresponding compression algorithm and compression parameters are automatically selected and invoked to perform compression operations on the data to be transmitted.

4. The method according to claim 1, characterized in that, The block partitioning strategy includes at least one of the following: For file types with a fixed frame format, the data is divided into blocks based on a fixed number of complete data frames. For raster image file types, divide them into blocks according to the gridding method; For video file types, divide them into blocks of fixed duration; For file types with ambiguous formats, divide them into blocks of fixed byte size.

5. The method according to claim 1, characterized in that, The preprocessing operation includes memory caching operations, and the preprocessing of the data to be transmitted in the subtask based on the transmission strategy includes: Utilize the memory of server nodes to build a memory virtualization logical cache pool; The data to be transmitted is loaded into the memory virtualization logical cache pool for caching, so that it can be accessed for reading and writing.

6. The method according to claim 1, characterized in that, The destination cloud node performs post-processing operations on the received data corresponding to the pre-processing, restoring the data set including: Identify the type of preprocessing operation that the received data has undergone; Based on the preprocessing operation type, the corresponding postprocessing service is invoked to perform the reverse operation; For the data that has undergone the inverse operation, perform integrity and correctness checks. The validated structured data and unstructured entity data files are stored according to their corresponding business logic relationships to generate a data set consistent with the business logic definition of the sending end.

7. A distributed multi-cloud node remote sensing data synchronization and transmission system, characterized in that, The system includes: The remote sensing data source configuration and synchronization customization module is used to receive data synchronization requests from users and extract all data entities corresponding to the data set logic from the distributed data source according to the business dimension information contained in the data synchronization request. The data synchronization request takes the data set defined by the business logic as the synchronization unit, and the logic of the data set is defined by at least one business dimension information. The data synchronization task management and scheduling module is connected to the remote sensing data source configuration and synchronization customization module. It is used to receive the data entity, decompose the synchronization task for the data entity into sub-tasks corresponding to different data types, and schedule the sub-tasks. The data preprocessing module to be synchronized is connected to the data synchronization task management and scheduling module, and is used to perform preprocessing operations on the data to be transmitted in the scheduled subtasks based on the transmission strategy. The database synchronization and transmission service module is connected to the data synchronization task management and scheduling module and is used to execute database synchronization subtasks; The data file synchronization and transmission service module is connected to the data synchronization task management and scheduling module, and is used to execute the data file synchronization sub-task and transmit it through a dynamically selected network link; The synchronous data cache management module is connected to the data to be synchronized preprocessing module, the database synchronization and transmission service module, and the data file synchronization and transmission service module, respectively. It is used to build and manage the memory virtualization logical cache pool at the sending end and the destination cloud node, and provide high-speed read and write access support for the preprocessing, transmission and postprocessing of the data to be transmitted. Among them, dynamically selected network links refer to the optimal path selected from multiple available paths between the source node and the destination node based on real-time network conditions; In response to the data synchronization request, extracting all data entities logically corresponding to the data set from the distributed data source based on the business dimension information includes: Receive the user's synchronization request description in natural language. The remote sensing big language model is used to perform semantic parsing on the synchronization requirement description, and to identify and extract the business dimension information that defines the logic of the data set; Based on the aforementioned business dimension information, generate a structured query statement; Based on the structured query statement, structured data logically related to the data set is extracted from the registered relational database, in-memory database, or spatiotemporal database. The structured data includes metadata, attribute data, and feature parameter data. Based on the entity data relationships recorded in the metadata, extract unstructured entity data files stored in memory cache, high-speed file storage system or distributed object storage; The structured data and unstructured entity data files are logically associated to form all the data entities of the data set defined by the business logic. The step of decomposing the synchronization task for all data entities into sub-tasks corresponding to different data types includes: Identify the format of data in all the data entities; The synchronization task is decomposed into at least two types of subtasks of different natures, including a database synchronization subtask and a data file synchronization subtask. For the data file synchronization subtask, based on the logical organization structure of the unstructured entity data file, subtasks for parallel transmission are divided and created using data scenes as the basic unit. The database synchronization subtask is used to synchronize structured data stored in relational databases, in-memory databases, or spatiotemporal databases; The preprocessing operation includes a block operation, and the preprocessing of the data to be transmitted in the subtask based on the transmission strategy includes: Identify the file type and network environment status of the data to be transmitted; Based on the file type of the data to be transmitted, select the corresponding chunking method from multiple chunking strategies; The size of the data blocks into which the data to be transmitted is divided is dynamically adjusted according to the network environment status.

8. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 6.

Images