Distributed Processing System of Remote Sensing Data
By dividing remote sensing data into concurrent processing of tile, the problem of low processing efficiency of remote sensing data is solved, the response time is shortened and the processing efficiency is improved, and the system load changes are adapted to changes.
Patent Information
- Application Number
- CN202310536829.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-10
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2043-05-10
AI Technical Summary
The existing interactive remote sensing data analysis system has a long response time and low remote sensing data processing efficiency.
Adopting a distributed processing system architecture, remote sensing data is divided into multiple tile data blocks, and distributed to multiple computing nodes for processing. The distributed storage system, access nodes and computing nodes respectively load and process the tile. Data blocks improve processing efficiency and response time.
By concurrently processing tile data blocks, the response time of remote sensing data processing is significantly shortened, processing efficiency is improved, and the distributed system architecture allows each node to independently expand and shrink capacity to adapt to load changes.
Smart Images

Figure CN116594977B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to computer technology, and in particular to a distributed processing system for remote sensing data. Background Art
[0002] In the field of remote sensing data processing, the interactive analysis system of remote sensing data is a cloud platform for processing and analyzing remote sensing data, which integrates online result visualization and offline result export functions. In developer mode, based on the massive remote sensing data and rich basic function capabilities provided by the system, users can develop online projects for remote sensing data processing logic. However, the current interactive analysis system of remote sensing data has a long response time from user code submission to system output of rendered visualization results, and the efficiency of remote sensing data processing is low. Summary of the invention
[0003] The present application provides a distributed processing system for remote sensing data, which is used to solve the problems of response time field of remote sensing data interactive analysis system and low efficiency of remote sensing data processing.
[0004] On the one hand, the present application provides a distributed processing system for remote sensing data, comprising: a cloud server,
[0005] The cloud server comprises:
[0006] Distributed storage system for storing collected remote sensing data;
[0007] The access node is used to respond to the remote sensing data processing request, obtain the area range to be calculated, the processing logic to be executed and the identification of the target remote sensing data to be processed, distribute the identification of the target tile falling within the area range to the computing node based on the tiles divided by the map space, and send the identification of the target remote sensing data and the processing logic to the computing node;
[0008] A computing node, configured to load the data block of the target remote sensing data falling into the target tile from the distributed storage system according to the identifier of the distributed target tile, and execute the processing logic on the data block to obtain the processing result of the data block in the target tile;
[0009] The access node is further used to output the remote sensing data processing result within the area, and the remote sensing data processing result within the area includes the processing result of the data block in each of the target tiles.
[0010] On the other hand, the present application provides a distributed processing system for remote sensing data, including: a terminal device,
[0011] The terminal device provides a visualization interface, wherein the visualization interface is used to determine the processing logic to be executed, the identification information of the remote sensing data to be processed, and the area range to be calculated;
[0012] In response to the run instruction, a remote sensing data processing request is sent to the cloud server, wherein the remote sensing data processing request includes a region range to be calculated, a processing logic to be executed, and an identifier of the target remote sensing data to be processed;
[0013] Receive the remote sensing data processing result within the area range sent by the cloud server, the remote sensing data processing result includes loading the data block of the target remote sensing data falling into the target tile from the distributed storage system according to the identifier of the target tile falling into the area range, and executing the processing logic on the data block to obtain the processing result of the data block in the target tile;
[0014] The remote sensing data processing results within the area are displayed on the visualization interface.
[0015] The distributed processing system of remote sensing data provided by the present application provides a visual interface for the end-side device. Based on the visual interface, the user can give the processing logic to be executed, the identification information of the remote sensing data to be processed, and the area range to be calculated. In response to the running instruction, the end-side device sends a remote sensing data processing request to the cloud server, and the remote sensing data processing request contains the area range to be calculated, the processing logic to be executed, and the identification of the target remote sensing data to be processed. The cloud server includes: a distributed storage system, an access node, and a computing node. The distributed storage system stores the collected remote sensing data. The access node responds to the remote sensing data processing request, obtains the area range to be calculated, the processing logic to be executed, and the identification of the target remote sensing data to be processed, and distributes the identification of the target tile falling within the area range to the computing node based on the tiles divided by the map space, and sends the identification of the target remote sensing data and the processing logic to the computing node. The computing node loads the data block of the target remote sensing data falling within the target tile from the distributed storage system according to the identification of the distributed target tile, and executes the processing logic on the data block to obtain the processing result of the data block within the target tile. The access node outputs the remote sensing data processing result within the area to the end-side device, and the remote sensing data processing result within the area includes the processing result of the data block in each target tile. The end-side device receives the remote sensing data processing result within the area sent by the cloud server, and displays the remote sensing data processing result within the area on the visualization interface.
[0016] The solution of the present application is that the cloud server adopts a distributed system architecture, divides the target remote sensing data within the area to be calculated into data blocks of multiple target tiles, distributes the data blocks of each target tile to multiple computing nodes, and each computing node executes the processing of the data blocks of the target remote sensing data falling into the target tile respectively, takes the data block of the tile as the processing unit, and concurrently executes the processing of the data blocks of each target tile, thereby improving the efficiency of remote sensing data processing and shortening the response time. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0018] Figure 1 An exemplary architecture diagram of a distributed processing system for remote sensing data applicable to the present application;
[0019] Figure 2 An architectural diagram of a cloud server provided for an exemplary embodiment of the present application;
[0020] Figure 3 A flow chart of a method for distributed processing of remote sensing data provided by an exemplary embodiment of the present application;
[0021] Figure 4 A schematic diagram of a visualization interface provided for an exemplary embodiment of the present application;
[0022] Figure 5 A flow chart of a method for distributed processing of remote sensing data provided by another exemplary embodiment of the present application;
[0023] Figure 6 A schematic diagram of the structure of a function abstract syntax tree provided for an exemplary embodiment of the present application;
[0024] Figure 7 A schematic diagram of an operation layer provided for an exemplary embodiment of the present application;
[0025] Figure 8 A schematic diagram of implementing data block loading within a tile based on a grid pyramid provided by an exemplary embodiment of the present application;
[0026] Fig. 9 A schematic diagram of the structure of a terminal device provided in an embodiment of the present application.
[0027] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION
[0028] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.
[0029] First, the terms involved in this application are explained:
[0030] Abstract Syntax Tree (AST): Also known as a syntax tree, it describes a directed acyclic graph (DAG) of function calls, with dependencies between nodes (referring to functions).
[0031] Basic operation unit (Operator): The basic operation unit for data set in the system, with predetermined input and output.
[0032] Operator Level: refers to the hierarchy of basic operation units (Operators) based on dependency relationships.
[0033] Raster pyramid: It is a downsampled version of the original raster dataset, which can contain multiple downsampled layers. In this application, the remote sensing data is rasterized into the original raster data, and multiple downsampled layers are generated based on the downsampling of the original raster data to obtain a raster pyramid consisting of multiple layers of raster data with different resolutions. Each continuous layer in the raster pyramid is downsampled at a ratio of 2:1.
[0034] GeoTiff: It is a geographic spatial raster data storage format based on the Tag Image File Format (TIFF), and is also one of the more common raster data storage formats.
[0035] Map space: refers to the complete map space, that is, the global map space.
[0036] Tile: refers to the sub-areas of a specified unit size (such as 256*256) that the map space is divided into. Different tiles in the map space have different identifiers and coordinate ranges.
[0037] In response to the problems of low response time field and remote sensing data processing efficiency of existing remote sensing data interactive analysis systems, the present application provides a distributed processing system for remote sensing data, including a cloud server and terminal devices.
[0038] The end-side device provides a visual interface, based on which the user can give the processing logic to be executed, the identification information of the remote sensing data to be processed, and the area to be calculated. In response to the run instruction, the end-side device sends a remote sensing data processing request to the cloud server, and the remote sensing data processing request includes the area to be calculated, the processing logic to be executed, and the identification of the target remote sensing data to be processed.
[0039] The cloud server includes: a distributed storage system, an access node and a computing node. The distributed storage system is used to store the collected remote sensing data. The access node is used to respond to remote sensing data processing requests, obtain the area range to be calculated, the processing logic to be executed and the identification of the target remote sensing data to be processed, distribute the identification of the target tiles falling within the area range to the computing node based on the tiles divided by the map space, and send the identification of the target remote sensing data and the processing logic to the computing node. The computing node is used to load the data blocks of the target remote sensing data falling within the target tile from the distributed storage system according to the identification of the distributed target tile, and execute the processing logic on the data block to obtain the processing results of the data block within the target tile. The access node is also used to output the remote sensing data processing results within the area range to the end-side device, and the remote sensing data processing results within the area range include the processing results of the data blocks within each target tile.
[0040] The end-side device receives the remote sensing data processing results within the regional range sent by the cloud server, and displays the remote sensing data processing results within the regional range on the visualization interface.
[0041] The distributed processing system of remote sensing data provided by the present application adopts a distributed system architecture in which the cloud server divides the target remote sensing data within the area to be calculated into data blocks of multiple target tiles, distributes the data blocks of each target tile to multiple computing nodes, and each computing node performs the processing of the data blocks of the target remote sensing data falling into the target tile respectively, takes the data block of the tile as the processing unit, and concurrently performs the processing of the data blocks of each target tile, thereby improving the efficiency of remote sensing data processing and shortening the response time. In addition, in the distributed system architecture of the cloud server, the access nodes, computing nodes and distributed storage systems are separated from each other, and have the ability to expand and shrink capacity independently.
[0042] Figure 1 This is an exemplary architecture diagram of a distributed processing system for remote sensing data applicable to this application, such as Figure 1 As shown, the system architecture includes a cloud server, a terminal device and a remote sensing data service device.
[0043] The cloud server may be a server cluster arranged in the cloud, such as a cloud computing platform, etc. A communication link is provided between the cloud server and each terminal device, so that a communication connection between the cloud server and each terminal device can be realized.
[0044] The terminal device is the device used by the user, which can be a hardware device with network communication function, computing function and information display function, including but not limited to terminal devices such as tablet computers and personal computers used by users, servers of service platforms or institutions, etc.
[0045] The remote sensing data service device is a device that provides remote sensing data to the cloud server. It can be various sensors that collect remote sensing data, such as satellites, drone photography equipment, ground Internet of Things devices (IoT), meteorological equipment, etc. It can also be other data service platforms, which are not specifically limited here.
[0046] The cloud server obtains multi-source remote sensing data collected by various types of sensors from the remote sensing data service device, including but not limited to satellite images, drone images, meteorological data, IoT data, etc. The cloud server divides the acquired remote sensing data into grids, generates raw raster data, and constructs a raster pyramid of remote sensing data. The raster pyramid contains multiple layers of raster data, and the raster data of different layers are remote sensing data of different resolutions. The cloud server stores the raster pyramid of remote sensing data in a distributed storage system. In addition, the cloud server divides the map space (global map space) into tiles of a specified unit size, determines the offset of the data block in the tile covered by the remote sensing data in the remote sensing data, and stores the offset corresponding to the tile covered by the remote sensing data as metadata information to the meta-information service node. Based on the offset of the data block in the tile, the computing node can load the data block of the remote sensing data falling into the tile from the remote sensing data stored in the distributed storage system without loading the entire remote sensing data.
[0047] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.
[0048] Figure 2 This is an architectural diagram of a cloud server in a distributed processing system for remote sensing data provided by an exemplary embodiment of the present application. Figure 2 As shown in FIG, the cloud server includes a distributed storage system, access nodes, and computing nodes. Figure 2 As shown in , the cloud server adopts a distributed system architecture and may include multiple access nodes and multiple computing nodes.
[0049] The distributed storage system is used to store the collected remote sensing data. Exemplarily, the distributed storage system can be implemented by the following distributed storage systems: Pangu file system, Object Storage Service (OSS), Distributed File System (HDFS), or other distributed storage systems, which are not specifically limited here. Pangu file system is a commonly used distributed file storage system.
[0050] The access node is used to receive and respond to remote sensing data processing requests from end-side devices. Based on the remote sensing data processing requests, it obtains the area scope to be calculated, the processing logic to be executed, and the identification of the target remote sensing data to be processed. Based on the tiles divided by the map space, the identification of the target tiles falling within the area scope is distributed to the computing node, and the identification and processing logic of the target remote sensing data are sent to the computing node.
[0051] The computing node includes components such as data loader, cache, operation execution engine, scheduler, etc., which are mainly responsible for data loading, calculation, result cache and task scheduling. Specifically, it is used to load the target remote sensing data from the distributed storage system into the data block in the target tile according to the identifier of the distributed target tile, and execute the processing logic on the data block to obtain the processing result of the data block in the target tile.
[0052] The access node is also used to output the remote sensing data processing results within the area, and the remote sensing data processing results within the area include the processing results of the data blocks in each target tile.
[0053] It should be noted that the distributed processing system of remote sensing data provides a rich data model, including predefined basic operation units, basic functions, system functions implemented based on basic functions, predefined data types, etc.
[0054] The predefined basic operation units are basic operations used in the remote sensing data processing process, including but not limited to reading (which can be expressed as OpRead), expression (which can be expressed as OpExpression), updating (which can be expressed as OpUpdateMask), and reducing the median (which can be expressed as OpReduceMedian). The expression operation includes operations of various expressions, including but not limited to comparison for equality, AND calculation, summation, and difference.
[0055] Basic functions are system functions provided by the system to implement basic operation units. Basic functions have corresponding basic operation units. It should be noted that for basic functions with different input data, different basic operation units can be encapsulated. For example, for a basic function that compares whether an input parameter is not equal to an input parameter, for different input parameters 3 and 10, comparing whether a parameter value of remote sensing data is not equal to 3 and comparing whether a parameter value of remote sensing data is not equal to 10 can be encapsulated into two different expression operations (i.e., two basic operation units): OpExpression'0'! = 3, OpExpression'0'! = 10.
[0056] Based on the defined basic functions, functions with more complex computing logic can be coded using basic functions. The distributed processing system for remote sensing data also provides system functions based on basic functions, thereby providing rich functional capabilities for users to choose and use.
[0057] Predefined data types are definitions of some data types provided by the system, including definitions of tile types, definitions of tile corresponding execution parameter arrays, definitions of single remote sensing images, definitions of image sets containing multiple remote sensing images, and definitions of other commonly used data types. Based on the standardized predefined data types provided by the system, it is convenient for users to program complex remote sensing data processing logic and to expand new functions based on tile types.
[0058] In addition, the distributed processing system for remote sensing data also provides basic functions, system function parameters, implemented processing logic, calling methods and other information for users to choose and use. Users can directly choose to use the provided system functions to process remote sensing data, or users can code and implement their desired remote sensing data processing logic based on the basic functions and system functions provided by the system.
[0059] In specific applications, the distributed processing system of remote sensing data provides a visual interface through the terminal device. Through the visual interface, the user can set the processing logic to be executed, the identification information of the remote sensing data to be processed, and the area to be calculated, and send an operation instruction to the terminal device through the visual interface. In response to the operation instruction, the terminal device sends a remote sensing data processing request to the cloud server. The remote sensing data processing request includes the area to be calculated, the processing logic to be executed, and the identification of the target remote sensing data to be processed. The cloud server receives the remote sensing data processing request, obtains the area to be calculated, the processing logic to be executed, and the identification of the target remote sensing data to be processed, and executes the processing logic on the target remote sensing data within the area to be calculated, obtains the remote sensing data processing result, and returns the remote sensing data processing result to the terminal device. The terminal device displays the remote sensing data processing result within the area on the visual interface.
[0060] like Figure 2 As shown, in the distributed processing system of remote sensing data provided by this embodiment, the computing and storage of the cloud server are separated, and defined data types, basic operation units, basic functions, system functions and other data models are provided, so that function expansion becomes easy, and data is distributed in the distributed storage system as tile types, and new functions only need to process and return tile type data. The defined data types, basic functions, system functions, etc. provided can be displayed in a visual interface, and users can directly reference the provided data model to realize the coding of remote sensing data processing logic. The cloud server can be deployed in a cloud native environment, and the access nodes, computing nodes and storage nodes are separated. Each type of node can automatically detect the system load and can expand and shrink capacity independently.
[0061] Figure 3 A flow chart of a method for distributed processing of remote sensing data provided by an exemplary embodiment of the present application is as follows: Figure 3 As shown, the method flow of realizing distributed processing of remote sensing data based on the remote sensing data distributed processing system is as follows:
[0062] Step S301: The terminal device provides a visualization interface, where the visualization interface is used to determine the processing logic to be executed, the identification information of the remote sensing data to be processed, and the area range to be calculated.
[0063] In this embodiment, the distributed processing system of remote sensing data provides a visualization interface, and displays the visualization interface through a client on the terminal device.
[0064] For example, Figure 4 A schematic diagram of a visualization interface provided in this embodiment, such as Figure 4 As shown, the visualization interface may include a coding area, a function list area, and a data visualization area. The coding area is an area for writing program codes for the processing logic of remote sensing data. Objects and functions can be defined using the syntax of programming languages such as Python to implement the processing logic of remote sensing data. The function list area displays the basic functions and system functions defined by the system for users to use when coding remote sensing data processing logic. The data visualization area provides a map component, through which the area to be calculated can be specified, and the processing results of remote sensing data can be displayed on the map component.
[0065] In actual applications, based on the visual interface provided by the system, users can use the predefined data types, basic functions, system functions, etc. provided by the system to encode and implement the remote sensing data processing logic they want to implement, and specify the target remote sensing data to be processed. Among them, the target remote sensing data can be a remote sensing image or a collection of multiple remote sensing images. The system provides remote sensing data from multiple sources and supports users to search and specify the target remote sensing data they want to process.
[0066] Based on the map component in the visualization interface provided by the system, the user can adjust the display area currently displayed in the map component by dragging, zooming, and expanding. In response to the adjustment operation of the display area in the map component, the terminal device adjusts the resolution and / or center point position of the display area. Further, after the adjustment is completed, the terminal device determines the area range to be calculated, as well as the width and height of the area range according to the resolution and / or center point position of the display area.
[0067] Optionally, the terminal device may use the entire display area as the calculation area, use the coordinate range of the display area as the area range to be calculated, and use the width and height of the display area as the width and height of the area range.
[0068] Optionally, the end-side device divides the display area into multiple grids, uses one grid as a calculation area, uses the coordinate range of each grid as the area range to be calculated, and uses the width and height of the grid area as the width and height of the corresponding area range. For any grid, the end-side device sends a remote sensing data processing request to the cloud server, which includes the coordinate range, width, and height of the grid. For the multiple divided grids, the end-side device sends multiple remote sensing data processing requests to the cloud server. Different remote sensing data processing requests are independent of each other, and the cloud server can process different remote sensing data processing requests in parallel.
[0069] Step S302: In response to the running instruction, the terminal device sends a remote sensing data processing request to the cloud server, where the remote sensing data processing request includes the area range to be calculated, the processing logic to be executed, and the identifier of the target remote sensing data to be processed.
[0070] like Figure 4 As shown, the visualization interface may also provide operation controls, such as a "Run" control, and a run instruction may be issued by triggering the "Run" control.
[0071] In response to the running instruction, the terminal device obtains the content edited by the user through the visual interface, and sends a remote sensing data processing request to the cloud server, which carries the processing logic to be executed, the identification information of the remote sensing data to be processed, and the area range to be calculated.
[0072] Step S303: The access node of the cloud server responds to the remote sensing data processing request, obtains the area range to be calculated, the processing logic to be executed, and the identifier of the target remote sensing data to be processed.
[0073] like Figure 2 As shown, the cloud server adopts a distributed system architecture and sets multiple access nodes. The cloud server processes remote sensing data processing requests from one or more end-side devices in parallel through multiple access nodes. Each remote sensing data processing request is independent of each other and processed in parallel.
[0074] Exemplarily, the cloud server may also include a request scheduling node, which allocates remote sensing data processing requests to multiple access nodes based on a load balancing strategy of remote sensing data processing requests of each access node load. Alternatively, each access node adopts a polling method to sequentially receive remote sensing data processing requests, and the scheduling method of multiple access nodes is not specifically limited here.
[0075] Step S304: The access node distributes the identifiers of the target tiles falling within the area range to the computing nodes based on the tiles divided into the map space, and sends the identifiers and processing logic of the target remote sensing data to the computing nodes.
[0076] In this embodiment, the access node divides the map space (global map space) into tiles according to their coordinate ranges, takes tiles that fall within (or partially fall within) the region range as target tiles, takes the target tiles as unit regions for remote sensing data processing, distributes the identifiers of the target tiles to each computing node, and sends the identifiers of the target remote sensing data and the processing logic to be executed to the computing nodes. Each computing node is responsible for executing the processing logic on the remote sensing data blocks within the target tiles of the distributed target tile identifiers.
[0077] Step S305 , the computing node loads the target remote sensing data into the data block in the target tile from the distributed storage system according to the identifier of the distributed target tile, and executes processing logic on the data block to obtain the processing result of the data block in the target tile.
[0078] Each computing node loads the target remote sensing data into the data block in the target tile from the distributed storage system according to the identifier of the distributed target tile, and executes the processing logic on the data block to obtain the processing result of the data block in the target tile. The processing results of the data blocks of each target tile in the region constitute the remote sensing data processing results in the entire region.
[0079] In an alternative embodiment, if Figure 2 As shown, the cloud server also includes a metadata service node. The metadata service node is mainly used to generate and store metadata information and provide the metadata information to the access node and the computing node.
[0080] The metadata information includes the offset of the tile in the map space in each remote sensing data. The metadata service node is used to determine the offset of the tile in the map space in each remote sensing data. The computing node loads the data block of the target remote sensing data falling into the target tile from the distributed storage system, which can be implemented in the following way:
[0081] The computing node obtains the offset of the target tile in the target remote sensing data from the meta information service node according to the identifier of the target tile. Further, the computing node loads the data block falling into the target tile from the target remote sensing data stored in the distributed storage system into the memory according to the offset of the target tile in the target remote sensing data.
[0082] Exemplarily, the target remote sensing data may include one or more remote sensing images (i.e., target remote sensing images). According to the offset of the target tile in each target remote sensing image, the computing node may determine the starting position of the data block of the target tile in each target remote sensing image, read the data block of the tile size from the starting position in each target remote sensing image in the distributed storage system, and load it into the memory, thereby completing the loading of the data block of each target remote sensing image that falls into the target tile.
[0083] In an optional embodiment, the distributed storage system stores a raster pyramid of remote sensing data, and the raster data at different layers in the raster pyramid are remote sensing data of different resolutions.
[0084] When loading the data block of the target remote sensing data falling into the target tile from the distributed storage system, the computing node loads the data block of the target remote sensing data falling into the target tile from the remote sensing data of the corresponding resolution layer in the grid pyramid according to the resolution of the regional range.
[0085] The resolution of the area range refers to the resolution of the display area after adjustment in the visualization interface.
[0086] Specifically, according to the resolution of the regional scope and the resolution of the raster data of each layer of the raster pyramid, if it is determined that there is a layer in the raster pyramid that is consistent with the resolution of the regional scope, the layer that is consistent with the resolution of the regional scope is used as the corresponding layer of the resolution of the regional scope. If there is no layer that is consistent with the resolution of the regional scope, the layer with the resolution of the regional scope and a small difference with the resolution of the regional scope is used as the corresponding layer of the resolution of the regional scope. From the remote sensing data of the corresponding layer of the resolution of the regional scope in the raster pyramid, the data block of the target remote sensing data falling into the target tile is loaded.
[0087] Optionally, if the resolution of the loaded data block is inconsistent with the resolution of the regional range, the computing node can resample the loaded data block according to the resolution of the regional range so that the resolution of the data block is consistent with the resolution of the regional range and store it in the memory. In addition, based on the parsing result of the function AST, the read remote sensing data can also be processed by regional clipping, reprojection, etc.
[0088] Optionally, the resolution of the region range can be determined in the following manner: the remote sensing data processing request also includes the height and width of the region range. The access node sends the height and width of the region range to the computing node that has distributed the identifier of the target tile. The computing node determines the resolution of the region range based on the region range and the height and width of the region range.
[0089] Optionally, the resolution of the area range can be calculated by the terminal device according to the coordinate range, width and height of the display area, and carried in the remote sensing data processing request and sent to the cloud server.
[0090] After loading the target remote sensing data from the distributed storage system into the data block in the target tile, the computing node executes the processing logic on the data in the target tile to obtain the processing result of the data block in the target tile. Through the distributed system architecture, multiple computing nodes can load and execute processing logic on the data blocks in each target tile in parallel, which improves the efficiency of remote sensing data processing.
[0091] Step S306: The access node outputs the remote sensing data processing result within the area to the terminal device, where the remote sensing data processing result within the area includes the processing result of the data blocks in each target tile.
[0092] In this embodiment, the computing node returns the processing results of the data blocks in the target tiles within the regional range to the access node. The access node aggregates the processing results of the databases in the target tiles within the regional range sent by each computing node, obtains the remote sensing data processing results within the regional range, and outputs the remote sensing data processing results within the regional range to the terminal device.
[0093] Step S307: The terminal device displays the remote sensing data processing results within the regional range on a visualization interface.
[0094] The end-side device receives the remote sensing data processing results within the regional range sent by the cloud server, and displays the remote sensing data processing results within the regional range on the visualization interface.
[0095] Specifically, based on Figure 4 In the visualization interface shown, the terminal device displays the remote sensing data processing results within the regional range on the map component provided in the data visualization area on the visualization interface.
[0096] The solution of this embodiment is based on the distributed processing system of remote sensing data. The cloud server adopts a distributed system architecture. Multiple access nodes process remote sensing data processing requests from one or more end-side devices in parallel. Based on each request, the target remote sensing data within the area to be calculated is divided into data blocks of multiple target tiles. The data blocks of each target tile are distributed to multiple computing nodes. Each computing node performs the processing of the data blocks of the target remote sensing data falling into the target tile. With the data block of the tile as the processing unit, multiple computing nodes concurrently perform the processing of the data blocks of each target tile, which improves the efficiency of remote sensing data processing and shortens the response time. In addition, in the distributed system architecture of the cloud server, the access nodes, computing nodes and distributed storage system are separated and have the ability to expand and shrink independently.
[0097] Figure 5 A flow chart of a method for distributed processing of remote sensing data provided for another exemplary embodiment of the present application. In an optional embodiment, in step S302, the terminal device responds to the running instruction and sends a remote sensing data processing request to the cloud server, which is specifically implemented in the following manner: the terminal device responds to the running instruction and converts the processing logic within the coding area into a function abstract syntax tree, which contains identification information of the remote sensing data; and sends a remote sensing data processing request to the cloud server, where the remote sensing data processing request contains the area range to be calculated and the function abstract syntax tree.
[0098] Furthermore, in step S303, the access node of the cloud server obtains the processing logic to be executed and the identification of the target remote sensing data to be processed, which is specifically implemented in the following manner: the access node receives the function abstract syntax tree of the processing logic to be executed, and the function abstract syntax tree includes the identification of the target remote sensing data to be processed.
[0099] Furthermore, in step S304, the access node sends the identification and processing logic of the target remote sensing data to the computing node, which is specifically implemented in the following manner: the access node parses the function abstract syntax tree to determine the basic operation units to be executed, as well as the hierarchical information and dependency relationships of the basic operation units; the access node sends the basic operation units and dependency relationships of each level to the computing node.
[0100] Furthermore, in step S305, the computing node loads the target remote sensing data from the distributed storage system into the data block within the target tile according to the identifier of the distributed target tile, and executes processing logic on the data block to obtain the processing result of the data block within the target tile. It is specifically implemented in the following way: the computing node executes the basic operation units of the first level in parallel according to the identifier of the distributed target tile, and loads the target remote sensing data from the distributed storage system into the data block within the target tile; and executes the basic operation units of the second and lower levels layer by layer according to the dependency relationship, and executes the basic operation units of the same level in parallel to execute processing logic on the data block to obtain the processing result of the data block.
[0101] like Figure 5 As shown, the method flow of realizing distributed processing of remote sensing data based on the remote sensing data distributed processing system is as follows:
[0102] Step S501: The terminal device provides a visualization interface, where the visualization interface is used to determine the processing logic to be executed, the identification information of the remote sensing data to be processed, and the area range to be calculated.
[0103] This step is implemented in the same manner as the aforementioned step S301. For details, please refer to the relevant contents of the aforementioned embodiment and will not be repeated here.
[0104] Step S502: In response to the run instruction, the terminal device converts the processing logic in the coding area into a function abstract syntax tree, where the function abstract syntax tree includes identification information of the remote sensing data.
[0105] like Figure 4 As shown, the visualization interface may also provide operation controls, such as a "Run" control, and a run instruction may be issued by triggering the "Run" control.
[0106] In response to the run instruction, the terminal device obtains the processing logic (program code) to be executed that is edited by the user in the coding area in the visual interface, and converts the processing logic in the coding area into a function abstract syntax tree, which contains the identification information of the remote sensing data.
[0107] Specifically, the end-side device can run the software development kit (SDK) provided by the distributed processing system of remote sensing data to convert the processing logic in the coding area into a function abstract syntax tree. Based on the known basic functions and grammatical specifications of the processing logic (code), an SDK for building a function abstract syntax tree can be implemented. Specifically, the SDK for building an abstract syntax tree of the corresponding programming language can be used, which is not specifically limited here.
[0108] Among them, the function abstract syntax tree (AST) is a directed acyclic graph (DAG) that describes the calling relationship (dependency relationship) between basic functions used in the processing logic.
[0109] For example, Figure 6 A schematic diagram of the structure of a function abstract syntax tree provided in this embodiment, Figure 6 The box nodes in the figure are basic functions for remote sensing data processing. The arrows represent the information flow and dependency between the basic functions. The output of the node at the beginning of the arrow is used as the input of the node at the end of the arrow. Figure 6 The function abstract syntax tree shown includes the following basic functions: ImageCollention.load, Image.neq(3), Image.neq(10), Image.and, Image.updateMask, and reduce.median. Among them, ImageCollention.load is used to load multiple remote sensing images (corresponding to Image) in the image collection specified by the input parameter. Image.neq(3) is used to compare whether the pixel value in the remote sensing image is not equal to 3. The pixel value that is not equal to 3 is set to 1, and the pixel value that is equal to 3 is set to 0. Image.neq(10) is used to compare whether the pixel value in the remote sensing image is not equal to 10. The pixel value that is not equal to 10 is set to 1, and the pixel value that is equal to 10 is set to 0. Image.and is used to perform an AND operation on two pixel values at the same pixel position in the two input images. If both pixel values are non-zero, the pixel value is set to 1, otherwise the pixel value is set to 0. Image.updateMask is used to perform a mask operation on the remote sensing image. The loop node indicates that the following processing in the curly braces is executed cyclically for each remote sensing image. reduce.median is used to perform a reduce operation on the results of partial processing in the loop for multiple remote sensing images in the specified image set. For example, the reduce operation can be to calculate the median. The input parameter of ImageCollention.load in the function abstract syntax tree specifies multiple remote sensing images in the image set, that is, the target remote sensing images to be processed. The input parameter can be the identifier of the image set, or multiple remote sensing image identifiers.
[0110] Step S503: The terminal device sends a remote sensing data processing request to the cloud server, where the remote sensing data processing request includes the area range to be calculated and the function abstract syntax tree.
[0111] In this embodiment, the remote sensing data processing request sent by the terminal device to the cloud server includes the area range to be calculated and the function abstract syntax tree, where the function abstract syntax tree includes identification information of the remote sensing data.
[0112] Step S504: The access node responds to the remote sensing data processing request and obtains the area range to be calculated and the function abstract syntax tree.
[0113] like Figure 2 As shown, the cloud server adopts a distributed system architecture and sets multiple access nodes. The cloud server processes remote sensing data processing requests from one or more end-side devices in parallel through multiple access nodes. Each remote sensing data processing request is independent of each other and processed in parallel.
[0114] Exemplarily, the cloud server may also include a request scheduling node, which allocates remote sensing data processing requests to multiple access nodes based on a load balancing strategy of remote sensing data processing requests of each access node load. Alternatively, each access node adopts a polling method to sequentially receive remote sensing data processing requests, and the scheduling method of multiple access nodes is not specifically limited here.
[0115] The access node responds to the remote sensing data processing request and extracts the area range to be calculated and the function abstract syntax tree from the remote sensing data processing request.
[0116] Step S505: The access node parses the function abstract syntax tree to determine the basic operation units to be executed, as well as the hierarchical information and dependency relationships of the basic operation units.
[0117] In this embodiment, the access node parses the function abstract syntax tree, and parses the basic functions for remote sensing data contained in the function abstract syntax tree into corresponding basic operation units; according to the dependency relationship between the basic functions, the dependency relationship and hierarchical information of the basic operation units are determined, wherein the input of any basic operation unit comes from the output of the basic operation unit of the previous layer, and there is no dependency relationship between the basic operation units at the same level.
[0118] Specifically, according to the dependency relationship between basic functions, when determining the dependency relationship and hierarchical information of the basic operation unit, the read operation in the basic operation unit is placed in the first level. By placing multiple read operations contained in the basic operation unit into the first level and centrally executing multiple read operations of the first level, one-time reading of remote sensing data is achieved, thereby improving data reading performance and maximizing reading performance.
[0119] In addition, for the basic functions that are executed cyclically in the function abstract syntax tree, each execution of the basic function can be encapsulated as a basic operation unit, and multiple basic operation units encapsulated in multiple cycles of the same basic function can be obtained. There is no dependency between the multiple basic operation units. They are put into the same level and executed in batches and in parallel when the basic operation units of this level are executed. This can improve the processing speed of remote sensing data and shorten the response time.
[0120] For example, based on Figure 6 The function abstract syntax tree shown in the figure can be determined as follows Figure 7 The operational layer shown, assuming Figure 6 In the function abstract syntax tree shown in the figure, the image collection specified by the input parameter of ImageCollention.load contains 5 remote sensing images. Then the processing in the loop is performed once for each of the 5 remote sensing images, that is, the processing in the loop is performed 5 times in a loop. For any basic function contained in the loop, when acting on the data of 5 different remote sensing images, it is encapsulated into 5 basic operation units. Figure 7 As shown in the figure, based on ImageCollention.load, five remote sensing images are loaded, which can be encapsulated into five read operations OpRead and placed in the first level. The outputs of the five read operations OpRead are the five loaded remote sensing images. Taking any of the remote sensing images as the processing object, execute Image.neq(3) and Image.neq(10), respectively encapsulated into OpExpression'0'! = 3, OpExpression'0'! = 10 basic operation units, and placed in the second level. The processing results of Image.neq(3) and Image.neq(10) are used as the input of Image.and, and Image.and is encapsulated as the OpExpressionAnd basic operation unit, which is placed in the third level, and the outputs of OpExpression'0'! = 3, OpExpression'0'! = 10 are used as the input of OpExpression And. The processing result of Image.and is used as the input of Image.updateMask, and Image.updateMask is encapsulated as the OpUpdateMask basic operation unit and placed in the fourth level. The output of OpExpression And is used as the input of OpUpdateMask. The same basic function that acts on the five remote sensing images is encapsulated into five basic operation units of the same type and placed in the same level. The five processing results obtained after the loop processing of the five remote sensing images are used as the input of reduce.median, and reduce.median is encapsulated as the OpReducemedian basic operation unit and placed in the fifth level. The output of each Image.updateMask in the previous level is used as the output of OpReducemedian. Figure 7 Only the Figure 6 The illustrated function abstract syntax tree shows the hierarchical information of the basic operation units, but does not show the dependency relationship between the basic operation units.
[0121] Optionally, after the access node parses the function abstract syntax tree and determines the basic operation units to be executed, as well as the hierarchical information and dependencies of the basic operation units, the access node can also determine the length and pre-allocation information of the execution parameter array based on the basic operation units to be executed, as well as the hierarchical information and dependencies of the basic operation units, and store the length and pre-allocation information of the execution parameter array to the metadata service node.
[0122] The execution parameter array is used to store the input data and output data of the basic operation unit. The pre-allocation information includes the storage position of the input data and output data of each basic operation unit in the execution parameter array.
[0123] Specifically, according to the hierarchical information and dependency relationship of the basic operation units to be executed, starting from the basic operation units of the first level, a position is allocated in turn in the execution parameter array for the output data of each read operation. For the basic operation units of the next level, their input data must come from the output data of one or more basic operation units of the previous level, so that the position corresponding to the input data of the basic operation unit can be determined, and further, the position is allocated for the output data of the basic operation unit in the allocatable position of the execution parameter array. For the position corresponding to the input data of the basic operation unit in the current level (that is, the first position), if the output of the basic operation unit allocated with the first position is not continued to be used as input by the basic operation unit of the lower level, the first position is re-assigned as the allocatable position. Continue to allocate positions for the basic operation units of the next level. After pre-allocating positions for the basic operation units of each level, the number of positions required to execute the basic operation unit, that is, the length of the execution parameter array, can be determined according to the position allocation result.
[0124] Optionally, when allocating positions for the output data of the basic operation unit from the allocatable positions of the execution parameter array, positions with smaller array subscripts may be allocated preferentially according to the size of the array subscript, and positions may be pre-allocated reasonably to shorten the length of the execution parameter array.
[0125] Optionally, when allocating positions for the output data of the basic operation unit from the allocable positions of the execution parameter array, unallocated positions may be preferentially selected until the number of allocated positions reaches a preset number, and positions with smaller array indexes are preferentially allocated.
[0126] In this embodiment, the access node can cache the basic operation units to be executed based on the function abstract syntax tree, as well as the hierarchical information and dependency relationships of the basic operation units. When the same function abstract syntax tree is subsequently processed, the corresponding basic operation units to be executed, as well as the hierarchical information and dependency relationships of the basic operation units can be directly obtained from the cache, thereby avoiding repeated interpretation of the same function abstract syntax tree, improving the efficiency of remote sensing data processing, and shortening the response time.
[0127] In addition, the access node can also cache metadata information obtained through time-consuming calculations, such as metadata information of remote sensing data files determined by the identifier of a given target remote sensing image, including but not limited to the storage address of the remote sensing data file, the latitude and longitude of the file, resolution, the number of remote sensing images contained, and other attribute information. When the same identifier is subsequently processed, the metadata information of the remote sensing data file corresponding to the identifier can be directly obtained from the cache without recalculation, which can improve the efficiency of remote sensing data processing and shorten the response time.
[0128] Step S506: The access node sends basic operation units and dependencies of each level to the computing node.
[0129] Step S507: The access node distributes the identifiers of the target tiles that fall within the area range to the computing nodes based on the tiles divided into the map space.
[0130] The above step S507 is executed after step S504, and can be executed in parallel with steps S505-S506. The execution order of step S507 and steps S505-S506 is not specifically limited. Figure 5 In the example, step S507 is performed before step S506.
[0131] In this embodiment, the access node divides the map space (global map space) into tiles according to their coordinate ranges, takes tiles that fall within (or partially fall within) the region range as target tiles, takes the target tiles as unit regions for remote sensing data processing, distributes the identifiers of the target tiles to each computing node, and sends the identifiers of the target remote sensing data and the processing logic to be executed to the computing nodes. Each computing node is responsible for executing the processing logic on the remote sensing data blocks within the target tiles of the distributed target tile identifiers.
[0132] Step S508 : The computing node centrally executes the basic operation unit of the first level according to the identifier of the distributed target tile, and loads the data block of the target remote sensing data falling into the target tile from the distributed storage system.
[0133] In this step, the computing node reads the target remote sensing data blocks falling into the target tile from the distributed storage system at one time by centrally executing multiple read operations (basic operation units) of the first level according to the identifier of the distributed target tile, thereby realizing one-time reading of remote sensing data, improving data reading performance, and maximizing reading performance.
[0134] In an optional embodiment, the meta information service node is also used to generate and store the offset of the tile in the map space in each remote sensing data. In this step, the computing node obtains the offset of the target tile in the target remote sensing data from the meta information service node according to the identifier of the target tile. Further, the computing node centrally executes the first-level read operation (basic operation unit), and according to the offset of the target tile in the target remote sensing data, loads the data block falling into the target tile from the target remote sensing data stored in the distributed storage system into the memory.
[0135] Exemplarily, the target remote sensing data may include one or more remote sensing images (i.e., target remote sensing images). According to the offset of the target tile in each target remote sensing image, the computing node may determine the starting position of the data block of the target tile in each target remote sensing image, read the data block of the tile size from the starting position in each target remote sensing image in the distributed storage system, and load it into the memory, thereby completing the loading of the data block of each target remote sensing image that falls into the target tile.
[0136] In an optional embodiment, the distributed storage system stores a raster pyramid of remote sensing data, and the raster data at different layers in the raster pyramid are remote sensing data of different resolutions.
[0137] When loading the data block of the target remote sensing data falling into the target tile from the distributed storage system, the computing node loads the data block of the target remote sensing data falling into the target tile from the remote sensing data of the layer corresponding to the resolution in the raster pyramid according to the resolution of the regional range. The resolution of the regional range refers to the resolution of the display area after adjustment in the visualization interface.
[0138] Specifically, according to the resolution of the regional scope and the resolution of the raster data of each layer of the raster pyramid, if it is determined that there is a layer in the raster pyramid that is consistent with the resolution of the regional scope, the layer that is consistent with the resolution of the regional scope is used as the corresponding layer of the resolution of the regional scope. If there is no layer that is consistent with the resolution of the regional scope, the layer with the resolution of the regional scope and a small difference with the resolution of the regional scope is used as the corresponding layer of the resolution of the regional scope. From the remote sensing data of the corresponding layer of the resolution of the regional scope in the raster pyramid, the data block of the target remote sensing data falling into the target tile is loaded.
[0139] Optionally, if the resolution of the loaded data block is inconsistent with the resolution of the area range, the computing node can resample the loaded data block according to the resolution of the area range so that the resolution of the data block is consistent with the resolution of the area range, and store it in the memory.
[0140] Optionally, the resolution of the region range can be determined in the following manner: the remote sensing data processing request also includes the height and width of the region range. The access node sends the height and width of the region range to the computing node that has distributed the identifier of the target tile. The computing node determines the resolution of the region range based on the region range and the height and width of the region range.
[0141] For example, Figure 8 The schematic diagram of implementing data block loading in a tile based on a grid pyramid provided in this embodiment is as follows: Figure 8 As shown in the figure, the remote sensing data stored in the distributed storage system is stored in the form of a raster pyramid. Different layers in the raster pyramid contain raster data of different resolutions. The calculation area given by the user is divided into grids to obtain multiple grids (four grids are used as an example in the figure). Figure 8 In the figure, the latitude and longitude of the upper left corner and the lower right corner of each grid are marked on each grid (the latitude and longitude values in the figure are only used as examples, and the specific values are random examples), and the area range of each grid can be determined. Based on the coordinate range of the grid, the grid is tiled, that is, the target tile falling within the area range of the grid is determined. The identifier of the target tile (such as Figure 8 The (0,0)(0,1)(0,2)…(2,2)) shown in the figure are distributed to multiple computing nodes. The computing node is responsible for reading the data blocks in the target tile of the target remote sensing data in the level corresponding to the grid resolution from the grid pyramid according to the grid resolution. The target remote sensing data is stored in the form of data files, and each data file contains data blocks falling into different tiles.
[0142] In addition, the resolution of the area range can be calculated by the terminal device based on the coordinate range, width and height of the display area, and carried in the remote sensing data processing request and sent to the cloud server.
[0143] After loading the target remote sensing data from the distributed storage system into the data block in the target tile, the computing node executes the processing logic on the data in the target tile to obtain the processing result of the data block in the target tile. Through the distributed system architecture, multiple computing nodes can load and execute processing logic on the data blocks in each target tile in parallel, which improves the efficiency of remote sensing data processing.
[0144] Step S509: The computing node executes the basic operation units of the second and lower levels layer by layer according to the dependency relationship, and executes the basic operation units of the same level in parallel, so as to execute the processing logic on the data block and obtain the processing result of the data block.
[0145] In this embodiment, the first level is the top level. For basic operation units of the same level, computing nodes can execute in batches to improve execution efficiency. For example, for read operations of the first level, computing nodes can execute multiple read operations in a centralized manner and perform one-time reading to maximize read performance. For various computing operations of the second level, computing nodes can execute multiple computing operations in parallel to improve computing efficiency.
[0146] In an optional embodiment, after the computing node loads the target remote sensing data from the distributed storage system into the data block within the target tile, the computing node can obtain the length and pre-allocation information of the execution parameter array from the metadata service node, and construct the execution parameter array corresponding to the target tile.
[0147] Furthermore, when the computing node executes the basic operation unit on the data block of the target tile, the computing node determines the first position of the input data of the basic operation unit in the execution parameter array of the target tile, and the second position of the output data of the basic operation unit in the execution parameter array of the target tile according to the pre-allocated information. The computing node reads the input data of the basic operation unit from the first position in the execution parameter array of the target tile; and stores the output data obtained by executing the basic operation unit in the second position in the execution parameter array of the target tile. During the execution of the basic operation unit, the input data and output data of each basic operation unit are stored in the local execution parameter array, which avoids data exchange between multiple computing nodes and improves the efficiency of remote sensing data processing.
[0148] In this embodiment, the access node may send basic operation units to the computing node layer by layer. After the computing node executes one layer of basic operation units, the access node sends the basic operation units of the next layer to the computing node to ensure data correctness.
[0149] In addition, the access node can also send basic operation units of each layer to the computing node at one time. The computing node can start to execute the basic operation unit of the next layer only after executing the basic operation unit of the previous layer to ensure data correctness.
[0150] Step S510: The access node outputs the remote sensing data processing result within the area to the terminal device, where the remote sensing data processing result within the area includes the processing result of the data block in each target tile.
[0151] In this embodiment, the computing node returns the processing results of the data blocks in the target tiles within the regional range to the access node. The access node aggregates the processing results of the databases in the target tiles within the regional range sent by each computing node, obtains the remote sensing data processing results within the regional range, and outputs the remote sensing data processing results within the regional range to the terminal device.
[0152] Step S511: The terminal device displays the remote sensing data processing results within the regional range on a visualization interface.
[0153] The end-side device receives the remote sensing data processing results within the regional range sent by the cloud server, and displays the remote sensing data processing results within the regional range on the visualization interface.
[0154] Specifically, based on Figure 4 In the visualization interface shown, the terminal device displays the remote sensing data processing results within the regional range on the map component provided in the data visualization area on the visualization interface.
[0155] The scheme of this embodiment converts the processing logic written by the user into a function abstract syntax tree. The cloud server determines the basic operation unit to be executed, as well as the hierarchical information and dependency relationship of the basic operation unit by parsing the function abstract syntax tree. The input of any basic operation unit comes from the output of the basic operation unit of the previous layer, and there is no dependency relationship between the basic operation units at the same level; when executing the basic operation units of each level, the read operation of the first level is executed centrally to realize the one-time reading of remote sensing data, improve the data reading performance, and maximize the reading performance; the basic operation units of the second and lower levels are executed layer by layer, and the basic operation units of the same level are executed in parallel. By reasonably arranging and scheduling the execution of the basic operation units of each layer, the processing speed of remote sensing data can be improved and the response time can be shortened. In addition, according to the basic operation unit to be executed, as well as the hierarchical information and dependency relationship of the basic operation unit, the length and pre-allocation information of the execution parameter array are determined, and the position of the input and output data of each basic operation unit in the remote sensing data processing process is pre-allocated, avoiding data exchange between multiple computing nodes, further improving the remote sensing data processing efficiency, and shortening the response time.
[0156] Fig. 9 This is a schematic diagram of the structure of a terminal device provided in an embodiment of the present application. Fig. 9As shown, the terminal device includes: a memory 901 and a processor 902. The memory 901 is used to store computer-executable instructions and can be configured to store various other data to support operations on the terminal device. The processor 902 is connected to the memory 901 in communication and is used to execute the computer-executable instructions stored in the memory 901 to implement the technical solution executed by the terminal device in any of the above method embodiments. Its specific functions and technical effects that can be achieved are similar and will not be repeated here.
[0157] Optional, such as Fig. 9 As shown, the end-side device also includes: a firewall 903, a load balancer 904, a communication component 905, a power supply component 906 and other components. Fig. 9 Only some components are shown schematically, which does not mean that the terminal device only includes Fig. 9 Components shown.
[0158] An embodiment of the present application also provides a computer-readable storage medium, in which computer execution instructions are stored. When the computer execution instructions are executed by a processor, they are used to implement the solution provided by any of the above method embodiments. The specific functions and technical effects that can be achieved are not repeated here.
[0159] The embodiment of the present application also provides a computer program product, which includes: a computer program, the computer program is stored in a readable storage medium, at least one processor of the end-side device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the end-side device executes the solution provided by any of the above-mentioned method embodiments. The specific functions and technical effects that can be achieved are not repeated here. The embodiment of the present application provides a chip, including: a processing module and a communication interface, and the processing module can execute the technical solution of the end-side device in the aforementioned method embodiment. Optionally, the chip also includes a storage module (such as a memory), the storage module is used to store instructions, the processing module is used to execute the instructions stored in the storage module, and the execution of the instructions stored in the storage module enables the processing module to execute the technical solution provided by any of the aforementioned method embodiments.
[0160] The above storage may be an object storage service (OSS).
[0161] The above-mentioned memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.
[0162] The above-mentioned communication component is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as a mobile hotspot (WiFi), a second-generation mobile communication system (2G), a third-generation mobile communication system (3G), a fourth-generation mobile communication system (4G) / long-term evolution (LTE), a fifth-generation mobile communication system (5G) and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.
[0163] The power supply assembly provides power to various components of the device where the power supply assembly is located. The power supply assembly may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply assembly is located.
[0164] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0165] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0166] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0167] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0168] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0169] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0170] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0171] It should be noted that the user information (including but not limited to user device information, user attribute information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and provide corresponding operation entrances for users to choose to authorize or refuse.
[0172] In addition, in some of the processes described in the above embodiments and the accompanying drawings, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel, and are only used to distinguish between different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit "first" and "second" to different types. The meaning of "multiple" is more than two, unless otherwise clearly and specifically defined.
[0173] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and provide corresponding operation entrances for users to choose to authorize or refuse.
[0174] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.
[0175] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.
Claims
1. A distributed processing system for remote sensing data, characterized in that: include: Cloud Servers, The cloud server comprises: Distributed storage system for storing collected remote sensing data; The access node is used to respond to the remote sensing data processing request, obtain the area range to be calculated, the processing logic to be executed and the identification of the target remote sensing data to be processed, distribute the identification of the target tile falling within the area range to the computing node based on the tiles divided by the map space, and send the identification of the target remote sensing data and the processing logic to the computing node; the access node has the ability to automatically detect the system load and independently expand and shrink capacity; A computing node, used to load the data block of the target remote sensing data falling into the target tile from the distributed storage system according to the identifier of the distributed target tile, and execute the processing logic on the data block to obtain the processing result of the data block in the target tile; the computing node has the ability to expand and shrink capacity independently; the computing node at least includes a data loader, a cache, an operation execution engine and a scheduler; The access node is further used to output the remote sensing data processing result within the area, and the remote sensing data processing result within the area includes the processing result of the data block in each of the target tiles.
2. The system according to claim 1, characterized in that The access node obtains the processing logic to be executed and the identification of the target remote sensing data to be processed, including: The access node receives a function abstract syntax tree of a processing logic to be executed, wherein the function abstract syntax tree includes an identifier of target remote sensing data to be processed; The access node sends the identifier of the target remote sensing data and the processing logic to the computing node, including: The access node parses the function abstract syntax tree to determine the basic operation units to be executed, and the hierarchical information and dependency relationships of the basic operation units; The access node sends the basic operation units of each level and the dependency relationship to the computing node.
3. The system according to claim 2, characterized in that The access node parses the function abstract syntax tree to determine the basic operation unit to be executed, and the hierarchical information and dependency relationship of the basic operation unit, including: The access node parses the basic functions for remote sensing data contained in the function abstract syntax tree into corresponding basic operation units; According to the dependency relationship between the basic functions, the dependency relationship and hierarchical information of the basic operation units are determined, wherein the input of any basic operation unit comes from the output of the basic operation unit of the upper layer, and there is no dependency relationship between the basic operation units at the same level.
4. The system according to claim 3, characterized in that The step of determining the dependency relationship and hierarchical information of the basic operation unit according to the dependency relationship between the basic functions includes: The read operation in the basic operation unit is placed in the first level.
5. The system according to claim 3, characterized in that The computing node loads the data block of the target remote sensing data falling into the target tile from the distributed storage system according to the identifier of the distributed target tile, and executes the processing logic on the data block to obtain the processing result of the data block in the target tile, including: The computing node centrally executes the basic operation unit of the first level according to the identifier of the distributed target tile, and loads the data block of the target remote sensing data falling into the target tile from the distributed storage system; And according to the dependency relationship, the basic operation units of the second and lower levels are executed layer by layer, and the basic operation units of the same level are executed in parallel, so as to execute the processing logic on the data block and obtain the processing result of the data block.
6. The system according to any one of claims 3 to 5, characterized in that: After the access node parses the function abstract syntax tree to determine the basic operation units to be executed, and the hierarchical information and dependency relationships of the basic operation units, the method further includes: The access node determines the length and pre-allocation information of an execution parameter array according to the basic operation unit to be executed, and the hierarchical information and dependency relationship of the basic operation unit, wherein the execution parameter array is used to store input data and output data of the basic operation unit, and the pre-allocation information includes the storage position of the input data and output data of each basic operation unit in the execution parameter array; The access node stores the length of the execution parameter array and the pre-allocation information in the meta information service node.
7. The system according to claim 6, characterized in that After the computing node loads the data block of the target remote sensing data falling into the target tile from the distributed storage system, the method further includes: The computing node obtains the length and pre-allocation information of the execution parameter array from the meta information service node, and constructs the execution parameter array corresponding to the target tile; When the computing node executes the basic operation unit on the data block of the target tile, the computing node determines, according to the pre-allocation information, a first position of input data of the basic operation unit in the execution parameter array of the target tile and a second position of output data of the basic operation unit in the execution parameter array of the target tile, The computing node reads input data of the basic operation unit from a first position in the execution parameter array of the target tile; and stores output data obtained by executing the basic operation unit in a second position in the execution parameter array of the target tile.
8. The system according to claim 1, characterized in that Also includes: A meta information service node, used to determine the offset of the tile in the map space in each remote sensing data; The computing node loads the target remote sensing data from the distributed storage system into a data block of the target tile, including: The computing node obtains, from the meta information service node, an offset of the target tile in the target remote sensing data according to the identifier of the target tile; And according to the offset, the data block falling within the target tile is loaded into the memory from the target remote sensing data stored in the distributed storage system.
9. The system according to claim 1 or 8, characterized in that: The distributed storage system stores a grid pyramid of collected remote sensing data, wherein different layers of the grid pyramid include remote sensing data of different resolutions. The computing node loads the target remote sensing data from the distributed storage system into a data block within the target tile, including: The computing node loads the data block of the target remote sensing data falling within the target tile from the remote sensing data of the layer corresponding to the resolution in the grid pyramid according to the resolution of the regional range.
10. The system according to claim 9, characterized in that The remote sensing data processing request also includes the height and width of the area range. The access node is further used to send the height and width of the area range to the computing node that distributes the identifier of the target tile; The computing node is further used to determine the resolution of the area range according to the area range and the height and width of the area range.
11. A distributed processing system for remote sensing data, characterized in that: include: End-side equipment, The terminal device provides a visualization interface, wherein the visualization interface is used to determine the processing logic to be executed, the identification information of the remote sensing data to be processed, and the area range to be calculated; In response to the run instruction, a remote sensing data processing request is sent to the cloud server, wherein the remote sensing data processing request includes a region range to be calculated, a processing logic to be executed, and an identifier of the target remote sensing data to be processed; The access node of the cloud server distributes the identification of the target tile falling within the area to the computing node according to the remote sensing data processing request and based on the tiles divided by the map space, and sends the identification of the target remote sensing data and the processing logic to the computing node; the access node has the ability to automatically detect the system load, expand and shrink capacity independently; the computing node has the ability to expand and shrink capacity independently; the computing node includes at least a data loader, a cache, an operation execution engine and a scheduler; Receive the remote sensing data processing result within the area range sent by the cloud server, wherein the remote sensing data processing result is obtained by the computing node loading the data block of the target remote sensing data falling into the target tile from the distributed storage system according to the identifier of the target tile falling into the area range, and executing the processing logic on the data block, thereby obtaining the processing result of the data block within the target tile; The remote sensing data processing results within the area are displayed on the visualization interface.
12. The system according to claim 11, characterized in that The visual interface provides a coding area, and the coding area is used to edit the processing logic to be executed and specify the identification information of the remote sensing data to be processed; The visualization interface provides a map component. In response to an adjustment operation on a display area within the map component, the terminal device adjusts the resolution and / or center point position of the display area; based on the resolution and / or center point position of the display area, determines the area range to be calculated, as well as the width and height of the area range.
13. The system according to claim 12, characterized in that The terminal device sends a remote sensing data processing request to the cloud server in response to the running instruction, including: In response to the running instruction, the terminal device converts the processing logic in the coding area into a function abstract syntax tree, where the function abstract syntax tree includes identification information of the remote sensing data; A remote sensing data processing request is sent to a cloud server, wherein the remote sensing data processing request includes a region range to be calculated and the function abstract syntax tree.
14. The system according to claim 13, characterized in that The terminal device determines the area range to be calculated and the width and height of the area range according to the resolution and / or the center point position of the display area, including: The terminal device uses the coordinate range of the display area as the area range to be calculated, and uses the width and height of the display area as the width and height of the area range; or, The terminal device divides the display area into a plurality of grids, uses the coordinate range of each grid as the area range to be calculated, and uses the width and height of the grid area as the width and height of the corresponding area range.
