Cloud native geographic space analysis and calculation method and device

Abstract

The invention provides a cloud native geographic space analysis and calculation method and device, and belongs to the technical field of geographic space information. A cloud native geographic space analysis and calculation method comprises the following steps: constructing a geographic model library by packaging a heterogeneous geographic model and a running environment thereof into a standardized Docker mirror image to realize unified management and calling, and meanwhile, constructing a cloud native-based calculation environment to realize dynamic monitoring and elastic expansion of calculation resources. Coexistence and efficient calculation of heterogeneous models on the same platform are supported, and finally, through a geographic model calculation engine, a geographic model calculation task is planned and executed according to a workflow model, and task scheduling and resource elastic adjustment are carried out; according to the method, the micro-service architecture consisting of the geographic model library, the geographic model calculation environment and the geographic model calculation engine is adopted, so that coexistence, efficient calculation and flexible scheduling of heterogeneous models on the same platform are ensured, and the problem of model coexistence is solved.

Description

Technical Field

[0001] The present invention relates to the field of geospatial information technology, and in particular to a cloud-native geospatial analysis and computing method and device. Background Art

[0002] Geospatial analysis and calculation is a complex process, which requires organizing multiple basic geographic models into workflows according to the process of geographic events to achieve computational simulation of complex geographic event processes.

[0003] In practical applications, geographic models are often heterogeneous, which is manifested in differences in the development language, operating environment, dependency library, and API interface of geographic models. This leads to direct API calls between geographic models, and differences in operating environment such as dependency library version conflicts make it difficult for different geographic models to coexist and run in a unified environment, which ultimately makes it difficult to seamlessly integrate geographic models and flexibly build complex geospatial workflows to solve complex geospatial analysis problems. On the other hand, geospatial analysis and computing services are oriented to many users and diverse user needs, which requires solving the problem of integrated operation of heterogeneous basic geographic models, being able to flexibly assemble basic geographic models, and solving the problem of building complex geographic models. Therefore, in order to solve the problem of operation and scheduling of geospatial computing workflows, realize the elastic scaling problem of geospatial computing, and maximize the utilization of geospatial computing service resources to provide high-quality geospatial analysis and computing services, the present invention proposes a cloud-native geospatial analysis and computing method and device. Summary of the invention

[0004] The technical problem to be solved by the present invention is to overcome the deficiencies of the prior art and provide a cloud-native geospatial analysis computing method and device that can overcome the above problems or at least partially solve the above problems.

[0005] In order to solve the above technical problems, the basic concept of the technical solution adopted by the present invention is: a cloud-native geographic spatial analysis and calculation method, including: S1, building a geographic model library, encapsulating heterogeneous geographic models and their operating environments into standardized Docker images, and uniformly managing and calling the geographic models; S2, building a geographic model computing environment, realizing monitoring and elastic scaling of computing resources based on the cloud-native environment, so that heterogeneous geographic models can coexist, calculate and elastically scale on the same computing platform; S3, building a geographic model computing engine, building a geographic model calculation execution plan according to the geographic spatial analysis workflow model, sending geographic model calculation tasks to the geographic computing environment for execution, and scheduling and elastically scaling the geographic computing tasks.

[0006] Preferably, the management of the geographic model library includes: geographic model image construction, encapsulating the geographic model and its dependent environment, configuration files and other resources required for its operation into a standardized Docker image according to the geographic model image specification; geographic model warehousing, pushing the encapsulated geographic model image to the geographic model image library for unified management; geographic model library management includes: geographic model image library, used to manage encapsulated geographic model images; geographic model metadata construction, used to expand metadata information related to the model itself.

[0007] Preferably, the geographic model image specification includes an image construction specification, an image description specification and an image interface specification, wherein: the image construction specification is used to standardize the naming and version of the geographic model image; the image description specification defines the metadata of the image, including the image name, model name, model category, model encoding, function description, interface description, parameter description and operating environment; the image interface specification unifies the API interface definition of the geographic model so that the geographic model has a unified API calling method.

[0008] Preferably, the geographic model computing environment includes: a geographic computing environment, which is built on the basis of CPU, GPU and parallel computing frameworks MPI and OpenMP, and encapsulates the GDAL geographic data processing library, CUDA parallel computing platform and the specific computing environment of the PyTorch deep learning framework through Docker container technology to form a geographic computing environment mirror library; resource monitoring and elastic computing, collecting resource operation indicators of geographic model container instances, and monitoring resource utilization; when the resource utilization exceeds a preset threshold, the number of copies of the geographic model container instance is increased by using the elastic scaling mechanism of the cloud native environment.

[0009] Preferably, the geographic model computing engine includes: a computing job scheduler, which generates computing jobs, maintains a computing job queue, and schedules computing jobs to the geographic model computing environment to perform computing; a computing task planner, which performs execution planning for complex geographic models, converts complex geographic model workflow graphs into directed acyclic computing graphs composed of simple geographic models, and generates an execution queue; a computing task executor, which obtains computing tasks from the computing job scheduler, and starts a geographic model container instance on a cloud native environment to perform model computing.

[0010] The present invention also provides a cloud-native geographic spatial analysis and computing device for implementing the above-mentioned cloud-native geographic spatial analysis and computing method. The device includes: a geographic model library module, which is encapsulated according to the model image construction specification, encapsulates heterogeneous geographic models into standardized Docker images, and aggregates and uniformly manages them in the geographic model library; a geographic model computing environment module, which is used to execute the construction of the geographic model computing environment; and a geographic model computing engine module, which is used to execute the construction of the geographic model computing engine.

[0011] Preferably, the geographic model library module includes: a geographic model image construction unit, which encapsulates the dependent environment and configuration files required for the operation of the geographic model into a standardized Docker image; a geographic model warehousing unit, which pushes the constructed and packaged geographic model image to the image library for unified management; and a geographic model library management unit, which is used to uniformly manage the packaged geographic model image and metadata information related to the extended model.

[0012] Preferably, the geographic model computing environment module includes: a geographic computing environment unit, which integrates the computing resources of CPU, GPU, MPI and OpenMP, encapsulates the environment of GDAL, CUDA and PyTorch through Docker, and builds a standardized image library; a resource monitoring and elastic computing unit, which is responsible for collecting resource usage data of geographic model container instances and monitoring the operating indicators of resource utilization in real time.

[0013] Preferably, the geographic model computing engine module includes: a computing job scheduler unit, which generates jobs, maintains queues, and schedules jobs to a computing environment; a computing task planner unit, which plans the execution of complex geographic models and generates execution queues; and a computing task executor unit, which, after obtaining tasks from the scheduler, starts corresponding geographic model container instances on a cloud-native environment to perform calculations.

[0014] Preferably, the device also includes a geographic model elastic computing module, which dynamically adjusts the number and type of geographic model container instances according to the requirements of the computing tasks and the status of current computing resources.

[0015] After adopting the above technical scheme, the present invention has the following beneficial effects compared with the prior art: the present invention takes into account that in the traditional computing environment, due to the limitations of technical architecture, data format and algorithm design, different types of geographic spatial models are often difficult to achieve seamless connection and efficient calculation on the same platform, which not only increases the complexity and cost of data processing, but also limits the application scope and depth of geographic spatial analysis. Therefore, the present invention realizes efficient integration and unified management of heterogeneous geographic spatial models in the cloud by adopting a microservice architecture composed of a geographic model library, a geographic model computing environment, and a geographic model computing engine. This framework can support the access of multiple data formats and model structures, ensuring that different types of geographic spatial data can recognize and communicate with each other in the same computing environment, thereby solving the problem of model coexistence. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A schematic diagram of a process flow of a cloud-native geospatial analysis and calculation method provided by the present invention;

[0017] Figure 2A schematic diagram of the module functional structure of a cloud-native geospatial analysis and computing device provided by the present invention. DETAILED DESCRIPTION

[0018] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments so that those skilled in the art can implement the invention with reference to the description.

[0019] It should be understood that the terms such as “having”, “including” and “comprising” used herein do not exclude the existence or addition of one or more other elements or combinations thereof.

[0020] In the description of the present invention, the terms "lateral", "longitudinal", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside" and "outside" etc. to indicate directions or positional relationships are based on the directions or positional relationships shown in the accompanying drawings and are only for the convenience of describing the present invention and simplifying the description. They do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore should not be understood as a limitation on the present invention.

[0021] Reference Figure 1 and Figure 2 A cloud-native geospatial analysis and computing method includes building a geo-model library, encapsulating heterogeneous geo-models and their operating environments into standardized Docker images, uniformly managing and calling the geo-model library, building a geo-model computing environment again, and realizing monitoring and elastic scaling of computing resources based on the cloud-native environment, so that heterogeneous geo-models can coexist, compute, and elastically scale on the same computing platform. Finally, a geo-model computing engine is built, and a geo-model computing execution plan is constructed according to a geo-spatial analysis workflow model, and geo-model computing tasks are sent to the geo-computing environment for execution, and the geo-computing tasks are scheduled and elastically scaled.

[0022] Among them, the construction of the geographic model library includes three parts, namely, geographic model image construction, geographic model library management, and geographic model storage. In this embodiment, the geographic model image construction encapsulates three different types of geographic models, namely, the DBSCAN clustering model, the GDAL geographic data processing model, and the custom watershed simulation model. According to the image construction specification, a Dockerfile is created for each model, and the dependent environment, configuration files, and other resources required by the model are included. For example, the DBSCAN model requires a Python environment and a scikit-learn library, and the GDAL model requires a GDAL library and related dependencies. Then, the constructed geographic model image is pushed to a Docker image repository (such as Harbor) for unified management. At the same time, the metadata of each model is recorded in the geographic model metadata repository, including the model name, function description, interface description, parameter description, and operating environment. This information is crucial for the model caller in the geographic model calculation scheduling engine. The caller can perform accurate retrieval in the geographic model image library based on these description information, so as to quickly find and pull the required target model image. After that, the caller can deploy and execute these models in the cloud-native geographic computing environment.

[0023] It should be noted that in the process of storing geographic models, the geographic model image not only encapsulates the geographic model itself, but also includes its operating environment and calling interface, thus forming a standardized geographic model operating environment and a unified API interface calling method. In this way, the encapsulated geographic model will be connected to the geographic model library for unified management. Through Docker's push command, these geographic model images can be easily pushed to the image library, thereby completing the storage operation of the geographic model image.

[0024] The construction of the geographic model computing environment includes the geographic computing environment and resource monitoring and elastic computing. The geographic computing environment is built based on Kubernetes and configured with different types of computing nodes, including CPU nodes, GPU nodes, and nodes that support MPI and OpenMP. In order to meet the computing needs of different geographic models, corresponding computing environment images such as GDAL, CUDA, and PyTorch are installed on each node. Resource monitoring and elastic computing use the resource monitoring function of Kubernetes to collect the CPU, memory and other resource usage of the geographic model container instance in real time. When the resource usage exceeds the preset threshold, Kubernetes will automatically trigger the elastic scaling mechanism to increase the number of copies of the geographic model container instance to meet the computing needs.

[0025] The geographic model computing engine includes a computing job scheduler, a computing task planner, and a computing task executor. The computing job scheduler is responsible for receiving and managing new computing jobs, pushing these jobs to a computing job queue, and scheduling them according to the first-in-first-out principle. When a computing job is scheduled, the computing job scheduler submits it to the computing task planner for planning. The computing task planner implements the computing task planner for complex geographic models to decompose them into directed acyclic computing graphs consisting of simple geographic models. For example, a complex geographic model including DBSCAN clustering, GDAL data processing, and watershed simulation is decomposed into three sub-computing tasks, and the corresponding execution queues are generated. The computing task executor obtains computing tasks from the computing job scheduler and starts the corresponding geographic model container instance on the cloud native environment to perform computing. During the computing process, the geographic model container instance communicates and collaborates asynchronously through the message system to ensure the correctness and consistency of the computing results.

[0026] Finally, the geographic model elastic computing module dynamically adjusts the number and type of geographic model container instances according to the needs of computing tasks and the current status of computing resources. This elastic computing mechanism ensures that computing tasks can be executed efficiently while maximizing the use of available computing resources. When computing resources are insufficient, the elastic computing module will automatically increase the number or type of container instances to meet demand. When resources are in excess, the number of container instances will be reduced accordingly to save resources.

[0027] In summary, the cloud-native geospatial analysis computing framework proposed in the present invention can successfully encapsulate heterogeneous geographic models into standardized Docker images, and coexist and compute in the same computing environment. At the same time, the framework can also realize flexible construction of the computing process of complex geographic models and elastic scaling of computing, thereby improving the efficiency and accuracy of geospatial analysis computing.

[0028] The above embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which are equivalent modifications and improvements made to the above embodiments based on the essential technology of the present invention, and all of them belong to the protection scope of the present invention.

Claims

1. A cloud-native geospatial analysis and computing method, characterized in that: include: S1. Build a geographic model library, encapsulate heterogeneous geographic models and their operating environments into standardized Docker images, and uniformly manage and call the geographic models; S2. Build a geographic model computing environment, implement monitoring and elastic scaling of computing resources based on a cloud-native environment, and enable heterogeneous geographic models to coexist, compute, and elastically scale on the same computing platform; S3. Build a geographic model computing engine, build a geographic model computing execution plan based on the geographic spatial analysis workflow model, send the geographic model computing tasks to the geographic computing environment for execution, and schedule and elastically scale the geographic computing tasks.

2. A cloud-native geospatial analysis and calculation method according to claim 1, characterized in that: The management of the geographic model library includes: Geographic model image construction: according to the geographic model image specification, the geographic model and its dependent environment, configuration files and other resources required for its operation are encapsulated into a standardized Docker image; Geographic model storage: push the packaged geographic model image to the geographic model image library for unified management; Geographic model library management, including: Geographic model image library, used to manage packaged geographic model images; The metadata construction of geographic models is used to extend the metadata information related to the model itself.

3. A cloud-native geospatial analysis and calculation method according to claim 2, characterized in that: The geographic model image specification includes an image construction specification, an image description specification and an image interface specification, wherein: Image building specifications, used to standardize the naming and versioning of geographic model images; Image description specification, which defines the metadata of the image, including image name, model name, model category, model code, function description, interface description, parameter description and operating environment; Mirror interface specifications, unify the API interface definition of geographic models, and give geographic models a unified API calling method.

4. A cloud-native geospatial analysis and calculation method according to claim 1, characterized in that: The geographic model computing environment includes: The geographic computing environment is built on the basis of CPU, GPU and parallel computing frameworks MPI and OpenMP. Through Docker container technology, the GDAL geographic data processing library, CUDA parallel computing platform and the specific computing environment of the PyTorch deep learning framework are encapsulated to form a geographic computing environment image library; Resource monitoring and elastic computing: collect resource operation indicators of geographic model container instances and monitor resource utilization; When resource usage exceeds the preset threshold, the number of geographic model container instance replicas is increased using the cloud native environment's elastic scaling mechanism.

5. A cloud-native geospatial analysis and calculation method according to claim 1, characterized in that: The geographic model calculation engine includes: The computing job scheduler generates computing jobs, maintains the computing job queue, and schedules the computing jobs to the geographic model computing environment for computing execution; The computing task planner performs execution planning for complex geographic models, converts the workflow graph of complex geographic models into a directed acyclic computing graph consisting of simple geographic models, and generates an execution queue; The computing task executor obtains computing tasks from the computing job scheduler and starts the geographic model container instance in the cloud native environment to perform model computing.

6. A cloud-native geospatial analysis and computing device, used to implement a cloud-native geospatial analysis and computing method according to any one of claims 1 to 5, characterized in that: The device includes: The geographic model library module encapsulates heterogeneous geographic models into standardized Docker images according to the model image construction specifications, and aggregates and uniformly manages them in the geographic model library; A geographic model computing environment module is used to execute the construction of the geographic model computing environment; The geographic model calculation engine module is used to execute the construction of the geographic model calculation engine.

7. A cloud-native geospatial analysis computing device according to claim 6, characterized in that: The geographic model library module includes: The geographic model image building unit encapsulates the dependent environment and configuration files required for the operation of the geographic model into a standardized Docker image; The geographic model storage unit pushes the built and packaged geographic model image to the image library for unified management; The geographic model library management unit is used to uniformly manage the metadata information related to the encapsulated geographic model images and extended models.

8. A cloud-native geospatial analysis computing device according to claim 6, characterized in that: The geographic model computing environment module includes: The geographic computing environment unit integrates the computing resources of CPU, GPU, MPI and OpenMP, encapsulates the environment of GDAL, CUDA and PyTorch through Docker, and builds a standardized image library; The resource monitoring and elastic computing unit is responsible for collecting resource usage data of geographic model container instances and monitoring the operating indicators of resource usage in real time.

9. A cloud-native geospatial analysis computing device according to claim 6, characterized in that: The geographic model calculation engine module includes: The computing job scheduler unit generates jobs, maintains queues, and schedules jobs to the computing environment; The computational task planner unit plans the execution of complex geographic models and generates execution queues; The computing task executor unit, after obtaining the task from the scheduler, starts the corresponding geographic model container instance on the cloud native environment to perform the calculation.

10. A cloud-native geospatial analysis computing device according to any one of claims 6 to 9, characterized in that: The device also includes a geographic model elastic computing module, which dynamically adjusts the number and type of geographic model container instances according to the needs of computing tasks and the status of current computing resources.

Images