A weather prediction method based on componentized workflow orchestration and AI meteorological model

Abstract

A weather forecasting method based on component-based workflow orchestration and AI weather models is provided, including splitting the weather forecasting task into multiple subtasks; wherein each subtask corresponds to a standardized component in a standardized component library; step S2: orchestrating the workflow and generating a workflow description; wherein the workflow description is used to describe the combination of multiple subtasks that implement the weather forecasting task split in step S1 and their execution order; loading and parsing the workflow description generated in step S2 through a workflow engine, creating a corresponding execution graph, and scheduling the standardized components in the execution graph in sequence; formatting and outputting the output data through the output component of the last node of the execution graph.

Description

Technical Field

[0001] This invention relates to the intersection of meteorological forecasting and artificial intelligence technologies, specifically to a method and system for efficient and flexible integration and collaborative meteorological forecasting of multiple data sources, AI meteorological models, noise disturbances, post-processing and other components through modular workflow orchestration. Background Technology

[0002] In recent years, deep learning-based artificial intelligence (AI) models have made groundbreaking progress in the field of weather forecasting, resulting in high-performance global weather forecasting models such as FourCastNet (https: / / arxiv.org / abs / 2202.11214), GraphCast (https: / / arxiv.org / abs / 2212.12794), and Pangu-Weather (doi: 10.1038 / s41586-023-06185-3.). These AI weather models have shown great potential in terms of forecasting speed and accuracy. However, the following technical bottlenecks exist in practical operational applications: 1. Difficulty in model integration: Different AI meteorological models are developed by different teams, and their input and output formats, data preprocessing methods, and operating environments vary. When combining them to complete complex forecasting tasks (such as "global forecasting → regional downscaling → professional diagnosis"), a large amount of adaptation and glue code needs to be written, resulting in high integration complexity and poor system stability.

[0003] 2. Insufficient utilization of computing resources: In the traditional approach, when multiple models are executed sequentially, intermediate data often needs to be written to disk before being read, which cannot fully utilize GPU memory for high-speed pipelined computation, resulting in I / O bottlenecks and computational latency.

[0004] 3. Insufficient flexibility and maintainability: The forecasting process (workflow) is tightly coupled with the specific model implementation. When it is necessary to change the data source, upgrade the model, or adjust the process, the entire system code needs to be modified, resulting in poor maintainability and difficulty in quickly responding to changes in business needs.

[0005] Therefore, there is an urgent need for a method that can uniformly manage, flexibly orchestrate, and efficiently execute multiple AI meteorological models for weather forecasting, in order to lower the barrier to entry and improve the overall performance and scalability of the weather forecasting system. Summary of the Invention

[0006] This invention aims to overcome the shortcomings of existing technologies and provide a modular prediction method and system for AI meteorological models based on workflow orchestration. By defining standardized component interfaces and a unified data exchange format, and based on zero-copy data streams, it enables "plug-and-play" modular assembly and efficient pipeline execution of data acquisition, noise disturbance, model inference, post-processing, and result output in meteorological prediction tasks.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a weather forecasting method based on an AI weather model using component-based workflow orchestration, characterized by the following steps: Step S1: Decomposing the weather forecasting task into multiple subtasks; wherein each subtask corresponds to a standardized component in a standardized component library; the subtasks have various types, including data source subtasks, input preprocessing subtasks, model prediction subtasks, data post-processing operator subtasks, and output subtasks; the standardized components have predefined standardized interfaces, through which data is exchanged between standardized components; the multiple subtasks include at least one data source subtask, at least one model prediction subtask, and at least one output subtask; Step S2: Orchestrending the workflow to generate a workflow description; wherein the workflow description is used to describe the combination of multiple subtasks implementing the weather forecasting task decomposed in Step S1 and their execution order; the described workflow has a directed acyclic graph (DAG) form; the nodes of the DAG correspond one-to-one with the subtasks and represent pairs of subtasks. The standardized component call corresponding to the task, the directed edges of the directed acyclic graph represent the data transfer and direction between the standardized components corresponding to the subtask; the task type of the subtask corresponding to the root node of the directed acyclic graph is the data source subtask, and the type of the subtask corresponding to the last node of the directed acyclic graph is the output subtask; the workflow description also includes the parameters required by each node; Step S3: Load and parse the workflow description generated in step S2 through the workflow engine, create the corresponding execution graph, and schedule the standardized components in the execution graph in sequence; wherein, the execution graph corresponds to the directed acyclic graph; during the process of scheduling each standardized component according to the execution graph, the workflow engine adopts a data exchange mechanism based on zero-copy memory to transfer data between standardized components, and the data transferred between standardized components is a memory reference stored in a multidimensional array or tensor object; when transferring data between standardized components, the workflow engine also performs data format conversion; Step S4: Format and output the output data through the output component of the last node of the execution graph.

[0008] According to the meteorological forecasting method of the first aspect of the present invention, the meteorological forecasting task includes a determination forecasting task, an ensemble forecasting task, and an integrated forecasting task; the data source subtask corresponds to a data source component, the input preprocessing subtask corresponds to a data preprocessing component, the model prediction subtask corresponds to a meteorological forecasting model component, the data post-processing operator subtask corresponds to a prediction post-processing operator component, and the output subtask corresponds to an output component; the meteorological forecasting model component supports meteorological forecasting models including Fourier neural network operator models, graph neural network models, Transformer models, and combinations thereof; the prediction post-processing operator component supports operators for generating initial field perturbation operators for ensemble forecasts, super-resolution operators for regional downscaling, and diagnostic operators for calculating derived meteorological variables.

[0009] According to the meteorological forecasting method of the first aspect of the present invention, the workflow description is a workflow definition file in YAML format; the workflow description specifies the corresponding normalized component, the parameters used when the normalized component is executed, and the predecessor node of the node for each node in the directed acyclic graph, and the input data of the node comes from the predecessor node; the workflow description also describes the type of subtask corresponding to each node.

[0010] According to the meteorological forecasting method of the first aspect of the present invention, during the scheduling of standardized components by the workflow engine, an execution context is provided for the standardized component to be executed based on the subtask type corresponding to each node described in the workflow description, and the standardized component corresponding to the node is scheduled to be executed in the execution context; wherein the workflow engine provides a GPU-based execution context for the meteorological forecasting model component and the forecast post-processing operator component; the workflow engine configures the standardized components according to the parameters of the nodes in the workflow description; the workflow engine identifies the order in which each node is scheduled to be executed according to the dependency relationship of the nodes in the workflow description; wherein, before scheduling the first node, the workflow engine determines that the first node can be scheduled to be executed based on the fact that all the nodes it depends on have been executed.

[0011] According to a meteorological forecasting method of a first aspect of the present invention, the meteorological forecasting model component integrates a FourCastNet model, which depends on the meteorological data output by the data source component; the workflow engine schedules the execution of the data source component before scheduling the execution of the meteorological forecasting model component; the forecast post-processing operator component integrates a Wind-Topo wind field and terrain downscaling model, which depends on the meteorological forecasting results output by the meteorological forecasting model component and the terrain data output by the data source component; the workflow engine schedules the execution of the meteorological forecasting model component before scheduling the execution of the forecast post-processing operator component; the output component saves the meteorological forecasting results output by the forecast post-processing operator component to a disk; the workflow engine schedules the execution of the forecast post-processing operator component before scheduling the execution of the output component.

[0012] According to the meteorological forecasting method of the first aspect of the present invention, the directed acyclic graph described in the workflow realizes ensemble forecasting or parallel execution of multiple diagnostic tasks by describing branching and aggregation structures; wherein, in the directed acyclic graph described in the workflow, multiple nodes that depend on the same node together with the same node they depend on constitute a branching structure, the nodes that depend on the multiple nodes of the branching structure are aggregation nodes, and the multiple nodes of the branching structure and the aggregation nodes constitute an aggregation structure.

[0013] According to the meteorological forecasting method of the first aspect of the present invention, the common nodes of the branch structure are data preprocessing components for adding perturbation noise to meteorological data to generate different initial fields respectively provided to the plurality of nodes; each of the plurality of nodes is a meteorological forecasting model component; the ensemble node is a forecasting post-processing operator component for performing ensemble forecasting on the calculation results of the plurality of nodes and calculating probability forecasting results.

[0014] According to the meteorological forecasting method of the first aspect of the present invention, wherein, The workflow engine, in response to recognizing a directed acyclic graph (DAG) of the workflow description, identifies that the upstream node outputs data with a multidimensional array structure, and the downstream node, dependent on this upstream node, receives data with a tensor structure. The workflow engine converts the multidimensional array structure data output by the upstream node into a tensor object. This conversion process directly accesses the cached data source, shares the underlying memory, and does not copy the underlying data buffer. The workflow engine then provides a memory reference to the tensor object to the downstream node; or... In response to the recognition of a directed acyclic graph in the workflow description, the workflow engine identifies that the upstream node outputs data with a tensor structure, and the downstream node, which depends on the upstream node, receives data with a tensor structure. The workflow engine then directly provides the memory reference of the tensor object output by the upstream node to the downstream node.

[0015] According to the meteorological forecasting method of the first aspect of the present invention, the workflow engine also provides a cache: for an upstream node whose output is used by multiple downstream nodes, its output result is cached in memory for direct reuse by subsequent nodes.

[0016] According to a second aspect of the present invention, a weather forecasting method based on an AI weather model using component-based workflow orchestration is provided, characterized by the following steps: Step S1: Registering multiple standardized components to a standardized component library, wherein the multiple standardized components correspond to multiple sub-tasks obtained by splitting the weather forecasting task; the sub-tasks have various types, including data source sub-tasks, input preprocessing sub-tasks, model prediction sub-tasks, data post-processing operator sub-tasks, and output sub-tasks; the standardized components have predefined standardized interfaces, through which data is exchanged between the standardized components; the multiple sub-tasks include at least one data source sub-task, at least one model prediction sub-task, and at least one output sub-task; Step S2: Orchestrending a workflow to generate a workflow description; wherein the workflow description is used to describe the combination of multiple sub-tasks implementing the weather forecasting task split in step S1 and their execution order; the described workflow has a directed acyclic graph form; the nodes of the directed acyclic graph correspond one-to-one with the sub-tasks. The directed edges of the directed acyclic graph (DAG) represent the data transfer and direction between the standardized components corresponding to the subtasks. The root node of the DAG corresponds to a data source subtask, and the last node of the DAG corresponds to an output subtask. The workflow description also includes the parameters required by each node. Step S3: Load and parse the workflow description generated in step S2 through the workflow engine, create the corresponding execution graph, and schedule the standardized components in the execution graph in sequence. The execution graph corresponds to the DAG. During the scheduling of standardized components according to the execution graph, the workflow engine uses a zero-copy memory data exchange mechanism to transfer data between standardized components. The data transferred between standardized components is a memory reference stored in a multidimensional array or tensor object. The workflow engine also performs data format conversion when transferring data between standardized components. Step S4: Format and output the output data through the output component of the last node of the execution graph.

[0017] According to a third aspect of the present invention, an information processing device is provided, comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that the processor, when executing the program, implements a weather forecasting method according to a first or second aspect of the present invention.

[0018] The present invention also provides a system for implementing the above method, comprising: a standard component for storing encapsulated standardized components; an orchestrator for receiving user-defined workflow configurations and generating executable workflow descriptions; a workflow engine for loading workflow descriptions, instantiating components, scheduling execution, and managing data flow between components; and an execution context for providing unified computing resource management and data exchange services.

[0019] Compared with the prior art, the present invention has the following significant advantages: 1. High flexibility and maintainability: Through modular design and standardized interfaces, the weather forecasting process can be assembled in a "building block" manner. Users only need to modify the configuration file to change the model and adjust the process without having to delve into the underlying code, which greatly improves the system's maintainability and business response speed.

[0020] 2. High efficiency and high performance: By adopting memory-based zero-copy data exchange and a unified tensor format, the serialization overhead and I / O bottleneck of data conversion between models are eliminated, enabling multiple AI models to form an efficient computing pipeline in GPU memory, which significantly improves the overall throughput from data input to result output.

[0021] 3. Excellent scalability: Supports the integration of any new model, data source, or algorithm that conforms to standardized interface specifications. This architecture is easily extended to distributed computing environments, supporting high-performance computing scenarios such as large-scale ensemble forecasting, providing a solid foundation for the operational application of AI meteorological models. Attached Figure Description

[0022] Figure 1 This is a block diagram of the AI ​​weather forecasting system according to an embodiment of the present invention.

[0023] Figure 2 A flowchart of a weather forecasting method based on an AI weather model with workflow orchestration, provided for an embodiment of the present invention.

[0024] Figure 3A The flowchart illustrates the meteorological forecasting method based on the workflow orchestration-based AI meteorological model of this application, which realizes both global and local refined meteorological forecasts.

[0025] Figure 3B and Figure 3C Showing Figure 3A The implementation map of weather forecasting to achieve global weather forecasting and local refined weather forecasting.

[0026] Figure 4 The diagram illustrates the execution of ensemble forecasting using multiple meteorological forecasting models. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0028] Figure 1 This is a block diagram of the AI ​​weather forecasting system according to an embodiment of the present invention.

[0029] The AI ​​weather forecasting system of this invention is used to execute the AI ​​weather model forecasting method based on workflow orchestration of this invention, and includes a standardized component library, an orchestrator, and a workflow engine. The AI ​​weather forecasting system runs on a computer system including a GPU, which accelerates the computation of the AI ​​model.

[0030] The standardized component library stores encapsulated standardized components. To facilitate integration and interaction through the workflow engine, these standardized components have predefined standardized interfaces. For example, standardized interfaces use tensors as input and output objects, thereby unifying the data exchange format between different standardized components.

[0031] Meteorological forecasting tasks include deterministic forecasting, ensemble forecasting, and integrated forecasting. Deterministic forecasting is based on a single initial field and model parameters, generating a uniquely determined weather forecast field. Ensemble forecasting, by applying small perturbations to the initial field or model parameters, generates multiple forecast members forming an ensemble, used to quantify forecast uncertainty. Integrated forecasting combines the outputs of multiple different meteorological models, obtaining a more accurate forecast through weighted or learning methods.

[0032] To facilitate the implementation of various types of weather forecasts, this invention breaks down the weather forecasting task into multiple components according to its processing stages, including a data source component, a data preprocessing component, a weather forecasting model component, a forecasting post-processing operator component, and an output component. A unified data exchange format is used between components, and standardized interfaces are set for each component, making each component a standardized component. Each standardized component is stored and managed through standardized component groups. Within the standardized components, each component is independent and instantiated according to configuration parameters. Therefore, it can be scheduled and executed by the workflow engine. Furthermore, multiple standardized components can be scheduled and executed in parallel without interfering with each other.

[0033] The data source component acquires meteorological data locally or remotely. It obtains initial field data (including temperature, specific humidity, topographic data, etc.) from meteorological data providers such as ECMWF (European Centre for Medium-Range Weather Forecasts, https: / / data.ecmwf.int / ) and NOAA (National Oceanic and Atmospheric Administration, https: / / nomads.ncep.noaa.gov). The data source component may also acquire raw data from databases or files. This invention does not limit the source of the data acquired by the data source.

[0034] The data preprocessing component is used to transform the raw data acquired by the data source component, add noise perturbations, etc. In the AI ​​weather forecasting system of this invention, the data preprocessing component is not necessary. Some data source components perform conversion of the data source data into a format that can be received by other standardized components, such as standardized meteorological data tensors.

[0035] The weather forecasting model component receives standardized weather data tensors and outputs a forecast result tensor through forward inference of the AI ​​weather model. This component includes, for example, AI large-scale model components, AI mesoscale model components, and AI microscale model components. For instance, AI weather models such as FourCastNet, GraphCast, and Pangu-Weather are encapsulated as standardized components.

[0036] The prediction post-processing operator component is used to perform post-processing operations such as transformation and diagnosis on tensors. The output component is used to persistently store the result tensor.

[0037] An orchestrator receives user-defined workflow configurations and generates executable workflow descriptions. A workflow description is a configuration file in a language such as YAML that describes the workflow. The workflow description specifies the standardized components used, their input and output data, and the data flow input and output relationships between these standardized components.

[0038] The workflow engine is used to load workflow descriptions generated by the orchestrator, instantiate standardized components, schedule, execute, and manage data flows between standardized components.

[0039] An execution context provides unified computing resource management and data exchange services. An execution context can include multiple instances to run multiple instantiated standardized components. Typical examples of execution contexts include virtual machines, processes, threads, containers, and GPU contexts.

[0040] To implement the AI ​​meteorological model prediction method based on workflow orchestration according to embodiments of the present invention, users specify workflow configurations through an orchestrator and generate workflow descriptions. The workflow engine loads the workflow descriptions and, based on the content of each node in the workflow descriptions, loads and schedules the execution of each standardized component. Each standardized component executes independently and generates meteorological prediction results. Users can combine multiple standardized components from the standardized component library in a "building block" assembly manner according to business needs, enabling them to work collaboratively to produce specified meteorological prediction results. Users only need to modify the workflow description configuration file to change models and adjust processes without modifying the underlying prediction model code, greatly improving system maintainability and business response speed.

[0041] Figure 2 A flowchart of a weather forecasting method based on an AI weather model with workflow orchestration, provided for an embodiment of the present invention.

[0042] Weather forecasting is a highly complex task, involving large amounts of data, complex algorithms, and sensitivity to disturbances. In this embodiment of the invention, in step S1, the complex weather forecasting task is broken down into multiple subtasks. Each subtask corresponds to a standardized component in a standardized component library and can be executed by the corresponding standardized component. For example, the determination forecasting task, ensemble forecasting task, and integrated forecasting task can be broken down into data source subtasks, input preprocessing subtasks, AI large-scale model subtasks, AI mesoscale model subtasks, AI microscale model subtasks, data post-processing operator subtasks, and output subtasks. The AI ​​large-scale model subtask, AI mesoscale model subtask, and AI microscale model subtask are collectively referred to as the model prediction subtask. The data source subtask is composed of a data source component (see also...). Figure 1 The input preprocessing subtask is executed by the data preprocessing component, the model prediction subtask is executed by the corresponding meteorological prediction model component, the data postprocessing operator subtask is executed by the prediction postprocessing operator component, and the output subtask is executed by the output component.

[0043] Within a single weather forecasting task, there can be one or more subtasks of the same type. For example, there may be two or more data source subtasks, each used to acquire data from different data sources, with the input preprocessing subtask aggregating the data from these different sources. Another example is that a single forecasting task may include AI large-scale model subtasks, AI mesoscale model subtasks, and AI microscale model subtasks to perform weather forecasts for different spatial ranges. Yet another example is that a single forecasting task may include multiple data post-processing operator subtasks, which collaboratively complete complex data post-processing. Optionally, these data post-processing operator subtasks may have a unified data interaction format, facilitating the direct use of the output of one data post-processing operator subtask as the input of other data post-processing operator subtasks, thus promoting collaborative work among different data post-processing operator subtasks.

[0044] The meteorological forecasting model components corresponding to the AI ​​large-scale model subtask, AI mesoscale model subtask, and AI microscale model subtask have standardized interfaces, receiving input data in tensor format and generating prediction results in tensor format. The data post-processing operator subtask also has a standardized interface, receiving the tensor format prediction results output from the AI ​​large-scale model subtask, AI mesoscale model subtask, and AI microscale model subtask, and performing transformations, diagnostics, and other operations on the tensors.

[0045] The input preprocessing subtask is used to transform the raw data obtained from the data source and add noise perturbations. The output subtask is used to persistently store the result tensor.

[0046] The meteorological forecasting model components include, for example, meteorological forecasting models based on various neural network architectures, including but not limited to Fourier neural operator models, graph neural network models, Transformer models, and combinations thereof. The data post-processing components include initial field perturbation operators for generating ensemble forecasts, super-resolution operators for regional downscaling, and diagnostic operators for calculating derived meteorological variables.

[0047] By breaking down complex weather forecasting tasks into multiple sub-tasks through step S1, these tasks can be implemented using a combination of standardized components. This eliminates the need for tedious low-level coding and allows for rapid updates to weather forecasting tasks by replacing standardized components or changing their combination methods, thus accelerating the development process. Furthermore, the sub-task division and standardized interfaces facilitate the combination of different standardized components, further enriching the variety of weather forecasting tasks that the method provided by this invention can implement.

[0048] Optionally, standardized components are registered in a standardized component library to update the library. For some sub-tasks of a weather forecasting task, whose corresponding standardized components are not yet stored in the standardized component library, new standardized components are constructed and registered in the standardized component library, allowing such weather forecasting tasks to still be implemented by the embodiments of this application. Furthermore, components registered in the standardized component library are instantiated so that they can be called by the workflow engine. Optionally, a standardized component is instantiated only when the workflow engine calls it, to reduce the consumption of system resources in weather forecasting tasks.

[0049] Alternatively, the standardized components can have a unified interface but different granularities or complexities. Smaller granularity helps to achieve more diverse weather forecasting tasks by combining standardized components, while larger granularity helps to reduce the complexity of orchestration workflows.

[0050] Continue reading Figure 2In step S2, the workflow is orchestrated, and a workflow description is generated. The workflow description describes the combination of subtasks and their execution order that implement the weather forecasting task broken down in step S1. The described workflow has a directed acyclic graph (DAG) form. Nodes in the DAG are subtasks or calls to standardized components. Directed edges represent data transfer and its direction between subtasks, reflecting data dependencies between them; that is, the output of the preceding node connected by a directed edge serves as the input of the following node. The DAG in the workflow description includes branches and allows single or multiple branches to execute in parallel. The nodes represented by the data source subtasks are typically located at the start or root node of the DAG. Data source subtasks are also allowed to appear at intermediate nodes in the DAG to introduce new meteorological element data, such as specific humidity and topography, during the weather forecasting task processing. The workflow description also specifies the parameters required for the execution of each standardized component.

[0051] For example, a workflow description is a workflow definition file in YAML format. For each node in the directed acyclic graph, the workflow description specifies its corresponding normalized component, the parameters used when the normalized component is executed, and the predecessor node from which the node's input data comes. The workflow description also describes the type of each node, indicating whether the subtask it represents is a data source subtask, a model prediction subtask, a data post-processing operator subtask, or an output subtask. When parsing and processing the workflow description, the workflow engine can determine how to execute the corresponding normalized component based on the node's type, such as providing the necessary execution context for the normalized component.

[0052] The directed acyclic graph (DAG) described in the workflow can depict branching and aggregation structures, used to implement ensemble forecasts or execute multiple diagnostic tasks in parallel. Diagnostic tasks are used to analyze and verify the calculated meteorological fields to evaluate or correct the meteorological forecast results.

[0053] Continue reading Figure 2 In step S3, the workflow description generated in step S2 is executed using a zero-copy memory method. This is done through the workflow engine (see also...). Figure 1 Load and parse the workflow description generated in step S2, create the corresponding execution graph (the directed acyclic graph represented by the workflow description), and schedule the standardized components in the execution graph in sequence.

[0054] Weather forecasting tasks typically require processing large amounts of data. Breaking down the forecasting task into multiple subtasks introduces significant data migration overhead when transferring data between standardized components of each subtask, leading to performance degradation. To address this issue, this application employs a zero-copy memory-based data exchange mechanism for data transfer between standardized components within the execution workflow description. The resulting data generated by the standardized components is stored, for example, in GPU memory, while memory references to the resulting data objects are passed between the standardized components, significantly reducing the amount of data transferred.

[0055] The data source component typically acquires data in the form of a multidimensional array, or data preprocessing generates a multidimensional data structure from the data source data. In contrast, the AI ​​weather model typically processes data in the form of tensors. The workflow engine also automatically performs necessary data format conversions between adjacent standardized components.

[0056] For example, for a multidimensional array object output by an upstream component, the workflow engine converts the multidimensional array into a tensor object required by the downstream component before passing it to the downstream component. This conversion process does not copy the underlying data buffer to achieve zero-copy data exchange.

[0057] Optionally or further, the workflow engine also provides a caching mechanism: for standardized components that are computationally expensive and whose output is reused in multiple subtasks, their output is cached in memory for direct reuse in subsequent subtasks.

[0058] Furthermore, adjacent nodes in the execution graph may be executed by different devices; for example, the data source component might run on a CPU, while the weather forecasting model component might run on a GPU. For adjacent nodes with data transfer relationships, if they are executed by different devices, the workflow engine automatically handles data transfer between devices, such as moving data from CPU memory to GPU memory, thus ensuring continuous data flow within GPU memory and avoiding unnecessary serialization and disk I / O operations. As an example, when scheduling execution graph nodes, the workflow engine creates or provides execution contexts for the standardized components being executed. Based on the type of execution context, the workflow engine identifies the location of the data processed by each node (GPU memory or CPU memory). When data shared by adjacent nodes is located on different devices, data transfer and optional data format conversion operations are performed. Further, optionally, the workflow engine also utilizes high-speed data transfer mechanisms such as DMA or NVLink to accelerate data transfer between devices.

[0059] Continue reading Figure 2In step S4, the calculation results are collected and output. The final node of the execution graph represented by the workflow description is usually an output subtask, which formats and persistently stores the final results of the entire weather forecasting task through a specified output component. For example, it may be stored as a file in a specified format on a high-capacity storage device, or presented to the user in a visual format.

[0060] Figure 3A The flowchart illustrates the meteorological forecasting method based on the workflow orchestration-based AI meteorological model of this application, which realizes both global and local refined meteorological forecasts. Figure 3B and Figure 3C Showing Figure 3A The implementation map of weather forecasting to achieve global weather forecasting and local refined weather forecasting.

[0061] exist Figures 3A-3C In the embodiments, the forecasting task of "global medium-range weather forecast → regional downscaling" is achieved.

[0062] See Figure 2 and Figure 3A In step S1, the prediction task is broken down into multiple subtasks, including the data source subtask, the AI ​​large-scale model subtask, the data post-processing operator subtask, and the output subtask.

[0063] The data source subtask is implemented by instantiating the DataProvider data source component and configuring it to obtain initial field data (including temperature, specific humidity, topographic data, etc.) from meteorological data providers such as ECMWF and NOAA.

[0064] The AI ​​large-scale model subtask is implemented by instantiating the FourCastNet weather prediction model component, loading its pre-trained weights, and using it to generate a global atmospheric state forecast for the next 10 days.

[0065] The data post-processing operator subtask is implemented by instantiating prediction post-processing operator components (such as regional clipping and downscaling models like Wind-Topo, TerraWind, and CorrDiff). It is used to downscale large-scale global forecast results to high-resolution fields of the target region. It can also add key regional variables, such as complex terrain areas, according to the prediction scenario to improve the regional fine forecasting capability.

[0066] The output subtask is implemented by instantiating the ZarrProvider output component and configuring it with the result saving format, storage path, etc.

[0067] In step S2, the workflow for implementing the forecasting task of "global medium-range weather forecast → regional downscaling" is defined and orchestrated, and a workflow description in YAML format is generated.

[0068] As an example, a workflow can be defined using a YAML configuration file as follows: workflow: name: global_downscaling steps: - type: data_provider component: DataProvider params: {date: "2026-01-01-00", variables:["temperature", ...,"orography"]} - type: forecast component: FourCastNet params: {lead_time: 240} depends_on: [data_provider] - type: post_processing component: Wind-Topo params: {region: “guizhou”} depends_on: [forecast, data_provider] - type: output component: ZarrProvider params: {path: “ / results / forecast.zarr”} depends_on: [post_processing] In the YAML configuration file, the "workflow" field defines the name of the workflow and includes a "steps" field. The "steps" field defines the individual steps of the workflow, represented by a directed acyclic graph (DAG). The nodes of the graph include the steps themselves, and the graph also represents the directed connections between nodes. The "-type" field indicates each node in the DAG and specifically describes the node's type (indicated by content such as "data_provider" after -type), the node's corresponding normalized component (indicated by "component"), the node's parameters (indicated by "params"), and the node's dependencies (indicated by "depends_on").

[0069] Based on the node type in the workflow description, the workflow engine knows how to execute that node, such as what execution context to obtain for that node. Based on the standardized component corresponding to the node, the workflow engine retrieves the corresponding instantiated standardized component from the standard component library and schedules or executes it. Based on the node's parameters, the standardized component is configured. In the workflow description, the node's parameter field can describe one or more parameters for use by various standardized components. Node dependencies describe the nodes that the execution of the current node depends on. For example, the node type in the workflow description indicates the nodes that the execution of the current node depends on. Based on the node dependencies in the workflow description, the workflow engine identifies the order in which nodes are scheduled for execution. Before scheduling a node, its dependent nodes should have already been executed. Multiple nodes whose dependent nodes have been executed can be scheduled for parallel execution. Node dependencies can describe zero or one or more nodes that the execution of the current node depends on.

[0070] In step S3, the workflow engine executes the workflow description generated in step S2 using a zero-copy memory method.

[0071] Taking the workflow description in the generated YAML configuration file as an example, the workflow engine parses the YAML configuration file and constructs an execution graph. As an example, the constructed execution graph consists of... Figure 3B and Figure 3C exhibit.

[0072] See Figure 3BThe execution graph consists of four nodes, representing the execution of the data source component DataProvider, the weather forecasting model component FourCastNet, the post-processing operator component Wind-Topo, and the output component ZarrProvider. The weather forecasting model component FourCastNet depends on the output of the data source component DataProvider. The post-processing operator component Wind-Topo depends on the terrain data output by the data source component DataProvider and the weather forecast results output by the weather forecasting model component FourCastNet. The output component ZarrProvider depends on the output of the post-processing operator component Wind-Topo.

[0073] Therefore, when the workflow engine schedules the execution of each component according to the execution graph, it first executes the data source component DataProvider to obtain meteorological and terrain data. The meteorological data is provided to the meteorological forecasting model component FourCastNet in a zero-copy manner. After the meteorological forecasting model component FourCastNet generates meteorological results, the execution dependency conditions of the forecast post-processing operator component Wind-Topo are met, and then the workflow engine schedules the forecast post-processing operator component Wind-Topo. After the forecast post-processing operator component Wind-Topo generates output, it then schedules the output component ZarrProvider.

[0074] See also Figure 3C ,same Figure 3B compared to, Figure 3C The document also demonstrates the data processing performed by the workflow engine and the specified execution context when scheduling according to the execution graph. For example, the meteorological data output by the data source component DataProvider is of type xarray.DataArray (a multidimensional array), which the workflow engine further converts to type torch.Tensor (a tensor) without requiring normalization. The workflow engine also provides GPU-based execution contexts for the meteorological forecasting model component and the prediction post-processing operator component, enabling the execution of the FourCastNet meteorological forecasting model component and the Wind-Topo prediction post-processing operator component via the GPU. The workflow engine also converts the torch.Tensor format processing results output by the Wind-Topo prediction post-processing operator component into type xarray.DataArray and provides them to the output component ZarrProvider.

[0075] The dataflow engine also transfers data between the FourCastNet weather forecasting model component and the Wind-Topo post-processing operator component using a zero-copy memory data transfer mechanism. The output of the FourCastNet weather forecasting model component is stored in GPU memory as a tensor. The dataflow engine passes the memory reference of the tensor to the Wind-Topo post-processing operator component, allowing Wind-Topo to obtain the tensor-type input data required for computation through the memory reference.

[0076] Optionally or further, the data stream engine, in addition to format conversion, moves the data output by the data source component DataProvider to GPU memory. Alternatively, the data stream engine can be configured to allow the data source component DataProvider to directly generate the acquired meteorological data in GPU memory, and the operation of directly outputting it as a tensor is also performed in GPU memory, thus eliminating the need for subsequent data movement operations. All data is retained in GPU memory, achieving zero-copy transfer.

[0077] In step S4, the output component ZarrProvider collects the weather forecast results data generated by the forecast post-processing operator component Wind-Topo and outputs them. Optionally, the output component ZarrProvider adds geographic coordinate information to the data and writes it to disk.

[0078] Ensemble forecasting is a forecasting method used in meteorological prediction to quantify forecast uncertainty. It generates multiple sets of slightly different initial meteorological data by applying small perturbations to the initial meteorological field, then independently forecasts each set, ultimately obtaining multiple forecast results. Statistical analysis of these multiple forecast results yields information such as the reliability, uncertainty, and probability of occurrence of the forecast.

[0079] The meteorological forecasting method based on workflow orchestration of the AI ​​meteorological model in this application is also applicable to realizing ensemble forecasting based on multiple meteorological forecasting models.

[0080] Figure 4 The diagram illustrates the execution of ensemble forecasting using multiple meteorological forecasting models. Figure 4 The paper also demonstrates a branching workflow constructed through a workflow description to achieve ensemble forecasting. This branching structure enriches the meteorological forecasting process that the method of this invention can support.

[0081] Combination Figure 2 and Figure 4 In step S1, the ensemble forecast task is divided into multiple subtasks, including a data source subtask, a data preprocessing subtask, three AI model subtasks, an ensemble forecast subtask, and an output subtask.

[0082] The data source subtask is implemented by instantiating the data source component. The data preprocessing subtask is implemented by instantiating the data preprocessing component. The three model prediction subtasks are implemented by instances of the three meteorological prediction model components, GraphCast. Figure 4 The model is shown as meteorological forecasting model component 0, meteorological forecasting model component 1, and meteorological forecasting model component 2. The ensemble forecasting task is implemented by instantiating the forecast post-processing operator component EnsembleNode. The output subtask is implemented by instantiating the output component.

[0083] In step S2, the directed acyclic graph task chain workflow is defined and orchestrated. The workflow description defines a data preprocessing subtask, which adds perturbation noise to the meteorological data output by the data source subtask to generate multiple slightly different initial fields.

[0084] The workflow description also includes multiple parallel GraphCast instances of the weather forecasting model component (each GraphCast instance uses an initial field with different initial perturbations for weather forecasting). These GraphCast instances all depend on the data preprocessing subtask and form a parallel, branching structure. Optionally, the multiple parallel weather forecasting model components do not need to use the same weather forecasting model; for example, one or some weather forecasting model components may use the FourCastNet model, while another or some other weather forecasting model components may use the GraphCast model.

[0085] The workflow description also includes the use of a prediction post-processing operator component, EnsembleNode, to perform ensemble forecasting on the results of parallel branches and calculate probability forecasts. The forecast results are then output through an output component.

[0086] In step S3, the workflow is interpreted and executed based on the zero-copy data stream. The workflow engine loads and executes the generated workflow description. When the data preprocessing subtasks are completed, and parallel branches are identified, it is determined that these GraphCast instances of the weather forecasting model components can be executed in parallel because they all depend on the data preprocessing subtasks.

[0087] The workflow engine utilizes multithreading or asynchronous I / O to simultaneously schedule multiple GraphCast instances of the weather forecasting model component. The EnsembleNode, a post-forecasting operator component, waits for all GraphCast instances in all branches to complete before performing statistical calculations and generating probabilistic forecast results.

[0088] In step S4, the scheduling output component outputs the probability prediction results.

[0089] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A weather forecasting method based on an AI weather model with component-based workflow orchestration, characterized in that, Includes the following steps: Step S1: Divide the weather forecasting task into multiple subtasks; each subtask corresponds to a standardized component in the standardized component library; subtasks have various types, including data source subtasks, input preprocessing subtasks, model prediction subtasks, data post-processing operator subtasks, and output subtasks; standardized components have predefined standardized interfaces, through which data is exchanged between standardized components; the multiple subtasks include at least one data source subtask, at least one model prediction subtask, and at least one output subtask; Step S2: Arrange the workflow and generate a workflow description; wherein, the workflow description is used to describe the combination of multiple subtasks and their execution order for implementing the weather forecasting task split in Step S1; the described workflow has a directed acyclic graph (DAG) form; the nodes of the DAG correspond one-to-one with the subtasks and represent the invocation of the standardized components corresponding to the subtasks; the directed edges of the DAG represent the data transfer and transfer direction between the standardized components corresponding to the subtasks; the task type of the subtask corresponding to the root node of the DAG is a data source subtask, and the type of the subtask corresponding to the last node of the DAG is an output subtask; the workflow description also includes the parameters required for each node. Step S3: The workflow engine loads and parses the workflow description generated in step S2, creates a corresponding execution graph, and sequentially schedules the standardized components in the execution graph; wherein, the execution graph corresponds to the directed acyclic graph; during the scheduling of each standardized component according to the execution graph, the workflow engine uses a zero-copy memory-based data exchange mechanism to transmit data between standardized components, and the data transmitted between standardized components is a memory reference stored in a multidimensional array or tensor object; the workflow engine also performs data format conversion when transmitting data between standardized components; Step S4: Format and output the output data through the output component of the last node of the execution graph.

2. The weather forecasting method according to claim 1, characterized in that, The meteorological forecasting task includes a determination forecasting task, an aggregate forecasting task, and an integrated forecasting task; the data source subtask corresponds to a data source component, the input preprocessing subtask corresponds to a data preprocessing component, the model forecasting subtask corresponds to a meteorological forecasting model component, the data postprocessing operator subtask corresponds to a forecasting postprocessing operator component, and the output subtask corresponds to an output component. The meteorological forecasting model components support meteorological forecasting models including Fourier neural operator models, graph neural network models, Transformer models, and combinations thereof. The post-processing operator component supports operators for generating initial field perturbation operators for ensemble forecasts, super-resolution operators for regional downscaling, and diagnostic operators for calculating derived meteorological variables.

3. The meteorological forecasting method according to claim 2, characterized in that, The workflow description is a workflow definition file in YAML format; The workflow description specifies the normalized component, the parameters used when the normalized component is executed, and the predecessor node of the node for each node in the directed acyclic graph. The input data of the node comes from the predecessor node. The workflow description also describes the type of subtask corresponding to each node.

4. The weather forecasting method according to claim 3, characterized in that, During the scheduling of standardized components, the workflow engine provides an execution context for the standardized component to be executed based on the subtask type corresponding to each node described in the workflow description, and schedules the execution of the standardized component corresponding to that node in the execution context; wherein the workflow engine provides a GPU-based execution context for the meteorological prediction model component and the prediction post-processing operator component; The workflow engine configures standardized components based on the parameters of the nodes in the workflow description; The workflow engine identifies the order in which nodes are scheduled for execution based on the dependencies between nodes in the workflow description; wherein, before scheduling the first node, the workflow engine determines that the first node can be scheduled for execution based on the fact that all the nodes it depends on have been executed.

5. The weather forecasting method according to claim 4, characterized in that, The weather forecasting model component integrates the FourCastNet model, which depends on the weather data output by the data source component; the workflow engine schedules the execution of the data source component and then schedules the execution of the weather forecasting model component. The prediction post-processing operator component integrates the Wind-Topo wind field terrain downscaling model, which depends on the meteorological prediction results output by the meteorological prediction model component and the terrain data output by the data source component; the workflow engine schedules the execution of the meteorological prediction model component and then schedules the execution of the prediction post-processing operator component. The output component saves the meteorological forecast results output by the prediction post-processing operator component to the disk; The workflow engine schedules the execution of the output component after scheduling the execution of the prediction post-processing operator component.

6. The meteorological forecasting method according to claim 4, characterized in that, The directed acyclic graph described in the workflow achieves set prediction or parallel execution of multiple diagnostic tasks by describing branching and aggregation structures; wherein, in the directed acyclic graph described in the workflow, multiple nodes that depend on the same node together with the same node they depend on constitute a branching structure, the nodes that depend on the multiple nodes of the branching structure are set nodes, and the multiple nodes of the branching structure and the set nodes constitute an aggregation structure.

7. The weather forecasting method according to claim 6, characterized in that, The common nodes in the branch structure are data preprocessing components used to add perturbation noise to the meteorological data to generate different initial fields for each of the multiple nodes. Each of the plurality of nodes is a weather forecasting model component; The set node is a prediction post-processing operator component used to perform set prediction on the calculation results of the multiple nodes and calculate the probability prediction result.

8. The meteorological forecasting method according to claim 7, characterized in that, In response to the recognition of a directed acyclic graph in the workflow description, the workflow engine identifies that the upstream node outputs data with a multidimensional array structure, and the downstream node receives data with a tensor structure that depends on the upstream node. The workflow engine converts the multidimensional array structure data output by the upstream node into a tensor object. This conversion process directly accesses the cached data source and shares the underlying memory. The workflow engine provides the memory reference of the tensor object to the downstream node. or In response to the recognition of a directed acyclic graph in the workflow description, the workflow engine identifies that the upstream node outputs data with a tensor structure, and the downstream node, which depends on the upstream node, receives data with a tensor structure. The workflow engine then directly provides the memory reference of the tensor object output by the upstream node to the downstream node.

9. A meteorological forecasting method based on an AI meteorological model with component-based workflow orchestration, characterized in that, Includes the following steps: Step S1: Register multiple standardized components to the standardized component library. The multiple standardized components correspond to multiple sub-tasks obtained by splitting the weather forecasting task. The sub-tasks have various types, including data source sub-tasks, input preprocessing sub-tasks, model prediction sub-tasks, data post-processing operator sub-tasks, and output sub-tasks. The standardized components have predefined standardized interfaces, through which data is exchanged between the standardized components; the multiple subtasks include at least one data source subtask, at least one model prediction subtask, and at least one output subtask; Step S2: Arrange the workflow and generate a workflow description; wherein, the workflow description is used to describe the combination of multiple subtasks and their execution order for implementing the weather forecasting task split in Step S1; the described workflow has a directed acyclic graph (DAG) form; the nodes of the DAG correspond one-to-one with the subtasks and represent the invocation of the standardized components corresponding to the subtasks; the directed edges of the DAG represent the data transfer and transfer direction between the standardized components corresponding to the subtasks; the task type of the subtask corresponding to the root node of the DAG is a data source subtask, and the type of the subtask corresponding to the last node of the DAG is an output subtask; the workflow description also includes the parameters required for each node. Step S3: The workflow engine loads and parses the workflow description generated in step S2, creates a corresponding execution graph, and sequentially schedules the standardized components in the execution graph; wherein, the execution graph corresponds to the directed acyclic graph; during the scheduling of each standardized component according to the execution graph, the workflow engine uses a zero-copy memory-based data exchange mechanism to transmit data between standardized components, and the data transmitted between standardized components is a memory reference stored in a multidimensional array or tensor object; the workflow engine also performs data format conversion when transmitting data between standardized components; Step S4: Format and output the output data through the output component of the last node of the execution graph.

10. An information processing device, comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method according to any one of claims 1-9.

Images