Global atmospheric spectral model i / o optimization method, system, device, and medium

Abstract

The application relates to a global atmospheric spectrum model I / O optimization method, system, device and medium. The method comprises the following steps: constructing an asynchronous I / O framework integrated with a global atmospheric spectrum model and XIOS, in the asynchronous I / O framework, the global atmospheric spectrum model as a client is only used for executing a model calculation process, data buffering, model calculation data aggregation and parallel writing are managed by a server while the model calculation is continuously carried out, and the asynchrony of the client-server ensures that the model calculation is largely hindered by I / O tasks, and network congestion and communication delay are reduced. Through a buffer area arranged in the asynchronous I / O framework and an optimization mechanism based on the buffer area, model calculation data is first buffered in the buffer area, so that the overlapping of I / O operation and the model calculation process is maximized, the overhead of frequent I / O operation is reduced, and the global atmospheric spectrum model I / O bottleneck is successfully relieved.

Description

Technical Field

[0001] This application relates to the field of numerical weather prediction technology, and in particular to a global atmospheric spectrum model I / O optimization method, system, device and medium. Background Technology

[0002] Numerical weather prediction (NWP) models are crucial for weather forecasting and climate research. With high-performance computing (HPC) entering the exascale era, these models are moving towards unprecedented resolutions, generating an explosive volume of data. Consequently, input / output (I / O) operations have become a major performance bottleneck, hindering scalability and efficiency.

[0003] This I / O bottleneck is particularly severe for numerical weather prediction models like the Galaxy Global Atmospheric Spectrum Model (YHGSM), which generates high-frequency computational outputs on large 3D grids. YHGSM's original I / O scheme relies on a centralized "N-to-1" approach, where a single master process collects and writes all data. This master-process-based data integration method is prone to severe communication latency, bandwidth limitations, and resource utilization inefficiencies due to load imbalances when dealing with large datasets. Summary of the Invention

[0004] Therefore, it is necessary to provide a global atmospheric spectrum model I / O optimization method, system, equipment, and medium to address the aforementioned technical problems.

[0005] A global atmospheric spectral model I / O optimization method, the method comprising:

[0006] An asynchronous I / O framework integrating the global atmospheric spectrum model and XIOS is constructed. The asynchronous I / O framework consists of a client-server architecture provided by XIOS, with the global atmospheric spectrum model acting as the client in XIOS, and data transmission between the client and server based on buffers.

[0007] During the initialization phase of the global atmospheric spectrum model, communication initialization processing and metadata configuration are performed on the asynchronous I / O framework;

[0008] During the main time step loop phase of the global atmospheric spectrum model, the model calculations are performed by the client. A buffer-based sending optimization mechanism is adopted to send the model calculation data output by the client to a dedicated server in a non-blocking communication manner. The server manages the data buffer and aggregates the model calculation data output by one or more clients. After the server performs data post-processing operations, the data is output in a parallel data format and written to the target file, thereby realizing asynchronous I / O optimization of the global atmospheric spectrum model.

[0009] In one embodiment, data transmission between the client and server is based on a buffer, including:

[0010] Within the asynchronous I / O framework, each client is equipped with a double buffer. When the client performs global atmospheric spectrum model calculations, a double buffering strategy is adopted. The model calculation data output by the client is placed into one of the corresponding double buffers. When one buffer is full, the model calculation data in the full buffer is sent to the server, while the other buffer is switched to continuously receive the model calculation data output by the corresponding client.

[0011] In one embodiment, data transmission between the client and server based on a buffer further includes:

[0012] Within the asynchronous I / O framework, each server is equipped with a circular buffer. In the current time step of the global atmospheric spectrum model calculation, one end of the circular buffer acts as the write end, receiving the model calculation data for the current time step sent by the full buffer, while the other end acts as the read end, reading the model calculation data received in the previous time step and writing it to the target file. In the next time step of the global atmospheric spectrum model calculation, the functions of the two ends of the circular buffer are switched, continuously receiving model calculation data output by the client and synchronously outputting data to the target file.

[0013] In one embodiment, during the initialization phase of the global atmospheric spectrum model, communication initialization processing and metadata configuration are performed on the asynchronous I / O framework, including:

[0014] During the initialization phase of the global atmospheric spectrum model, the global atmospheric spectrum model, acting as a client, calls the XIOS application programming interface to establish a parallel communication context and configure metadata.

[0015] The metadata configuration includes defining key metadata output by the client and transmitting it to the server in XIOS. This includes defining the start date and time step of the global atmospheric spectral model as the calendar of the XIOS simulation, defining the vertical structure of the model as the XIOS axis, and defining the decomposed horizontal grid output by the global atmospheric spectral model as the XIOS domain.

[0016] In one embodiment, during the main time step loop phase of the global atmospheric spectrum model, the global atmospheric spectrum model calculations are performed using a client, and a buffer-based send optimization mechanism is employed to send the model calculation data output by the client to a dedicated server in a non-blocking communication manner, including:

[0017] During the main time step cycle of the global atmospheric spectral model, the global atmospheric spectral model calculations are performed using the client, and the calendar of the XIOS simulation is updated using the current time step calculated by the global atmospheric spectral model.

[0018] Then, iterate through the list of fields of the model calculation data that the client needs to output at the current time step, and determine whether to enable the buffer-based sending optimization mechanism;

[0019] If not enabled, check whether each field in the field list is marked as active by the server, and directly send the data of the active fields to the server in a non-blocking communication manner.

[0020] If enabled, for each field in the field list, the corresponding buffer and counter are located according to the field type, and the field data is copied to the specified position in the corresponding buffer. The buffer counter value is incremented by 1, and the buffer flag is set to true, indicating that there is field data to be sent in the current buffer.

[0021] Iterate through all buffers. When the buffer flag of the current buffer is true and the corresponding counter value is greater than 0, retrieve the field data in the current buffer in a loop. When the retrieved field is active, send the field data to the server in a non-blocking communication manner. Decrement the counter value of the current buffer by 1 until all field data in the current buffer has been sent. Then reset the buffer flag of the current buffer to False.

[0022] In one embodiment, before iterating through the list of fields of the model computation data that the client needs to output at the current time step and determining whether to enable the buffer-based send optimization mechanism, the method further includes:

[0023] The physical field data that the global atmospheric spectrum model needs to output at the current time step is confirmed by XIOS.

[0024] The physical field data is converted from spectral space to grid space, and the I / O dispatch routine in XIOS is called to integrate the converted physical field data into a field list. The boolean parameter LOPT_SEND used during the execution of the I / O dispatch routine controls whether to enable the buffer-based send optimization mechanism.

[0025] In one embodiment, the server performs data post-processing operations, including:

[0026] Perform time averaging and re-grid processing on model calculation data output from single or multiple clients aggregated by the server.

[0027] A global atmospheric spectrum model I / O optimization system, the system comprising:

[0028] The asynchronous framework building module is used to build an asynchronous I / O framework that integrates the global atmospheric spectrum model and XIOS. The asynchronous I / O framework consists of a client-server architecture provided by XIOS, with the global atmospheric spectrum model acting as the client in XIOS, and data transmission between the client and server based on buffers.

[0029] The initialization module is used during the initialization phase of the global atmospheric spectrum model to perform communication initialization processing and metadata configuration for the asynchronous I / O framework.

[0030] The send optimization module is used during the main time step loop phase of the global atmospheric spectrum model. It utilizes the client to perform global atmospheric spectrum model calculations and employs a buffer-based send optimization mechanism to send the model calculation data output by the client to a dedicated server in a non-blocking communication manner. The server manages the data buffer and aggregates the model calculation data output by one or more clients. After the server performs data post-processing operations, it outputs the data in a parallel data format and writes it to the target file, thereby realizing asynchronous I / O optimization of the global atmospheric spectrum model.

[0031] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program performing the following steps:

[0032] An asynchronous I / O framework integrating the global atmospheric spectrum model and XIOS is constructed. The asynchronous I / O framework consists of a client-server architecture provided by XIOS, with the global atmospheric spectrum model acting as the client in XIOS, and data transmission between the client and server based on buffers.

[0033] During the initialization phase of the global atmospheric spectrum model, communication initialization processing and metadata configuration are performed on the asynchronous I / O framework;

[0034] During the main time step loop phase of the global atmospheric spectrum model, the model calculations are performed by the client. A buffer-based sending optimization mechanism is adopted to send the model calculation data output by the client to a dedicated server in a non-blocking communication manner. The server manages the data buffer and aggregates the model calculation data output by one or more clients. After the server performs data post-processing operations, the data is output in a parallel data format and written to the target file, thereby realizing asynchronous I / O optimization of the global atmospheric spectrum model.

[0035] A computer-readable storage medium having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0036] An asynchronous I / O framework integrating the global atmospheric spectrum model and XIOS is constructed. The asynchronous I / O framework consists of a client-server architecture provided by XIOS, with the global atmospheric spectrum model acting as the client in XIOS, and data transmission between the client and server based on buffers.

[0037] During the initialization phase of the global atmospheric spectrum model, communication initialization processing and metadata configuration are performed on the asynchronous I / O framework;

[0038] During the main time step loop phase of the global atmospheric spectrum model, the model calculations are performed by the client. A buffer-based sending optimization mechanism is adopted to send the model calculation data output by the client to a dedicated server in a non-blocking communication manner. The server manages the data buffer and aggregates the model calculation data output by one or more clients. After the server performs data post-processing operations, the data is output in a parallel data format and written to the target file, thereby realizing asynchronous I / O optimization of the global atmospheric spectrum model.

[0039] The aforementioned global atmospheric spectrum model I / O optimization method, system, equipment, and media construct an asynchronous I / O framework integrating the global atmospheric spectrum model and XIOS. Within this framework, the global atmospheric spectrum model, acting as the client, is solely used for model computation, while its I / O tasks are offloaded to the server. The server manages data buffering, aggregates model computation data, and performs parallel writes while continuing model computation. This client-server asynchronicity ensures that model computation is largely unaffected by I / O tasks, significantly reducing network congestion and communication latency. Furthermore, through the buffer settings for data transmission between the client and server within this asynchronous I / O framework, and the buffer-based transmission optimization mechanism, model computation data is first cached in the buffer. This maximizes the overlap between I / O operations and model computation, reducing the overhead of frequent I / O operations and successfully alleviating I / O bottlenecks. This significantly improves the overall efficiency and scalability of the global atmospheric spectrum model, providing a promising path for future high-resolution weather forecasting and climate research. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of a single time step calculation for the YHGSM time integral in one embodiment;

[0041] Figure 2 This is a schematic diagram of the data flow in the I / O stack of the original YHGSM in one embodiment;

[0042] Figure 3 This is a flowchart illustrating the global atmospheric spectrum model I / O optimization method in one embodiment;

[0043] Figure 4 This is a schematic diagram of an asynchronous I / O framework integrating YHGSM and XIOS in one embodiment;

[0044] Figure 5 This is a schematic diagram of an algorithm in one embodiment that controls whether a buffer-based send optimization mechanism is enabled based on the Boolean parameter LOPT_SEND during the I / O dispatch routine runtime.

[0045] Figure 6 This is a schematic diagram of the data flow in the I / O stack of an asynchronous I / O framework integrating YHGSM and XIOS in one embodiment;

[0046] Figure 7 This is a diagram illustrating the percentage of I / O time in the total execution time for the original YHGSM I / O scheme under different numbers of MPI (Message Passing Interface) processes in one embodiment.

[0047] Figure 8 This diagram illustrates the percentage of I / O time in total execution time for an asynchronous I / O framework integrating YHGSM and XIOS, with 4096 client processes, varying with the number of servers; where, Figure 8 (a) is a diagram showing the percentage of client-side I / O time to total execution time as the number of servers varies; Figure 8 (b) is a diagram showing the percentage of I / O time to total execution time on the server side as a function of the number of servers.

[0048] Figure 9 This is a schematic diagram comparing the I / O time and total wall clock time of the original YHGSM and the asynchronous I / O framework integrating YHGSM and XIOS under different total number of processes in one embodiment;

[0049] Figure 10 This is a schematic diagram comparing the speedup and parallel efficiency of the original YHGSM and an asynchronous I / O framework configuration integrating YHGSM and XIOS, relative to a baseline of 1024 processes, in one embodiment; wherein, Figure 10 (a) is a schematic diagram comparing acceleration ratios; Figure 10 (b) is a schematic diagram comparing parallel efficiency;

[0050] Figure 11 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0052] To facilitate understanding, the relevant terms used in the embodiments of this application will be introduced first.

[0053] 1. HPC I / O stack.

[0054] Efficient data output in HPC applications involves a complex hierarchical structure known as the HPC I / O stack. Optimizing I / O performance requires understanding and targeting specific levels within this stack.

[0055] An HPC I / O stack typically includes application software (for generating I / O requests), a high-level I / O library (for simplifying data management and providing parallel I / O functionality), parallel I / O middleware (primarily MPI-IO, which provides a standard API (Application Programming Interface) for parallel file access), a parallel file system (designed for concurrent access), and storage hardware such as hard drives, solid-state drives, or NVMe (Non-Volatile Memory Host Controller Interface) devices. Effective I / O tuning depends on understanding the data flow between these interaction layers.

[0056] 2. YHGSM.

[0057] YHGSM is a global numerical weather prediction system used for operational forecasting and atmospheric research, such as... Figure 1 As shown, its time step alternates between grid-point computation and spectral space computation via Fourier transform and Legendre transform. In the horizontal direction, YHGSM uses a triangularly truncated Gaussian simplified grid, decreasing from longitude points to poles. Despite its advanced features such as the spectral transform dynamic core and complex physical parameterization, YHGSM's original I / O scheme presents significant scalability challenges.

[0058] YHGSM's original I / O scheme uses a centralized "N-to-1" approach, where N represents the total number of processes and 1 represents the main process that collects all local data from other processes. The data flow in the original YHGSM I / O stack is as follows: Figure 2As shown, all MPI computation processes on N nodes send their local data to a single master process (rank 0), which then takes full responsibility for collecting, rearranging, and sequentially writing the data to a single file. This strategy introduces a significant I / O bottleneck because aggregating all data into a single process causes communication latency and fails to fully utilize the available bandwidth of the parallel file system. The consequence of this design is severe load imbalance: the master process (rank 0) handles communication and I / O, while all other computation processes (ranks 1 to N-1) are idle, waiting for the I / O phase to complete. These limitations significantly impact the performance of YHGSM. For example, in a YHGSM Tco1279 simulation on 5120 cores, I / O operations can account for approximately 50% of the total runtime, highlighting the need for a scalable parallel I / O solution. The output of each time step of YHGSM typically includes standard meteorological variables such as temperature, zonal and meridional winds, humidity, geopotential height, and surface pressure, stored in GRIB (binary grid) format. A node can be understood as a physical computing unit on which the YHGSM operating environment depends, typically referring to a single server or computing node in a high-performance computing cluster.

[0059] 3. XIOS: XML Input / Output Server.

[0060] XIOS is an asynchronous I / O library designed specifically for climate and Earth system models. It employs a client-server architecture where clients dispatch data to dedicated "server" processes via non-blocking MPI communication. This achieves asynchronous I / O: clients continue computation while the XIOS server manages data buffering, aggregation, and parallel writes, effectively masking I / O latency. XIOS utilizes parallel libraries like NetCDF4 (Network Common Data Format) for efficient disk writes. Its behavior is configured via external XML files, eliminating the need to recompile the model for I / O modifications. It supports multiple operating modes, including a "server mode" to maximize computation-I / O overlap and an "additional mode" where server tasks run on a subset of the model's MPI processes, resulting in synchronous I / O. This application primarily employs the server mode.

[0061] In one embodiment, such as Figure 3 As shown, taking YHGSM as an example, a global atmospheric spectrum model I / O optimization method is provided, including the following steps:

[0062] Step 301: Construct an asynchronous I / O framework integrating the global atmospheric spectrum model and XIOS; wherein, the asynchronous I / O framework consists of a client-server architecture provided by XIOS, the global atmospheric spectrum model acts as the client in XIOS, and data transmission between the client and the server is based on a buffer.

[0063] Step 302: In the initialization phase of the global atmospheric spectrum model, communication initialization processing and metadata configuration are performed on the asynchronous I / O framework.

[0064] Step 303: During the main time step loop phase of the global atmospheric spectrum model, the global atmospheric spectrum model calculation is performed using the client. A buffer-based sending optimization mechanism is adopted to send the model calculation data output by the client to a dedicated server in a non-blocking communication manner. The server manages the data buffer and aggregates the model calculation data output by one or more clients. After the server performs data post-processing operations, the data is output in a parallel data format and written to the target file, thereby realizing asynchronous I / O optimization of the global atmospheric spectrum model.

[0065] The asynchronous I / O framework constructed by the aforementioned global atmospheric spectrum model I / O optimization method allows the global atmospheric spectrum model, acting as a client, to be used solely for model computation. I / O tasks are offloaded to the server, which manages data buffering, aggregates model computation data, and performs parallel writes while continuing model computation. This client-server asynchronicity ensures that model computation is largely unaffected by I / O tasks, significantly reducing network congestion and communication latency. Furthermore, through the buffer settings for data transmission between the client and server within this asynchronous I / O framework, and the buffer-based transmission optimization mechanism, model computation data is first cached in the buffer. This maximizes the overlap between I / O operations and model computation, reducing the overhead of frequent I / O operations, successfully alleviating I / O bottlenecks, and significantly improving the overall efficiency and scalability of the global atmospheric spectrum model.

[0066] In one embodiment, such as Figure 4 As shown, within the asynchronous I / O framework, each client is equipped with a double buffer (i.e., Figure 4 In the B1 and B2 buffers, when the client performs global atmospheric spectrum model calculations, a double buffering strategy is adopted. The model calculation data output by the client is placed into one of the corresponding double buffers. When one buffer is full, the model calculation data in the full buffer is sent to the server, while the other buffer is switched to continuously receive the model calculation data output by the client. Figure 4 In this context, "output" indicates the output.

[0067] It should be understood that this double-buffering strategy ensures that the client's model calculation and data transmission can be carried out simultaneously, avoiding the calculation process from being blocked due to waiting for transmission.

[0068] In one embodiment, within the asynchronous I / O framework, each server is equipped with a circular buffer (such as...). Figure 4 Each XIOS server is equipped with a circular buffer. In the current time step of the global atmospheric spectrum model calculation, one end of the circular buffer acts as the write end, used to receive the model calculation data of the current time step sent by the full buffer. The other end of the circular buffer acts as the read end, used to read the model calculation data received in the previous time step and write it to the target file. In the next time step of the global atmospheric spectrum model calculation, the functions of the two ends of the circular buffer are switched, continuously receiving the model calculation data output by the client and synchronously outputting the data to the target file.

[0069] It should be understood that this circular buffer setup allows the server to perform receive and output operations in parallel, receiving new data while outputting existing data, thus improving throughput. Furthermore, as long as the data reception and write overhead of the server process within a single time step is less than the data transmission and computation overhead of the client process, the overall I / O overhead of YHGSM can be effectively masked.

[0070] In one embodiment, during the initialization phase of the global atmospheric spectrum model, the global atmospheric spectrum model, acting as a client, calls the XIOS application programming interface to establish a parallel communication context (by calling the xios_initialize (XIOS initialization) function and the xios_context_initialize (XIOS communication context initialization) function respectively) and performs metadata configuration.

[0071] The metadata configuration includes defining key metadata output by the client and transmitting it to the server in XIOS. This includes defining the start date and time step of the global atmospheric spectral model as the XIOS simulation calendar, defining the model's vertical structure as XIOS axes (implemented by calling the `xios_set_axis_attr` function), and defining the decomposed horizontal grid of the global atmospheric spectral model output as XIOS domains (implemented by calling the `xios_set_domain_attr` function). The XIOS simulation calendar can be understood as the simulation's time axis and its attributes.

[0072] After configuration is complete, call the xios_close_context_definition (XIOS close communication context definition) function to end the metadata configuration phase, ensuring that the server fully understands the geometry, decomposition, and timing of the expected data field.

[0073] In one embodiment, during the main time step loop phase of the global atmospheric spectral model, the global atmospheric spectral model calculation is first performed using the client, and the calendar of the XIOS simulation is updated using the current time step calculated by the global atmospheric spectral model.

[0074] Then, iterate through the list of fields of the model calculation data that the client needs to output at the current time step, and determine whether to enable the buffer-based sending optimization mechanism.

[0075] If not enabled, check whether each field in the field list is marked as active by the server, and directly send the data of the active fields to the server in a non-blocking communication manner.

[0076] If enabled, for each field in the field list, the corresponding buffer and counter are located according to the field type, and the field data is copied to the specified position in the corresponding buffer. The buffer counter value is incremented by 1, and the buffer flag is set to true, indicating that there is field data to be sent in the current buffer.

[0077] Iterate through all buffers. When the buffer flag of the current buffer is true and the corresponding counter value is greater than 0, retrieve the field data in the current buffer in a loop. When the retrieved field is active, send the field data to the server in a non-blocking communication manner. Decrement the counter value of the current buffer by 1 until all field data in the current buffer has been sent. Then reset the buffer flag of the current buffer to False.

[0078] Understandably, by enabling the buffer-based send optimization mechanism, the model calculation data is first cached in the buffer and then sent in batches in a loop, reducing the overhead of frequent I / O operations. Furthermore, by sending only the field data in the active state, invalid communication can be avoided, thus improving communication performance.

[0079] In one embodiment, before iterating through the field list of model computation data that the client needs to output at the current time step and determining whether to enable the buffer-based transmission optimization mechanism, the method further includes: XIOS confirming the physical field data that the global atmospheric spectrum model model, which is the client at the current time step, needs to output; converting the physical field data from the spectral space to the grid space, and calling the I / O dispatch routine (I / O_Dispatch) in XIOS to integrate the converted physical field data into a field list, and controlling whether to enable the buffer-based transmission optimization mechanism according to the Boolean parameter LOPT_SEND during the execution of the I / O dispatch routine.

[0080] Furthermore, the LOPT_SEND parameter during I / O dispatch routine execution controls whether the buffer-based send optimization mechanism is enabled, such as... Figure 5 As shown.

[0081] In one embodiment, the server performs data post-processing operations, including: performing time averaging and re-grid processing on model computation data output from one or more clients aggregated by the server.

[0082] The purpose of time averaging is to reduce high-frequency fluctuations and highlight long-term trends. The purpose of re-gridification is to transform data from the original grid to a target grid (such as different resolutions or projection methods).

[0083] Furthermore, the data flow in the I / O stack of the asynchronous I / O framework integrating YHGSM and XIOS constructed according to the method proposed in this application is as follows: Figure 6 As shown, Figure 6 In this code, C represents YHGSM. Two YHGSMs correspond to one dedicated server S, and these servers can interact with each other. Each server outputs data in a parallel data format and writes it to a target file. It then uses standard APIs provided by the I / O middleware to concurrently input the data to the parallel file system and store it in the storage hardware. It's important to note that the number of YHGSMs corresponding to each server is not fixed.

[0084] Parallel data formats include PnetCDF and PHDF5. PnetCDF (Parallel Network General Data Format Library) is an MPI-based parallel read / write tool capable of parallel read / write operations on NetCDF files, primarily optimized through collective I / O. PHDF5 (Parallel Hierarchical Data Format Library) is a parallel extension of HDF5 (Hierarchical Data Format), supporting multi-process collaborative access to HDF5 files and widely used for storing complex structured data.

[0085] Parallel file systems include Lustre, GPFS, BeeGFS, and DAOS. Lustre is a distributed file system based on object storage, consisting of a metadata server (MDS) and an object storage server (OSS), communicating at high speed through the Lustre network. GPFS (General Parallel File System) is a distributed file system that uses a shared disk architecture and supports block-level data sharding and a global namespace. BeeGFS (Bee Cluster File System) adopts a modular design, separating metadata services and storage services, allowing for flexible deployment. DAOS (Distributed Asynchronous Object Storage) is a next-generation parallel storage system developed by Intel, offering microsecond-level latency and supporting millions of IOPS (read / write operations per second).

[0086] Storage hardware includes HDDS (Hard Disk Drive), SSDS (Solid State Drive), and NVMe (Non-Volatile Memory Host Controller Interface).

[0087] Furthermore, to verify the performance of this application, the performance of YHGSM was evaluated on an HPC platform, comparing its original configuration with an asynchronous I / O framework integrating YHGSM and XIOS. The hardware and software configurations of the experiment are shown in Table 1.

[0088] Specifically, the YHGSM was configured with a Tco1279 spectral resolution (9 km, 137 vertical levels). The simulation ran for 72 hours (576 steps) with a time step of 450 seconds, outputting data every 3 hours. The output dataset included a standard suite of atmospheric variables, including temperature, wind component, specific humidity, and surface pressure. The XIOS configuration employed asynchronous processing in a "server mode," with key parameters including a performance-optimized buffer (optimal_buffer_size='performance'), a multiple_file output strategy using collective access, and a buffer-based send optimization mechanism. Performance evaluation was based on total wall clock time, I / O time, and I / O overhead (the percentage of total time spent on I / O). For the XIOS configuration, I / O time was primarily determined by client buffer wait times derived from XIOS performance reports. Scalability was assessed through speedup ratios (…). ) and parallel efficiency ( ) to quantify, respectively defined as:

[0089] ;

[0090] ;

[0091] in, and These represent the baseline time and the number of baseline processes for each experiment. and N These represent the time and number of processes for each data point in each experiment.

[0092]

[0093] To establish a performance benchmark, the original YHGSM I / O scheme underwent strong scalability testing from 1024 to 5120 processes. For example... Figure 7 As shown, although the total wall clock time decreased as expected, the absolute I / O time remained almost unchanged. Therefore, the I / O overhead increased dramatically from 19.34% to a staggering 43.08%, highlighting the severe bottleneck of centralized "N-to-1" designs when scaled up. Since performance gains begin to saturate after exceeding 4096 processes, this application chose this scale as a representative configuration for subsequent optimization experiments to balance significant computational load demands with resource utilization efficiency.

[0094] To study the impact of server allocation, a buffer-based send optimization mechanism was enabled within an asynchronous I / O framework integrating YHGSM and XIOS, with the number of YHGSM client processes fixed at 4096, while the number of XIOS servers was varied. Figure 8 This presentation shows the percentage of I / O time in total execution time, varying with the number of servers, from both client and server perspectives. Figure 8 (a) The performance was explained from the client's perspective, by Figure 8 (a) shows that as the number of servers increases, the I / O overhead and total clock time experienced by the client decrease. This is because a larger server pool can handle concurrent data streams from numerous clients more efficiently, reducing the time clients wait for I / O buffers to become available. The key to this asynchronous I / O scheme is to ensure that the server-side processing time is less than the client's computation time, thus "hiding" the I / O cost after the computation. Figure 8 (b) shows the time the server spent on its I / O tasks. (Compared to) Figure 8 A direct comparison in (b) shows that the server-side I / O time is much shorter than the client-side computation time (i.e., the time in other stages of the diagram). This confirms that server-side I / O operations can completely overlap with client-side computations, effectively preventing I / O from blocking the simulated process.

[0095] The most significant performance gains were observed when the number of servers was expanded to 256 and 512, corresponding to client-to-server ratios of 16:1 and 8:1, respectively. Beyond this range, improvements to wall clock time became negligible. This indicates that the I / O workload was adequately distributed, and the performance bottleneck was no longer the XIOS server, but rather other system limitations, such as the maximum bandwidth of the file system. Therefore, for this specific workload, a client-to-server ratio between 16:1 and 8:1 provided the optimal balance between performance gains and resource costs. This configuration successfully mitigated the I / O bottleneck and restored parallel efficiency by ensuring overlap between computation and I / O, without allocating excessive computational resources to the I / O subsystem.

[0096] Finally, a strong scalability comparison was performed between the original YHGSM and the asynchronous I / O framework integrating YHGSM and XIOS under different total number of processes (client to server ratio 15:1, with buffer-based send optimization enabled). Figure 9 As shown, the XIOS-based solution achieves significant performance improvements across all scales. With a total of 5120 processes, the asynchronous I / O framework integrating YHGSM and XIOS completed the simulation in 863 seconds (2.31 times faster than the baseline of 1992 seconds), and reduced I / O overhead from 43.08% to 5.27%.

[0097] Figure 10 The speedup and parallel efficiency of the original YHGSM and the asynchronous I / O framework configuration integrating YHGSM and XIOS, compared to a baseline of 1024 processes each, further illustrate that the asynchronous I / O framework integrating YHGSM and XIOS exhibits a significant improvement in scalability. At a total of 5120 processes, the parallel efficiency of the asynchronous I / O framework remains at 0.62, a substantial improvement compared to the baseline efficiency of 0.43. This demonstrates that the proposed method successfully eliminates the I / O bottleneck, significantly reduces wall clock time, and restores the parallel efficiency required to utilize large-scale computing resources.

[0098] In one embodiment, a global atmospheric spectrum model I / O optimization system is provided, comprising:

[0099] The asynchronous framework building module is used to build an asynchronous I / O framework that integrates the global atmospheric spectrum model and XIOS. The asynchronous I / O framework consists of a client-server architecture provided by XIOS, with the global atmospheric spectrum model acting as the client in XIOS, and data transmission between the client and server based on buffers.

[0100] The initialization module is used during the initialization phase of the global atmospheric spectrum model to perform communication initialization processing and metadata configuration for the asynchronous I / O framework.

[0101] The send optimization module is used during the main time step loop phase of the global atmospheric spectrum model. It utilizes the client to perform global atmospheric spectrum model calculations and employs a buffer-based send optimization mechanism to send the model calculation data output by the client to a dedicated server in a non-blocking communication manner. The server manages the data buffer and aggregates the model calculation data output by one or more clients. After the server performs data post-processing operations, it outputs the data in a parallel data format and writes it to the target file, thereby realizing asynchronous I / O optimization of the global atmospheric spectrum model.

[0102] Specific limitations regarding the global atmospheric spectrum model I / O optimization system can be found in the limitations of the global atmospheric spectrum model I / O optimization method described above, and will not be repeated here. Each module in the aforementioned global atmospheric spectrum model I / O optimization system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0103] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 11As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When executed by the processor, the computer program implements a global atmospheric spectrum model I / O optimization method. The display screen can be an LCD screen or an e-ink display screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0104] Those skilled in the art will understand that Figure 11 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0105] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to perform the following steps:

[0106] An asynchronous I / O framework integrating the global atmospheric spectrum model and XIOS is constructed. The asynchronous I / O framework consists of a client-server architecture provided by XIOS, with the global atmospheric spectrum model acting as the client in XIOS, and data transmission between the client and server based on buffers.

[0107] During the initialization phase of the global atmospheric spectrum model, communication initialization processing and metadata configuration are performed on the asynchronous I / O framework;

[0108] During the main time step loop phase of the global atmospheric spectrum model, the model calculations are performed by the client. A buffer-based sending optimization mechanism is adopted to send the model calculation data output by the client to a dedicated server in a non-blocking communication manner. The server manages the data buffer and aggregates the model calculation data output by one or more clients. After the server performs data post-processing operations, the data is output in a parallel data format and written to the target file, thereby realizing asynchronous I / O optimization of the global atmospheric spectrum model.

[0109] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0110] An asynchronous I / O framework integrating the global atmospheric spectrum model and XIOS is constructed. The asynchronous I / O framework consists of a client-server architecture provided by XIOS, with the global atmospheric spectrum model acting as the client in XIOS, and data transmission between the client and server based on buffers.

[0111] During the initialization phase of the global atmospheric spectrum model, communication initialization processing and metadata configuration are performed on the asynchronous I / O framework;

[0112] During the main time step loop phase of the global atmospheric spectrum model, the model calculations are performed by the client. A buffer-based sending optimization mechanism is adopted to send the model calculation data output by the client to a dedicated server in a non-blocking communication manner. The server manages the data buffer and aggregates the model calculation data output by one or more clients. After the server performs data post-processing operations, the data is output in a parallel data format and written to the target file, thereby realizing asynchronous I / O optimization of the global atmospheric spectrum model.

[0113] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0114] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0115] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application.

Claims

1. A global atmospheric spectrum model I / O optimization method, characterized in that, The method includes: An asynchronous I / O framework integrating a global atmospheric spectral model and XIOS is constructed; wherein, the asynchronous I / O framework consists of a client-server architecture provided by XIOS, the global atmospheric spectral model acts as a client in XIOS, and data transmission between the client and the server is based on a buffer. During the initialization phase of the global atmospheric spectrum model, the asynchronous I / O framework undergoes communication initialization processing and metadata configuration. During the main time step loop phase of the global atmospheric spectrum model, the client is used to perform global atmospheric spectrum model calculations. A buffer-based sending optimization mechanism is adopted to send the model calculation data output by the client to a dedicated server in a non-blocking communication manner. The server manages the data buffer and aggregates the model calculation data output by one or more clients. After the server performs data post-processing operations, the data is output in a parallel data format and written to the target file, thereby realizing asynchronous I / O optimization of the global atmospheric spectrum model. Specifically, during the main time step loop phase of the global atmospheric spectrum model, the client performs global atmospheric spectrum model calculations, and a buffer-based transmission optimization mechanism is used to send the model calculation data output by the client to a dedicated server in a non-blocking communication manner, including: During the main time step cycle of the global atmospheric spectral model, the client is used to perform calculations of the global atmospheric spectral model, and the calendar of the XIOS simulation is updated using the current time step calculated by the global atmospheric spectral model. Then, iterate through the list of fields of the model calculation data that the client needs to output at the current time step, and determine whether to enable the buffer-based sending optimization mechanism; If not enabled, check whether each field in the field list is marked as active by the server, and directly send the data of the active fields to the server in a non-blocking communication manner. If enabled, for each field in the field list, the corresponding buffer and counter are located according to the field type, and the field data is copied to the specified position in the corresponding buffer. The counter value of the buffer is incremented by 1, and the buffer flag of the buffer is set to true, indicating that there is field data to be sent in the current buffer. Iterate through all buffers. When the buffer flag of the current buffer is true and the corresponding counter value is greater than 0, retrieve the field data in the current buffer in a loop. When the retrieved field is active, send the field data to the server in a non-blocking communication manner. Decrement the counter value of the current buffer by 1 until all field data in the current buffer has been sent. Then reset the buffer flag of the current buffer to False.

2. The method according to claim 1, characterized in that, The data transmission between the client and the server is based on a buffer, including: Within the asynchronous I / O framework, each client is equipped with a dual buffer. When the client performs global atmospheric spectrum model calculations, a dual buffering strategy is adopted. The model calculation data output by the client is placed into one of the corresponding dual buffers. When one buffer is full, the model calculation data in the full buffer is sent to the server, while the other buffer is switched to continuously receive the model calculation data output by the corresponding client.

3. The method according to claim 2, characterized in that, The data transmission between the client and server based on a buffer also includes: Within the asynchronous I / O framework, each server is equipped with a circular buffer. In the current time step of the global atmospheric spectrum model calculation, one segment of the circular buffer acts as a write end, used to receive model calculation data for the current time step sent by a full buffer. The other segment of the circular buffer acts as a read end, used to read model calculation data received in the previous time step and write it to the target file. In the next time step of the global atmospheric spectrum model calculation, the functions of the two segments of the circular buffer are switched, continuously receiving model calculation data output by the client and synchronously outputting data to the target file.

4. The method according to claim 1, characterized in that, During the initialization phase of the global atmospheric spectrum model, the asynchronous I / O framework undergoes communication initialization processing and metadata configuration, including: During the initialization phase of the global atmospheric spectrum model, the global atmospheric spectrum model, acting as a client, calls the XIOS application programming interface to establish a parallel communication context and configure metadata. The metadata configuration includes defining key metadata output by the client and transmitting it to the server in XIOS, including defining the start date and time step of the global atmospheric spectrum model as the calendar of the XIOS simulation, defining the vertical structure of the model as the XIOS axis, and defining the decomposed horizontal grid output by the global atmospheric spectrum model as the XIOS domain.

5. The method according to claim 1, characterized in that, Before iterating through the list of fields of the model calculation data that the client needs to output at the current time step and determining whether to enable the buffer-based send optimization mechanism, the method further includes: The physical field data that the global atmospheric spectrum model needs to output at the current time step is confirmed by XIOS. The physical field data is converted from spectral space to grid space, and the I / O dispatch routine in XIOS is called to integrate the converted physical field data into a field list. The boolean parameter LOPT_SEND used during the execution of the I / O dispatch routine controls whether to enable the buffer-based send optimization mechanism.

6. The method according to claim 1, characterized in that, The server performs data post-processing operations, including: The model calculation data output by one or more clients aggregated by the server are subjected to time averaging and re-gridization.

7. A global atmospheric spectrum model I / O optimization system, characterized in that, The system includes: An asynchronous framework building module is used to build an asynchronous I / O framework integrating a global atmospheric spectrum model and XIOS. The asynchronous I / O framework consists of a client-server architecture provided by XIOS, with the global atmospheric spectrum model acting as a client in XIOS, and data transmission between the client and server based on a buffer. An initialization module is used to perform communication initialization processing and metadata configuration for the asynchronous I / O framework during the initialization phase of the global atmospheric spectrum model. The sending optimization module is used to perform global atmospheric spectrum model calculations using the client during the main time step loop phase of the global atmospheric spectrum model. It employs a buffer-based sending optimization mechanism to send the model calculation data output by the client to a dedicated server in a non-blocking communication manner. The server manages the data buffer and aggregates the model calculation data output by one or more clients. After the server performs data post-processing operations, it outputs the data in a parallel data format and writes it to the target file, thereby realizing asynchronous I / O optimization of the global atmospheric spectrum model. Specifically, during the main time step loop phase of the global atmospheric spectrum model, the client performs global atmospheric spectrum model calculations, and a buffer-based transmission optimization mechanism is used to send the model calculation data output by the client to a dedicated server in a non-blocking communication manner, including: During the main time step cycle of the global atmospheric spectral model, the client is used to perform calculations of the global atmospheric spectral model, and the calendar of the XIOS simulation is updated using the current time step calculated by the global atmospheric spectral model. Then, iterate through the list of fields of the model calculation data that the client needs to output at the current time step, and determine whether to enable the buffer-based sending optimization mechanism; If not enabled, check whether each field in the field list is marked as active by the server, and directly send the data of the active fields to the server in a non-blocking communication manner. If enabled, for each field in the field list, the corresponding buffer and counter are located according to the field type, and the field data is copied to the specified position in the corresponding buffer. The counter value of the buffer is incremented by 1, and the buffer flag of the buffer is set to true, indicating that there is field data to be sent in the current buffer. Iterate through all buffers. When the buffer flag of the current buffer is true and the corresponding counter value is greater than 0, retrieve the field data in the current buffer in a loop. When the retrieved field is active, send the field data to the server in a non-blocking communication manner. Decrement the counter value of the current buffer by 1 until all field data in the current buffer has been sent. Then reset the buffer flag of the current buffer to False.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images