A data assimilation method, device, equipment and storage medium

Abstract

The present application relates to a data assimilation method, device, equipment and storage medium. The data assimilation method comprises the following steps: calculating first position information of observation data to be processed by a plurality of processes in a current observation file; reading the observation data in parallel by the plurality of processes according to the first position information; determining second position information of state data to be processed by the plurality of processes in a current state file, wherein the current state file is a state estimation file obtained based on a historical observation file; reading the state data by the plurality of processes according to the second position information; and performing data assimilation on the state data and the observation data, and updating the current state file. The method provided by the present application can efficiently utilize resources in the data assimilation process, and reduces performance waste.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data assimilation method, device, equipment and storage medium. Background Art

[0002] Data assimilation is a core component of ocean numerical model operations. By integrating observational data to correct simulated state data, it effectively suppresses the accumulation of simulation errors over time. Currently used data assimilation techniques include the Ensemble Kalman Filter (EnKF). EnKF combines ensemble prediction with Kalman filtering, significantly improving simulation accuracy but significantly increasing computational costs.

[0003] As ocean models advance toward higher precision and resolution, data assimilation performance has become a bottleneck restricting computational efficiency and scalability. Existing data assimilation software faces multiple challenges: First, it involves large-scale data I / O operations and complex multi-process communication mechanisms, resulting in a surge in storage and transmission overhead, further increasing the system burden. Second, the nested structure of weakly and strongly dependent computations complicates parallelization. Furthermore, the complex algorithm code structure limits program scalability and execution efficiency, leading to significant performance waste and low execution efficiency. Therefore, optimizing the performance of key data assimilation algorithms based on domestically produced E-class supercomputer systems is crucial. This aims to address the inefficiencies inherent in existing technologies and leverage the advantages of new computing architectures to improve overall performance. However, current implementations still suffer from significant performance waste and inefficient execution, necessitating further exploration of effective solutions to overcome these challenges. Ultimately, how to efficiently utilize resources and minimize performance waste remains a pressing technical challenge. Summary of the Invention

[0004] In order to solve the above technical problems, the embodiments of the present disclosure provide a data assimilation method, apparatus, device and storage medium.

[0005] In a first aspect, an embodiment of the present disclosure provides a data assimilation method, comprising:

[0006] Calculate the first position information of observation data to be processed by multiple processes in the current observation file;

[0007] Reading observation data in parallel according to the first position information through multiple processes;

[0008] Determining second position information of state data to be processed by a plurality of processes in a current state file, where the current state file is a state estimation file obtained based on a historical observation file;

[0009] reading the status data according to the second location information through a plurality of processes;

[0010] Assimilate the status data and observation data and update the current status file.

[0011] Optionally, after reading the observation data in parallel according to the first position information through multiple processes, the method further includes:

[0012] The read observation data is stored in multiple buffers corresponding to multiple processes, and the buffers are used to read and parse the observation data.

[0013] Optionally, reading the status data according to the second location information through multiple processes includes:

[0014] Determining target observation data to be processed by multiple processes among multiple observation data included in the current observation file;

[0015] Determine target state data corresponding to target observation data among a plurality of state data included in the current state file;

[0016] The target state data is read according to the second position information of the target state data by using a preset communication method through multiple processes. The preset communication method refers to a method of directly performing data communication between multiple processes.

[0017] Optionally, among the multiple observation data included in the current observation file, target observation data to be processed by multiple processes are determined, including:

[0018] Calculating processing rounds of the plurality of processes according to a preset number and the number of the plurality of observation data included in the current observation file, wherein the preset number is determined based on the number of the plurality of processes;

[0019] Among the plurality of observation data, a plurality of target observation data to be processed by the plurality of processes in a current processing round is determined.

[0020] Optionally, among the multiple observation data included in the current observation file, target observation data to be processed by multiple processes are determined, including:

[0021] Statistics on the processing time of multiple processes for different observation data;

[0022] Calculate the sum of the processing time of a preset number of observation data, where the sum represents the total time it takes for multiple processes to complete processing the preset number of observation data in one round;

[0023] When the sum is less than the time threshold, a preset number of observation data are determined as target observation data.

[0024] Optionally, after calculating the sum of the processing times of a preset number of observation data, the method further includes:

[0025] When the sum is greater than or equal to the time threshold, the preset number is adjusted until the recalculated sum is less than the recalculated time threshold.

[0026] Optionally, after determining the second position information of the to-be-processed state data of the plurality of processes in the current state file, the method further includes:

[0027] Constructing a dynamic array according to the second position information, wherein the dynamic array refers to an index file of the state data in the current state file;

[0028] Dynamically allocate storage space for index files based on set grid information;

[0029] The state data is read according to the second location information through a plurality of processes, including:

[0030] The state data is read from the second location information in the storage space through multiple processes.

[0031] In a second aspect, an embodiment of the present disclosure provides a data assimilation device, comprising:

[0032] A calculation unit, configured to calculate first position information of observation data to be processed by multiple processes in a current observation file;

[0033] a first reading unit, configured to read the observation data in parallel according to the first position information through a plurality of processes;

[0034] a determining unit, configured to determine second position information of state data to be processed by a plurality of processes in a current state file, where the current state file is a state estimation file obtained based on a historical observation file;

[0035] a second reading unit, configured to read the status data according to the second location information through a plurality of processes;

[0036] The data assimilation unit is used to assimilate the status data and observation data and update the current status file.

[0037] In a third aspect, an embodiment of the present disclosure provides an electronic device, including:

[0038] Memory;

[0039] processor; and

[0040] computer programs;

[0041] The computer program is stored in the memory and is configured to be executed by the processor to implement the method of the first aspect as described above.

[0042] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the method of the first aspect described above when the computer program is executed by a processor.

[0043] The data assimilation method disclosed herein includes: calculating first position information of observation data to be processed by multiple processes in a current observation file; having multiple processes read the observation data in parallel based on the first position information; determining second position information of state data to be processed by multiple processes in a current state file, where the current state file is a state estimation file derived from historical observation files; having multiple processes read the state data based on the second position information; and assimilating the state data and the observation data to update the current state file. The method provided by this application enables efficient resource utilization and reduces performance waste during the data assimilation process. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0045] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0046] Figure 1 A schematic flow chart of an EnKF method provided in an embodiment of the present disclosure;

[0047] Figure 2 A flow chart of a data assimilation solution provided in an embodiment of the present disclosure;

[0048] Figure 3 A schematic diagram of a method of multiple processes reading data in parallel according to an embodiment of the present disclosure;

[0049] Figure 4 A schematic diagram of synchronization waiting time provided by an embodiment of the present disclosure;

[0050] Figure 5 A schematic diagram of a data update process provided in an embodiment of the present disclosure;

[0051] Figure 6 A schematic diagram of the structure of a data assimilation device provided in an embodiment of the present disclosure;

[0052] Figure 7 A schematic structural diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0053] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features therein can be combined with each other in the absence of conflict.

[0054] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0055] Data assimilation plays a crucial role in the operation of ocean numerical models. By integrating observational data to correct model results, it effectively suppresses the tendency for model errors to increase over time. Ocean numerical models utilize mathematical models and computer technology to simulate and predict ocean physical, chemical, and biological processes. Data assimilation combines observational data with state data output by numerical models to produce state data that accurately reflects the real world. Currently used data assimilation techniques include the ensemble Kalman filter and the improved local weighted ensemble Kalman filter (LwEnKF). The EnKF scheme combines the advantages of ensemble prediction and Kalman filtering to quantify uncertainty. The LwEnKF builds on these advantages by introducing localization techniques and a weighting algorithm to further enhance the ability to handle local outliers. While both methods significantly improve simulation accuracy, they also significantly increase computational costs.

[0056] As ocean models advance toward higher accuracy and resolution, data assimilation performance has become a bottleneck restricting computational efficiency and scalability. Existing data assimilation software faces multiple challenges: First, it involves large-scale data I / O operations and complex multi-process communication mechanisms, which increase the system burden. Second, uneven grid density leads to unbalanced load on the central processing unit (CPU), affecting overall computational efficiency. Third, the widespread nesting of weakly and strongly dependent computations complicates parallel optimization. Furthermore, the complex algorithm code structure limits program scalability and execution efficiency. These challenges combine to result in significant performance waste and low execution efficiency. Therefore, optimizing the performance of key data assimilation algorithms based on domestically produced E-class supercomputer systems is crucial. This aims to address the inefficiencies inherent in existing technologies and leverage the advantages of new computing architectures to improve overall performance. However, current implementations still suffer from significant performance waste and inefficient execution, requiring further exploration of effective solutions to overcome these challenges. Ultimately, how to efficiently utilize resources and minimize performance waste remains a pressing technical challenge.

[0057] Example 1:

[0058] To address the above technical issues, the present disclosure provides a data assimilation method that systematically optimizes data assimilation from multiple perspectives, including I / O performance optimization, communication efficiency enhancement, and algorithmic refinement. This method significantly improves process memory access efficiency and computational parallelism, while effectively reducing inter-process synchronization wait times. This is further illustrated in detail through one or more of the following embodiments.

[0059] The data assimilation method provided in the embodiments of the present disclosure is applicable to data assimilation scenarios. This method can be performed by a data assimilation device, which can be implemented using software and / or hardware and integrated into an electronic device. These electronic devices include, but are not limited to, mobile devices such as smartphones, laptops, digital broadcast receivers, personal digital assistants (PDAs), tablet personal computers (Tablet PCs), portable multimedia players (PMPs), in-vehicle terminals (e.g., in-vehicle navigation terminals), wearable devices, and fixed devices such as digital televisions, desktop computers, and smart home appliances.

[0060] Figure 1A flow chart of an EnKF method provided in an embodiment of the present disclosure is provided, specifically a code flow of data assimilation software using an ensemble Kalman filter scheme, wherein the main time-consuming sub-processes are sending and / or receiving observation data and obtaining adjacent state points and observation points of this process, resulting in system performance waste and low execution efficiency.

[0061] Figure 2 A flow chart of a data assimilation solution provided in an embodiment of the present disclosure specifically includes the following steps:

[0062] S201: Calculate the first position information of observation data to be processed by multiple processes in the current observation file.

[0063] As can be understood, the first position information of observation data to be processed by multiple processes within the current observation file is calculated based on the set distribution logic. This means that the position information of each observation data processed by each process within the entire observation file is determined to ensure that each process can read the required observation data. The set distribution logic refers to the logic by which a specific process performs multiple distribution cycles, distributing observation data to multiple processes at once. For example, the logic extracts the portion of data required by a receiving process from the observation file and packages it for transmission. The receiving process then assigns the received portion of data to an array it controls. A process is the basic unit for executing computing tasks, with independent address spaces and system resources. In a distributed computing environment, multiple processes working in parallel can accelerate data processing. Observation data is a record of actual environmental parameters captured by sensors or other measurement devices and is used to calibrate model predictions and improve simulation accuracy. The current observation file refers to a set of all observation data collected at a specific time point or time period, serving as an important input component in the data assimilation process. The first position information refers to the starting byte offset or index value of the observation data to be processed by each process within the current observation file. This is used to locate the starting point for reading observation data and ensure data distribution accuracy. It is understood that the multiple observation data to be processed by a process may not be contiguous within the observation file.

[0064] S202: Read observation data in parallel according to the first position information through multiple processes.

[0065] It is understandable that, based on the above S201, after determining the first position information of at least one observation data to be processed by each process in the entire observation file, multiple processes read at least one observation data at a time in parallel based on the first position information. This method of multiple processes reading multiple observation data in parallel based on the first position information optimizes the efficiency of I / O operations, reduces data redundancy and repeated reading, and thus improves the execution efficiency and resource utilization of the overall data assimilation. Specifically, the MPI_FILE_read_at interface can be used to allow each process to read the observation data at different positions in the observation file at the same time according to the index (first position information), wherein the Message Passing Interface (MPI) is a standard protocol for writing parallel programs, mainly used to realize communication between processes on distributed memory systems.

[0066] Understandably, compared to the prior art, which serially reads all observation data through a specific process, stores the read data in an intermediate array, and then extracts data from the intermediate array and sends it to other processes in a loop, while other processes need to wait and remain idle during the reading process. This process is not only complex but also results in significant performance waste. The method provided by the present disclosure, which allows all processes to read data in parallel, omits the intermediate array storage, data packaging, distribution, and parsing processes, simplifies the reading process, and effectively improves system performance.

[0067] Optionally, after reading the observation data in parallel according to the first position information through multiple processes, the method further includes:

[0068] The read observation data is stored in multiple buffers corresponding to multiple processes, and the buffers are used to read and parse the observation data.

[0069] It is understandable that at least one buffer is set for at least one process, for example, a buffer is set for each process, or a buffer is set for a certain number of processes, or multiple buffers are set for multiple processes, that is, the buffer corresponding to each process and the number of buffers are not limited, and can be determined according to the actual storage situation of the buffer and / or the amount of observation data to be processed by the process, and are not limited here. The following embodiment takes setting a buffer for each process as an example. After the process reads all the observation data of an observation object at a time, it stores it in the corresponding buffer. The number of observation data read by the process each time is not limited. During the reading process, the process will also read and parse the observation data in the buffer to complete the final acquisition of the observation data. The number of observation data read and parsed by the process each time is not limited. For example, part of the observation data in the buffer is read and parsed each time.

[0070] For example, see Figure 3 , Figure 3 A schematic diagram of a plurality of processes reading data in parallel provided in an embodiment of the present disclosure, wherein the plurality of processes are recorded as process 0 to process n. After determining the index of the observation data to be processed by each thread in the observation file or completing the offset calculation, process 0 to process n read the required observation data in parallel based on the index.

[0071] S203: Determine second position information of the to-be-processed status data of the plurality of processes in the current status file.

[0072] Among them, the current state file is a state estimation file obtained based on the historical observation file.

[0073] It is understood that, based on the above S202, a current state file corresponding to the current observation file is obtained, where the current observation file and the current state file can be understood as the actual data set and simulated data set of the same observation object. The current state file includes multiple state data, which can be understood as simulated data at each state point, and the current observation file includes multiple observation data, which can be understood as actual data at each observation point. The observation points and state points can be the same or different. Specifically, the state data represents the state information of the system or model at a certain moment, typically obtained through a series of processing and estimation based on historical observation data. The current state file can be understood as a file generated after state estimation based on the historical observation files. It contains the latest system state information and serves as a summary and prediction basis for the system's past behavior. The second position information refers to the position information of the state data processed by each process within the entire state file, namely, the starting byte offset or index value in the current state file. For example, in an ocean numerical simulation application, if future ocean conditions are to be predicted, state estimation must first be performed based on the historical observation files to generate the current state file. Then, to improve computational efficiency, this state file is divided into several parts, each of which is processed by a different process. Determines the secondary location of each process's pending status data within the status file. This helps optimize I / O operations and reduce unnecessary data transfer, thereby improving overall processing speed and resource utilization efficiency.

[0074] S204: Read status data according to the second location information through multiple processes.

[0075] It is understandable that, based on the above-mentioned S203, improving parallelism is crucial for MPI programs' operational efficiency. Before each process loop processes at least one observation data read, that is, before executing the loop calculation in S205, it preferentially obtains at least one state data corresponding to the at least one observation data based on the second position information. Specifically, the position information is first transmitted, allowing the process to preferentially read all state data and all observation data to be processed in a processing round, and the expansion information is then transmitted a second time. Because the position information is independent, the position information of the required state points and observation points remains unchanged throughout the calculation process when searching for adjacent points, and there are no data dependencies. Therefore, the first communication changes the communication method, allowing all processes to simultaneously package the position information of the observation points and state points to be processed, thereby reading all observation data and all state data. All processes communicate in an all-to-all manner, which does not significantly increase additional communication overhead. Furthermore, prioritizing the reading of the position index of at least one data point before looping through the data significantly improves the algorithm's parallelism and indirectly reduces the idle time of the process during synchronization waiting, thereby improving overall computational efficiency. This is explained in detail in the following steps.

[0076] Optionally, the state data is read according to the second location information by multiple processes, which can be specifically implemented by the following steps:

[0077] Among the multiple observation data included in the current observation file, target observation data to be processed by multiple processes are determined; among the multiple status data included in the current status file, target status data corresponding to the target observation data are determined; a preset communication method is adopted through multiple processes to read the target status data according to the second position information of the target status data. The preset communication method refers to a method for directly communicating data between multiple processes.

[0078] As can be understood, after a process stores multiple observation data read in parallel in a buffer, it determines at least one target observation data to be processed from the multiple observation data. Specifically, each process can process at least a portion of the data in each round. The specific method for determining this is not limited and can be based on a selection criterion or allocation strategy. For example, a certain number of observation data can be processed each time. The target observation data refers to at least one observation data selected from the current observation file that is relevant to a specific task or computational requirement. Subsequently, target state data related to the target observation data is determined from the multiple state data. Specifically, for each target observation data point, there are one or more associated state data points. These state data reflect the estimated state of the system at the same point in time or within a certain time period. Subsequently, multiple processes read the target state data based on the second location information of the target state data using a preset communication method. The preset communication method refers to a direct data exchange mechanism between multiple processes, such as the aforementioned all-to-all method. In this mode, processes can communicate directly with other processes without going through other nodes. This method not only improves data processing efficiency but also ensures data consistency and accuracy, providing strong support for large-scale parallel computing.

[0079] Optionally, among the multiple observation data included in the current observation file, target observation data to be processed by multiple processes may be determined, which may be specifically achieved by the following steps:

[0080] According to the preset number and the number of multiple observation data included in the current observation file, the processing rounds of multiple processes are calculated, and the preset number is determined based on the number of multiple processes; among the multiple observation data, the multiple target observation data to be processed by the multiple processes in the current processing round are determined.

[0081] Understandably, given the potentially large size of observation data, processing all data at once could result in excessive memory overhead (equivalent to each process needing to store all observation data). Therefore, the number of processing rounds for multiple processes is calculated based on a preset number and the actual number of observation data. This means that the processing task is divided into multiple rounds. For example, one process processes 1,000 observation data at a time. Specifically, the preset number can be an integer multiple of the total number of processes, where n is the total number of processes. Each round can process 2n observation data. This 2n observation data can be divided equally among the n processes or dynamically allocated based on the processing efficiency of each process. The specific allocation method is not limited. This multi-round processing approach ensures parallel efficiency while keeping additional memory overhead to a minimum. Subsequently, when determining the target observation data, the total number of processing rounds can also be considered, for example, each round can be allocated an equal amount of observation data to be processed.

[0082] Optionally, among the multiple observation data included in the current observation file, target observation data to be processed by multiple processes may be determined, which may be specifically achieved by the following steps:

[0083] Count the processing time of multiple processes for processing different observation data; calculate the sum of the processing time of a preset number of observation data, where the sum represents the total time it takes for multiple processes to process the preset number of observation data in one round; when the sum is less than a time threshold, determine the preset number of observation data as target observation data; or, when the sum is greater than or equal to the time threshold, adjust the preset number until the recalculated sum is less than the recalculated time threshold.

[0084] Understandably, the aforementioned round-by-round processing mechanism may involve hidden waiting times for at least one process. When determining target observations in rounds, the waiting times of different processes must be considered. The length of a process's waiting time is closely related to the amount of data processed in each round. Because different processes have varying workloads for each observation, the total processing time for a round is equivalent to the time it takes for the slowest process to complete processing all observations. Other processes must wait for the slowest process to complete processing all their corresponding observations, resulting in synchronization waiting times for other processes. Therefore, when a process processes multiple observations at once, the synchronization waiting time depends on the slowest process to complete processing all observations in a round. Specifically, assuming sufficient memory resources, the time required for each process to process different observations is calculated. This means that the specific time taken by each process to process a specific observation is recorded. Subsequently, based on these processing times, the total time required for all processes to process a preset number of observations (i.e., the "sum") is calculated. This sum represents the total time it takes for all processes to complete processing the specified number of observations in a round.

[0085] For example, see Figure 4 , Figure 4A schematic diagram of synchronization waiting time provided for an embodiment of the present disclosure, wherein the processing time for process 0 to process a single observation data A (observation A) is 10s, the processing time for process 1 to process a single observation data A is 4s, the processing time for process 0 to process a single observation data B (observation B) is 5s, and the processing time for process 1 to process a single observation data B is 9s. If each process processes only one observation data in one round, in the first round of processing observation data A, process 0 is the slowest process, process 1 needs to wait for 6s, and the processing time of the first round is 10s. In the second round of processing observation data B, process 1 is the slowest process, process 0 needs to wait for 4s, and the processing time of the first round is 9s. The total processing time of the two rounds is 19s. Not only is the waiting time for each process relatively long, but the total processing time of multiple rounds is also relatively long. In this case, it is preferred to count the processing time of each observation data, and consider processing multiple observation data at a time to reduce the synchronization waiting time, such as Figure 5 As shown, process 0 takes a total of 15 seconds (10 seconds + 5 seconds) to process two observations (observation A and observation B), while process 1 takes a total of 13 seconds (4 seconds + 9 seconds). Process 0 is the slowest process. To complete this processing round, process 1 only needs to wait 2 seconds, waiting for process 0 to finish processing the data. This synchronization wait time is 2 seconds. The approach where each process processes observations A and B simultaneously is 4 seconds faster than the approach where each process processes one observation at a time. Therefore, when memory resources are sufficient, processing more data per round can hide more synchronization wait time, ensuring that the system can efficiently complete tasks within the time limit while maximizing resource utilization, significantly improving overall performance. This approach strikes a balance between memory overhead and computational efficiency, avoiding excessive memory usage while reducing synchronization wait time by increasing the amount of data processed per round. Ultimately, this achieves higher parallel efficiency and better performance.

[0086] It can be understood that after more data is processed in each round to reduce the hidden waiting time, the calculated sum value can also be compared with the time threshold, that is, the maximum processing time that the system can accept can be limited. The time threshold is the maximum processing time that the system can accept in an ideal state, which can be determined by the user as needed. If the sum value is less than the time threshold, it is considered that the current preset number of observation data can be used as target observation data, that is, these data can be effectively processed by all processes without exceeding the expected time. If the sum value is greater than or equal to the time threshold, it indicates that the processing efficiency of all processes under the current configuration cannot meet the requirements, and the preset number needs to be adjusted. That is, in the case that the sum value does not meet the condition, the total processing time can be reduced by reducing the preset number of observation data, until the recalculated sum value is lower than the time threshold. This process may need to be iterated repeatedly until the optimal amount of observation data is found, so that the processing time meets the efficiency requirements and is as close as possible to but not exceeding the time threshold.

[0087] Optionally, after determining the second position information of the state data of the plurality of processes in the current state file, the method further comprises:

[0088] According to the second position information, a dynamic array is constructed, wherein the dynamic array refers to an index file of the state data in the current state file; a storage space is dynamically allocated for the index file according to the set grid information; and the state data is read by the plurality of processes according to the second position information, including reading the state data from the storage space at the second position information by the plurality of processes.

[0089] Understandably, the array used to store neighboring point indices and distances is declared to be the size of the total number of state elements within the process, but this approach results in significant space waste. Due to the limitation of the proximity radius, an observation point will only have neighboring points within a limited number of grids, so such a large intermediate array is completely unnecessary. To address this problem, the present disclosure constructs a dynamic array based on the second position information. The dynamic array records the specific location information of the state data in the current state file, facilitating rapid location and access. In other words, the intermediate array is converted to a dynamic array. When initializing the grid, storage space is dynamically allocated for the index file based on the set grid information. In other words, memory resources are rationally arranged to ensure that the index file can efficiently support subsequent data access operations. The set grid information refers to the spatial division information of the model, for example, the grid division in the ocean numerical model determines how the state data is organized and managed. Subsequently, the process can read the state data based on the second position information in this storage space, thereby avoiding unnecessary I / O operations and potential data redundancy issues. This method of dynamically allocating space for neighboring point indexes not only significantly reduces space consumption but also improves memory access efficiency. In addition, since the size of the intermediate array is reduced exponentially, it is no longer necessary to frequently replace the cache when accessing the array storing the index file, thereby greatly improving the cache hit rate and achieving faster memory access speed.

[0090] S205: Assimilate the status data and observation data and update the current status file.

[0091] It is understandable that, based on the above S204, data assimilation is performed based on the first position information, the second position information, the state data, and the observation data. Data assimilation is a method of incorporating observation data into a numerical model to improve the accuracy of model predictions. By considering the error between the observation data and its uncertainty and the state data predicted by the model, an optimized state estimate is provided. Specifically, the difference between the model prediction and the actual observation data and its uncertainty can be evaluated. Based on the observation data and its error, the state data of the model is adjusted to obtain a state estimate that is closer to the actual situation. Subsequently, the updated state data is written back to the current state file, replacing the original state estimate, ensuring that subsequent simulations and predictions are based on the latest and most accurate state information.

[0092] For example, see Figure 5 , Figure 5 A data update process diagram provided in an embodiment of the present disclosure specifically includes the following steps:

[0093] 1) After reading the observation data and status data, each process uses an all-to-all method to send and receive location information, where location information refers to the location of the observation point and the status point. 2) Based on this location information, it finds nearby observation objects (observation points) and nearby status points. 3) It determines whether the current observation point (observation element) is local. 4) If the current observation point is local, it calculates the expansion information and broadcasts it to all processes via MPI_BCAST. 5) If the current observation point is not local, it waits to receive the expansion information of the current observation point. 6) After all processes have received the expansion information, they update the status point and observation point, and then loop to process the next observation point.

[0094] It is understandable that each process will cyclically process multiple observation points in one processing round, but before cyclically processing the observation points, the location information of all observation points and all status points will be obtained first. Each cycle will only process one observation point (that is, the current observation point), and there is no need to obtain the location information of the current observation point and the adjacent status points.

[0095] A data assimilation method provided by the embodiment of the present disclosure, in terms of I / O optimization, by calculating the position information of the observation data in the observation file, realizes a parallel optimization scheme of multiple processes based on MPI-IO, significantly improves the operating efficiency of the observation data input module, and basically solves the performance bottleneck problem in the data reading process. In terms of communication optimization, the communication overhead is greatly reduced by adopting strategies such as data pre-packaging, optimizing the MPI communication method, and reducing the synchronization waiting time. In terms of algorithm optimization, the computing efficiency is significantly improved by separating the computing parts without data dependence and improving the adjacent array reuse mechanism. In addition, according to the data characteristics and task division characteristics, the size of the intermediate array is dynamically adjusted, which further improves the memory access efficiency while reducing the space complexity. It not only brings significant performance improvement to the data assimilation software, but also lays an important foundation for realizing higher precision and higher resolution ocean prediction models.

[0096] Example 2:

[0097] Figure 6 The data assimilation device provided in the embodiment of the present disclosure can execute the processing flow provided in the data assimilation method embodiment, such as Figure 6 As shown, the apparatus 600 includes:

[0098] A calculation unit 601 is used to calculate the first position information of the observation data to be processed by multiple processes in the current observation file;

[0099] A first reading unit 602 is configured to read the observation data in parallel according to the first position information through multiple processes;

[0100] A determining unit 603 is configured to determine second position information of the to-be-processed state data of the plurality of processes in a current state file, where the current state file is a state estimation file obtained based on the historical observation file;

[0101] A second reading unit 604 is configured to read the status data according to the second location information through multiple processes;

[0102] The data assimilation unit 605 is used to assimilate the state data and the observation data and update the current state file.

[0103] Optionally, the device 600 is further configured to:

[0104] The read observation data is stored in multiple buffers corresponding to multiple processes, and the buffers are used to read and parse the observation data.

[0105] Optionally, the second reading unit 604 is configured to:

[0106] Determining target observation data to be processed by multiple processes among multiple observation data included in the current observation file;

[0107] Determine target state data corresponding to target observation data among a plurality of state data included in the current state file;

[0108] The target state data is read according to the second position information of the target state data by using a preset communication method through multiple processes. The preset communication method refers to a method of directly performing data communication between multiple processes.

[0109] Optionally, the second reading unit 604 is configured to:

[0110] Calculating processing rounds of the plurality of processes according to a preset number and the number of the plurality of observation data included in the current observation file, wherein the preset number is determined based on the number of the plurality of processes;

[0111] Among the plurality of observation data, a plurality of target observation data to be processed by the plurality of processes in a current processing round is determined.

[0112] Optionally, the second reading unit 604 is configured to:

[0113] Statistics on the processing time of multiple processes for different observation data;

[0114] Calculate the sum of the processing time of a preset number of observation data, where the sum represents the total time it takes for multiple processes to complete processing the preset number of observation data in one round;

[0115] When the sum is less than the time threshold, a preset number of observation data are determined as target observation data.

[0116] Optionally, the second reading unit 604 is configured to:

[0117] When the sum is greater than or equal to the time threshold, the preset number is adjusted until the recalculated sum is less than the recalculated time threshold.

[0118] Optionally, the device 600 is further configured to:

[0119] Constructing a dynamic array according to the second position information, wherein the dynamic array refers to an index file of the state data in the current state file;

[0120] Dynamically allocate storage space for index files based on set grid information;

[0121] Optionally, the second reading unit 604 is further configured to:

[0122] The state data is read from the second location information in the storage space through multiple processes.

[0123] Figure 6 The data assimilation device of the illustrated embodiment can be used to implement the technical solution of the above-mentioned method embodiment. Its implementation principle and technical effects are similar and will not be described in detail here.

[0124] Example 3:

[0125] Figure 7 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. Figure 7 , which shows a schematic structural diagram of an electronic device 700 suitable for implementing the embodiments of the present disclosure. The electronic device 700 in the embodiments of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (such as in-vehicle navigation terminals), wearable electronic devices, and fixed terminals such as digital TVs, desktop computers, smart home devices, and the like. Figure 7 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0126] like Figure 7As shown, electronic device 700 may include a processing device 701 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes to implement the data assimilation method of the embodiments described in this disclosure based on programs stored in read-only memory (ROM) 702 or programs loaded from storage device 708 into random access memory (RAM) 703. RAM 703 also stores various programs and data required for the operation of electronic device 700. Processing device 701, ROM 702, and RAM 703 are interconnected via bus 704. An input / output (I / O) interface 705 is also connected to bus 704.

[0127] Typically, the following devices may be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the electronic device 700 to communicate with other devices wirelessly or by wire to exchange data. Figure 7 The electronic device 700 is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.

[0128] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart, thereby implementing the data assimilation method described above. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 709, or installed from the storage device 708, or installed from the ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0129] It should be noted that the computer-readable medium described above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. Computer-readable storage media may include, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or components, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such a propagated data signal may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wire, optical cable, RF (radio frequency), or any suitable combination thereof.

[0130] In some embodiments, the client and server can communicate using any currently known or later developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or later developed network.

[0131] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0132] Optionally, when the above one or more programs are executed by the electronic device, the electronic device may also execute other steps described in the above embodiments.

[0133] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including, but not limited to, object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0134] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0135] The units involved in the embodiments described in this disclosure may be implemented in software or hardware, wherein the name of a unit does not necessarily limit the unit itself.

[0136] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0137] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fibers, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0138] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or gateway that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or gateway. In the absence of further restrictions, the elements defined by the sentence "including a data assimilation" do not exclude the presence of additional identical elements in the process, method, article or gateway that includes the elements.

[0139] The foregoing description is intended only to provide specific embodiments of the present disclosure, intended to enable those skilled in the art to understand and implement the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the embodiments described herein, but rather to be construed in the broadest manner consistent with the principles and novel features disclosed herein.

Claims

1. A data assimilation method, characterized in that: include: Calculate the first position information of the to-be-processed observation data corresponding to the multiple processes in the current observation file; Reading observation data in parallel according to the first position information by the multiple processes; Determine second position information of the to-be-processed state data corresponding to the multiple processes in a current state file, where the current state file is a state estimation file obtained based on a historical observation file; reading, by the plurality of processes, state data according to the second location information; Assimilate the state data and the observation data, and update the current state file; The step of reading the status data according to the second location information through the multiple processes includes: Determining target observation data to be processed by the multiple processes from among the multiple observation data included in the current observation file, including: counting processing time of the multiple processes for processing different observation data; calculating the sum of processing time of the slowest process among the multiple processes for processing all observation data, wherein the sum represents the total time it takes for the slowest process to process all observation data in one round; and determining a preset number of observation data as target observation data when the sum is less than a time threshold, wherein the observation data are processed by the multiple processes according to the acquired processing rounds, the preset number refers to the total number of observation data processed by the multiple processes in one round, and the number of observation data processed by at least one process in one round is greater than 1; Determining target state data corresponding to the target observation data among a plurality of state data included in the current state file; The target state data is read according to the second position information of the target state data by the multiple processes using a preset communication method, where the preset communication method refers to a method of directly performing data communication between the multiple processes.

2. The method according to claim 1, characterized in that After the observation data is read in parallel by the multiple processes according to the first position information, the method further includes: The read observation data is stored in a plurality of buffers corresponding to the plurality of processes, and the buffers are used to read and parse the observation data.

3. The method according to claim 1, characterized in that Determining the target observation data to be processed by the multiple processes from the multiple observation data included in the current observation file includes: Calculating processing rounds of the multiple processes according to a preset number and the number of the multiple observation data included in the current observation file, wherein the preset number is determined based on the number of the multiple processes; Among the plurality of observation data, a plurality of target observation data to be processed by the plurality of processes in a current processing round are determined.

4. The method according to claim 1, wherein After calculating the sum of the processing time of the slowest process among the multiple processes to complete processing all observation data, the method further includes: In a case where the sum is greater than or equal to the time threshold, the preset number is adjusted until the recalculated sum is less than the recalculated time threshold.

5. The method according to claim 1, wherein After determining the second position information of the plurality of process to-be-processed state data in the current state file, the method further includes: constructing a dynamic array according to the second position information, wherein the dynamic array constitutes an index file of the state data in the current state file; Dynamically allocating storage space for the index file according to set grid information, wherein the set grid information refers to spatial division information of the numerical model for organizing and managing the state data; The reading the status data according to the second location information through the multiple processes includes: The state data is read from the second location information in the storage space through the multiple processes.

6. A data assimilation device, characterized in that: include: A calculation unit, configured to calculate first position information of to-be-processed observation data corresponding to a plurality of processes in a current observation file; a first reading unit, configured to read the observation data in parallel according to the first position information through the multiple processes; A determining unit, configured to determine second position information of the to-be-processed state data corresponding to the plurality of processes in a current state file, wherein the current state file is a state estimation file obtained based on a historical observation file; a second reading unit, configured to read status data according to the second location information through the multiple processes; a data assimilation unit, configured to assimilate the state data and the observation data, and update the current state file; Wherein, the second reading unit includes: Among the multiple observation data included in the current observation file, the target observation data to be processed by the multiple processes are determined, including: counting the processing time of the multiple processes for processing different observation data; calculating the sum of the processing time of the slowest process among the multiple processes to process all observation data, wherein the sum represents the total time it takes for the slowest process to process all observation data in one round; when the sum is less than the time threshold, a preset number of observation data are determined as target observation data, wherein the observation data are processed by the multiple processes according to the acquired processing rounds, the preset number refers to the total number of observation data processed by the multiple processes in one round, and the number of observation data processed by at least one process in one round is greater than 1; among the multiple status data included in the current status file, the target status data corresponding to the target observation data is determined; and the target status data is read according to the second position information of the target status data by the multiple processes using a preset communication method, wherein the preset communication method refers to a method for directly communicating data between the multiple processes.

7. An electronic device, characterized in that: include: Memory; processor; as well as computer programs; The computer program is stored in the memory and is configured to be executed by the processor to implement the data assimilation method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the data assimilation method according to any one of claims 1 to 5 are implemented.

Images