A scientific data condition filtering method for optimizing memory peak

Abstract

The application discloses a scientific data condition filtering method for optimizing memory peak, and relates to the technical field of scientific data post-processing. According to input scientific data sets and data filtering conditions, the overall dependency relationship of each data member is abstractly modeled in the process of determining a plurality of data members for performing a condition filtering task on the input scientific data sets, and the overall dependency relationship between each data member is taken as the basis for managing the life cycle of each element in the processing process. According to the overall dependency relationship of each data member in the condition filtering process, each data member is sequentially subjected to copy construction and memory resource release, so that dynamic task scheduling of various condition filtering programs is realized. The method can reduce the memory peak overhead in the algorithm execution process under the premise of ensuring the correctness of the condition filtering, and is favorable for improving the applicability and stability of the condition filtering method in a large-scale data processing scene.

Description

Technical Field

[0001] This application relates to the field of scientific data post-processing technology, and in particular to a scientific data conditional filtering method that optimizes memory peaks. Background Technology

[0002] With the rapid development of scientific computing technologies such as computational fluid dynamics, computational structural mechanics, and multiphysics coupling simulation, the scale and complexity of simulations continue to increase, resulting in scientific data characterized by large volume, complex structure, and diverse data types. For such scientific data, post-processing is typically required to filter, split, and reassemble the raw data to meet application needs such as component analysis, local visualization, and physical quantity statistics.

[0003] Conditional filtering methods are widely used in scientific data post-processing. Conditional filtering typically refers to extracting subsets of data from the original dataset based on preset conditions (such as element labels, material numbers, physical region identifiers, or threshold ranges), or splitting the original dataset into multiple non-overlapping subsets. This type of method plays an important role in scenarios such as unstructured mesh data processing, simulation component separation, and multi-material data analysis.

[0004] In order to avoid damaging the original data and ensure the correct execution of subsequent algorithms, existing conditional filtering methods typically require copying or reconstructing the geometric, topological, and auxiliary data of the original scientific data to generate a new dataset. For large-scale unstructured or mixed mesh data, this copying process involves the creation and storage of a large amount of data, such as vertex coordinates, cell indices, filter condition arrays, and intermediate auxiliary data, thus consuming significant memory resources during execution. Furthermore, existing technologies often employ fixed-sequence or staged data copying strategies, with various types of data generated and stored sequentially according to a preset process. Their lifecycle management is usually coarse-grained, controlled at the algorithm stage level. During processing, multiple types of data often need to be retained simultaneously, leading to high peak memory usage during algorithm execution. This peak memory usage problem is particularly pronounced when dealing with ultra-large-scale scientific data, easily causing memory shortages, frequent memory swapping, or even abnormal program termination, thus limiting the applicability and processing efficiency of conditional filtering methods in practical engineering applications. Summary of the Invention

[0005] This application addresses the aforementioned problems and technical requirements by proposing a scientific data conditional filtering method that optimizes memory peak values. The technical solution of this application is as follows: A scientific data conditional filtering method for optimizing memory peaks, the scientific data conditional filtering method comprising: Obtain the input scientific dataset to be filtered and the data filtering conditions used in the conditional filtering task; Based on the input scientific dataset and data filtering conditions, multiple data members are identified during the conditional filtering task on the input scientific dataset. Each data member is a data component that needs to be copied or reconstructed during the conditional filtering operation. In the conditional filtering task performed on the input scientific dataset according to the data filtering conditions, each data member is copied and constructed and memory resources are released in sequence according to the overall dependency relationship of each data member in the conditional filtering process, until the conditional filtering operation is completed and the output scientific dataset is generated.

[0006] A further technical solution involves sequentially copying and constructing each data member based on the overall dependency relationship between them, including: The overall dependency relationship between each data member is modeled as a dependency graph, and the dependency graph is topologically sorted to obtain the execution order of data members that satisfy the overall dependency relationship constraints. In the execution order of data members, each data member is ordered in order of its overall dependency relationship with other data members from weakest to strongest. Each data member is copied or constructed sequentially according to the execution order of the data members.

[0007] A further technical solution involves releasing memory resources for each data member sequentially based on the overall dependency relationship between data members, including: After copying or constructing a data member, check whether the current data member is still depended on by other unprocessed data members; when it is determined that the current data member is no longer depended on by other unprocessed data members, control the release of the memory resources occupied by the current data member.

[0008] The further technical solution involves determining the overall dependencies between data members, including: Identify the overall dependencies between data members based on the reference relationships, construction relationships, or data dependencies between them during the conditional filtering operation.

[0009] Further technical solutions include: the scientific data conditional filtering method also includes: The input scientific dataset is divided into multiple input data blocks according to the set maximum number of blocks, and a conditional filtering task is performed on each input data block according to the data filtering conditions.

[0010] The further technical solution involves performing conditional filtering tasks on each input data block according to the data filtering conditions, including: The conditional filtering task is executed in parallel on multiple input data blocks according to the data filtering conditions, and when the conditional filtering task is executed on a single input data block according to the data filtering conditions, multiple data members are copied and constructed in parallel.

[0011] The further technical solution is that the multiple data members in the process of performing conditional filtering on the input scientific dataset include input data members for representing the input scientific dataset, intermediate data members for supporting conditional filtering calculations, and output data members for constituting the conditional filtering results.

[0012] Its further technical solution involves determining multiple data members during the conditional filtering task on the input scientific dataset based on the input scientific dataset and data filtering conditions, including: Determine the data structure type of the output scientific dataset based on the data structure type of the input scientific dataset and the data filtering conditions; Based on the data structure type of the input scientific dataset, the data filtering conditions, and the data structure type of the output scientific dataset, determine multiple data members in the process of performing conditional filtering on the input scientific dataset.

[0013] A further technical solution is to input scientific datasets as gridless data, structured grid data, or unstructured grid data.

[0014] The further technical solution is that the data content in the input scientific dataset includes at least one of point set, cell set, vertex data, and cell data, where vertex data is attribute data associated with points and cell data is attribute data associated with cells.

[0015] The beneficial technical effects of this application are: This application discloses a scientific data conditional filtering method that optimizes memory peaks. After identifying each data member in the process of performing conditional filtering on the input scientific dataset, the method abstracts and models the overall dependency relationship between each data member. The overall dependency relationship between each data member is used as the basis for managing the life cycle of each element in the processing process, thereby realizing dynamic task scheduling for various conditional filtering programs. This method can reduce the peak memory overhead during algorithm execution while ensuring the correctness of conditional filtering, which is beneficial to improving the applicability and stability of conditional filtering methods in large-scale data processing scenarios.

[0016] This method can realize the dynamic release and reuse of memory resources, thereby achieving dynamic management of memory resources. Moreover, this scheduling method is applicable to various conditional filtering processes and can be applied in different types of conditional filtering algorithms. Attached Figure Description

[0017] Figure 1 This is a flowchart of a scientific data conditional filtering method in one embodiment of this application.

[0018] Figure 2This is a schematic diagram of data member dependencies and construction relationships in an example of a grid decomposition conditional filtering task in this application. Detailed Implementation

[0019] The specific embodiments of this application will be further described below with reference to the accompanying drawings.

[0020] This application discloses a scientific data conditional filtering method to optimize memory peaks. Please refer to [link / reference]. Figure 1 The flowchart shown illustrates that this scientific data conditional filtering method includes: Step 110: Obtain the input scientific dataset to be filtered and the data filtering conditions used in the conditional filtering task.

[0021] The original input scientific dataset can be meshless data (particle data), structured mesh data, or unstructured mesh data, and its data content includes at least one of point sets, cell sets, vertex data, and cell data. Vertex data is attribute data associated with points, and cell data is attribute data associated with cells.

[0022] The data filtering conditions used include filtering according to at least one of the following: unit label, number, region identifier, and threshold range.

[0023] Step 120: Determine multiple data members in the process of performing conditional filtering on the input scientific dataset based on the input scientific dataset and data filtering conditions.

[0024] Each data member identified in this step is a data component that needs to be copied or reconstructed from the input scientific dataset during the conditional filtering operation in order to preserve the integrity of the original input scientific dataset and the correctness of subsequent algorithms. This includes, but is not limited to, coordinates, indices, scalar fields, label arrays, and mapping tables.

[0025] When identifying data members, the data structure type of the output scientific dataset is first determined based on the data structure type of the input scientific dataset and the data filtering conditions. For example, it is determined whether the output scientific dataset contains points, cells, and their corresponding attribute data. Then, based on the data structure type of the input scientific dataset, the data filtering conditions, and the data structure type of the output scientific dataset, multiple data members can be identified during the conditional filtering task on the input scientific dataset. The specific identification method can be determined by referring to existing conditional filtering processes.

[0026] The data members identified differ across conditional filtering tasks, but the data members involved in performing a conditional filtering task include the following categories: input data members that characterize the input scientific dataset; intermediate data members that support the conditional filtering calculation, such as the filter condition array and intermediate auxiliary data; and output data members that constitute the conditional filtering result.

[0027] For example, in one instance, taking a decomposition conditional filtering task for unstructured grid data as an example, the identified data members include input data members, intermediate data members, and output data members, such as... Figure 2 As shown, the intermediate data members include a subset point index mapping table and a subset cell index table. The names of the data members shown in this example are only for describing their functional roles and do not constitute a limitation on the specific data structure or implementation method.

[0028] Step 130: During the process of performing conditional filtering on the input scientific dataset according to the data filtering conditions, each data member is copied and constructed and memory resources are released in sequence according to the overall dependency relationship of each data member in the conditional filtering process, until the conditional filtering operation is completed and the output scientific dataset is generated.

[0029] First, based on the reference relationships, construction relationships, or data dependencies between data members during the conditional filtering operation, identify the overall dependencies between data members.

[0030] Then, the overall dependencies between data members are modeled as a dependency graph, and the dependency graph is topologically sorted to obtain the execution order of data members that satisfies the overall dependency constraints. In the execution order, each data member is ordered in ascending order of its overall dependency with other data members. Finally, each data member is copied or constructed sequentially according to the determined execution order.

[0031] After copying or constructing a data member, it checks whether the current data member is still depended on by other unprocessed data members. If it is determined that the current data member is no longer depended on by other unprocessed data members, it releases the memory resources occupied by the current data member to reduce the overall memory usage peak during condition filtering.

[0032] The above steps continue to be executed until all data members involved in conditional filtering have been processed. At this point, the conditional filtering operation is considered complete, and one or more output scientific datasets that meet the data filtering conditions are generated. The conditional filtering process then ends.

[0033] Alternatively, in another embodiment, after dividing the input scientific dataset into multiple input data blocks according to a set maximum number of blocks, a conditional filtering task is performed on each input data block according to the data filtering conditions. After completing the data block processing, each input data block contains fewer units, thereby allowing the use of a shorter unit index representation, which helps to further reduce the overall memory overhead of the data.

[0034] Furthermore, when performing conditional filtering tasks on each input data block according to the data filtering conditions, the conditional filtering tasks are performed in parallel on multiple input data blocks. Moreover, when performing conditional filtering tasks on a single input data block according to the data filtering conditions, multiple data members are copied and constructed in parallel. This achieves coarse-grained parallelism based on data block partitioning and fine-grained parallelism based on the copying of specific data members, further optimizing the conditional filtering performance.

[0035] The above descriptions are merely preferred embodiments of this application, and this application is not limited to the above embodiments. It is understood that other improvements and variations that can be directly derived or conceived by those skilled in the art without departing from the spirit and concept of this application should be considered to be included within the protection scope of this application.

Claims

1. A scientific data conditional filtering method for optimizing memory peaks, characterized in that, The scientific data conditional filtering method includes: Obtain the input scientific dataset to be filtered and the data filtering conditions used in the conditional filtering task; Based on the input scientific dataset and data filtering conditions, multiple data members are identified during the conditional filtering task on the input scientific dataset. Each data member is a data component that needs to be copied or reconstructed during the conditional filtering operation. In the conditional filtering task performed on the input scientific dataset according to the data filtering conditions, each data member is copied and constructed and memory resources are released sequentially according to the overall dependency relationship of each data member in the conditional filtering process, until the conditional filtering operation is completed and the output scientific dataset is generated.

2. The scientific data conditional filtering method according to claim 1, characterized in that, Based on the overall dependencies between data members, each data member is copied and constructed sequentially, including: The overall dependency relationship between each data member is modeled as a dependency graph, and the dependency graph is topologically sorted to obtain the execution order of data members that satisfies the overall dependency relationship constraints. In the execution order of data members, each data member is ordered in order of increasing overall dependency relationship with other data members. Each data member is copied or constructed sequentially according to the execution order of the data members.

3. The scientific data conditional filtering method according to claim 1, characterized in that, Based on the overall dependency relationship between data members, memory resources are released sequentially for each data member, including: After copying or constructing a data member, check whether the current data member is still depended on by other unprocessed data members; when it is determined that the current data member is no longer depended on by other unprocessed data members, control the release of the memory resources occupied by the current data member.

4. The scientific data conditional filtering method according to claim 1, characterized in that, Determining the overall dependencies between data members includes: Identify the overall dependencies between data members based on the reference relationships, construction relationships, or data dependencies between them during the conditional filtering operation.

5. The scientific data conditional filtering method according to claim 1, characterized in that, The scientific data conditional filtering method also includes: The input scientific dataset is divided into multiple input data blocks according to the set maximum number of blocks, and a conditional filtering task is performed on each input data block according to the data filtering conditions.

6. The scientific data conditional filtering method according to claim 5, characterized in that, Performing conditional filtering tasks on each input data block according to the aforementioned data filtering conditions includes: The conditional filtering task is executed in parallel on multiple input data blocks according to the data filtering conditions, and when the conditional filtering task is executed on a single input data block according to the data filtering conditions, multiple data members are copied and constructed in parallel.

7. The scientific data conditional filtering method according to claim 1, characterized in that, The multiple data members in the process of performing a conditional filtering task on an input scientific dataset include input data members that characterize the input scientific dataset, intermediate data members that support the conditional filtering calculation, and output data members that constitute the conditional filtering result.

8. The scientific data conditional filtering method according to claim 1, characterized in that, Based on the input scientific dataset and data filtering criteria, several data members are identified during the process of performing a conditional filtering task on the input scientific dataset, including: Determine the data structure type of the output scientific dataset based on the data structure type of the input scientific dataset and the data filtering conditions; Based on the data structure type of the input scientific dataset, the data filtering conditions, and the data structure type of the output scientific dataset, determine multiple data members in the process of performing conditional filtering on the input scientific dataset.

9. The scientific data conditional filtering method according to claim 1, characterized in that, The input scientific dataset can be gridless, structured grid, or unstructured grid data.

10. The scientific data conditional filtering method according to claim 1, characterized in that, The data content in the input scientific dataset includes at least one of point sets, cell sets, vertex data, and cell data. Vertex data is attribute data associated with points, and cell data is attribute data associated with cells.

Images