Parallel processing methods, apparatus, computer equipment, and storage media for grid files

Abstract

This disclosure relates to a parallel processing method, apparatus, computer device, and storage medium for grid files. It includes controlling multiple target processes to read corresponding file data segments, and upon successful reading, controlling each target process to identify the header information contained in the read file data segment; for any file data segment, if the header information is incomplete based on the header information identification result, completing the header information; controlling communication between the multiple target processes, and constructing a global header information set based on the communication results and the header information identification results of each target process; and parsing the grid data in the file data segment of each target process based on the global header information set to obtain the grid data of each file data segment, and integrating the grid data of each file data segment to obtain the target grid file. This mechanism, which avoids repeated traversal, significantly shortens the data loading time.

Description

Technical Field

[0001] This disclosure relates to the field of grid processing technology, and in particular to a method, apparatus, computer device, and storage medium for parallel processing of grid files. Background Technology

[0002] With the increasing application of computational fluid dynamics (CFD) in the design and optimization of aero-engines, higher requirements have been placed on the size and complexity of computational grids.

[0003] In related technologies, the entire Fluent mesh file can be read first to identify and record the starting position and data length of all header information. Then, each process calculates its assigned file offset and data segment range based on its allocated data segment. Finally, the corresponding data is read into the memory of each process via parallel file I / O. However, to identify and locate the header information of the data segment, at least two complete I / O operations must be performed on the Fluent mesh file: the first is a global scan by the main process to extract the header information, and the second is when each process reads the actual data based on the header information. This repetitive traversal mechanism significantly reduces I / O efficiency. Summary of the Invention

[0004] In view of this, the present disclosure provides a method, apparatus, computer device, and storage medium for parallel processing of grid files to solve the problems existing in the related art.

[0005] A first aspect of this disclosure provides a parallel processing method for a grid file. The method includes: obtaining the file size of the grid file to be read, and determining multiple file data segments to be read by multiple target processes based on the file size; controlling the multiple target processes to read the corresponding file data segments, and after successful reading, controlling each target process to identify the header information contained in the read file data segment; for any file data segment, if it is determined based on the header information identification result that the header information in the file data segment is incomplete, completing the header information; controlling communication between the multiple target processes, and constructing a global header information set based on the communication results and the header information identification results of each target process; parsing the grid data in the file data segment of each target process based on the global header information set to obtain the grid data of each file data segment, and integrating the grid data of each file data segment to obtain a target grid file.

[0006] A second aspect of this disclosure provides a parallel processing apparatus for grid files, applied to the parallel processing selection method for grid files as described in the first aspect. The apparatus includes: an acquisition module, configured to acquire the file size of the grid file to be read, and determine multiple file data segments to be read by multiple target processes based on the file size; a control module, configured to control the multiple target processes to read the corresponding file data segments, and after successful reading, control each target process to identify the header information contained in the read file data segment; a completion module, configured to complete the header information for any file data segment when the header information in the file data segment is determined to be incomplete based on the header information identification result; a construction module, configured to control communication between the multiple target processes, and construct a global header information set based on the communication results and the header information identification result of each target process; and a parsing module, configured to parse the grid data in the file data segment of each target process based on the global header information set to obtain the grid data of each file data segment, and integrate the grid data of each file data segment to obtain a target grid file.

[0007] A third aspect of this disclosure provides a computer device including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-described parallel processing method for grid files.

[0008] A fourth aspect of this disclosure provides a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the steps of the parallel processing method for grid files described above.

[0009] According to a fifth aspect of this disclosure, a computer program product is provided that, when executed by a processor, implements the steps of the parallel processing method for the grid file described above.

[0010] The above-mentioned at least one technical solution adopted in the embodiments of this disclosure can achieve the following beneficial effects: by obtaining the file size of the grid file to be read, and determining the file data segments to be read by multiple target processes according to the file size; controlling multiple target processes to read the corresponding file data segments respectively, and after successful reading, controlling each target process to identify the header information contained in the read file data segment; for any file data segment, when it is determined from the header information identification result that the header information in the file data segment is incomplete, completing the header information; controlling multiple target processes to communicate with each other, and constructing a global header information set according to the communication result and the header information identification result of each target process; according to the global header information set, parsing the grid data in the file data segment of each target process to obtain the grid data of each file data segment, and integrating the grid data of each file data segment to obtain the target grid file.

[0011] As can be seen, in the initial stage, the grid file to be read is divided into multiple file data segments according to the file size and allocated to each target process. Each target process only needs to read the corresponding file data segment through a single read operation. Subsequently, each target process identifies the header information of its own file data segment and completes the incomplete header information. Then, a global header information set is constructed by combining the communication results between the target processes. Finally, based on the global header information set, the grid data in the file data segment of each target process is parsed to obtain the grid data of each file data segment. The grid data of each file data segment is then integrated to obtain the target grid file. In this way, multiple target processes can be controlled to read their respective file data segments in parallel at one time, thereby significantly reducing the disk access frequency and the overall data throughput pressure. Especially in large-scale grid file processing scenarios, this mechanism of avoiding repeated traversal not only greatly shortens the data loading time, but also effectively avoids the risk of file system rate limiting and network congestion caused by intensive operations. While ensuring data integrity and parsing accuracy, it achieves near-linear parallel scalability, greatly improving the preprocessing efficiency of engineering applications such as aero-engine combustion chamber simulation in supercomputing environments. Attached Figure Description

[0012] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0013] Figure 1 A flowchart illustrating a parallel processing method for grid files provided in an embodiment of this disclosure;

[0014] Figure 2 This is a schematic diagram illustrating a structure for reading data segments from a grid file, provided in one embodiment of the present disclosure.

[0015] Figure 3 This is a schematic diagram of the structure of a Fluent mesh file provided in an embodiment of the present disclosure;

[0016] Figure 4 This is a schematic diagram illustrating header information truncation according to an embodiment of the present disclosure;

[0017] Figure 5 A flowchart illustrating another parallel processing method for grid files provided in an embodiment of this disclosure;

[0018] Figure 6A schematic diagram of the structure of a parallel processing apparatus for grid files provided in an embodiment of this disclosure;

[0019] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure;

[0020] Figure 8 This is a schematic diagram of the structure of a computer system provided in an embodiment of the present disclosure;

[0021] Figure 9 A schematic diagram of a computer program product provided according to an embodiment of this disclosure. Detailed Implementation

[0022] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0023] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0024] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc., used in this disclosure are only used to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0025] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0026] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0027] With the increasing application of Computational Fluid Dynamics (CFD) in aero-engine design and optimization, higher demands are placed on the scale and complexity of computational grids. This is particularly evident in the representation of complex geometric configurations, such as the unstructured grids commonly used in Fluent. Unstructured grids can have hundreds of millions or even billions of nodes and elements. Such large grid sizes not only significantly increase the demand for computational resources but also make grid generation and preprocessing critical steps limiting the efficiency of the entire simulation process. Therefore, efficient and reliable grid processing methods have become one of the core challenges in improving the overall performance of CFD simulations. Here, CFD is a discipline that comprehensively utilizes computer science, numerical methods, and fluid mechanics theory to simulate fluid flow, heat transfer, and related physical phenomena through numerical computation; Fluent is a globally leading commercial computational fluid dynamics simulation software.

[0028] Typically, when using Fluent for simulation calculations, the preprocessing stage requires first reading the Fluent mesh file and loading various topological data, such as node coordinates, cell connection relationships, and boundary markers, into the memory of the current processing node, forming a structured data object that can be used for computation. The Fluent mesh file can use Fluent's proprietary text format, with different categories of data identified by specific header information and delimited by opening brackets "(" and closing brackets ")". However, typically, the various topological data are randomly distributed within the Fluent mesh file, lacking a fixed order, which significantly increases the complexity of reading and parsing.

[0029] To address the aforementioned issues, related technologies typically employ the following methods to read various types of topology data from Fluent mesh files:

[0030] Method 1, the serial reading method, involves scanning brackets to confirm the header information and the start and end of the data segment, and reading the data into memory byte by byte. However, this method cannot utilize parallel computing resources and has low reading efficiency.

[0031] Method 2, a multi-process parallel reading method, involves the main process sequentially reading the entire Fluent grid file, identifying and recording the starting position and data length of all header information. Subsequently, each process calculates the file data segment and its range to be read based on its allocated header metadata. Finally, parallel file I / O is used to read the corresponding data into the memory of each process. While this method achieves multi-process collaborative reading and possesses a certain degree of parallelism, it still suffers from two significant performance bottlenecks in ultra-large-scale grid processing due to the reliance on the main process for global scanning and the need for each process to independently initiate multiple I / O requests: firstly, the global pre-read operation introduces additional I / O overhead and synchronization latency; secondly, a large number of scattered I / O requests in the parallel file system easily cause access conflicts and system rate limiting, severely restricting overall reading efficiency and scalability.

[0032] It is evident that several fundamental performance bottlenecks exist in related technologies, severely limiting the efficiency of ultra-large-scale Fluent mesh processing. First, to identify and locate the header information of data segments, at least two complete I / O operations must be performed on the Fluent mesh file: the first is a global scan by the main process to extract the header information, and the second is that each process performs the actual data reading operation based on the header information. This repetitive traversal mechanism significantly reduces I / O efficiency. Second, because Fluent mesh files typically use a special text format delimited by brackets, and the data segments are distributed in an unordered manner, traditional parallel methods require byte-by-byte parsing to identify structural boundaries. In a supercomputing environment, this triggers a large number of fine-grained network and storage I / O requests. This not only causes excessive system load but also easily triggers the rate-limiting mechanism of the parallel file system, and may even cause computing nodes to fail due to I / O timeouts. Furthermore, although global scanning in related technologies avoids redundant readings, it increases the number of I / O operations, while segmented reading reduces the number of operations but may introduce redundant data. These limitations make it unsuitable for processing hundreds of millions to billions of meshes, severely restricting the efficiency of the entire simulation process and the system's scalability.

[0033] To address the aforementioned issues, this disclosure proposes a parallel processing method for grid files: The method allocates the reading range of the grid file to be read to all target processes using byte-level load balancing, and introduces a buffered data segment mechanism during each target process's reading to ensure header information integrity. Simultaneously, it utilizes MPI-IO to initiate large-block data reading operations, loading all file data segments into the memory of the corresponding target process at once, and storing identifiable data into the corresponding data structure by recognizing header information. Furthermore, it shares a global header information set through inter-process communication to complete missing or truncated header information for each target process. Finally, the processed grid data is written to a regular format file via a single large-block I / O operation, achieving efficient parallel parsing and distribution of the entire grid with only one file read and minimal I / O calls.

[0034] Figure 1 This is a flowchart illustrating a parallel processing method for mesh files according to an embodiment of this disclosure. Figure 1 As shown, it specifically includes:

[0035] S101: Obtain the file size of the grid file to be read, and calculate the file data segment to be read by each target process through equal distribution logic. Here, the grid file to be read can be a Fluent grid file, and the number of target processes is usually dynamically allocated by the main process or scheduling system during the program initialization phase.

[0036] In some embodiments, all target processes can be controlled to synchronously acquire the file size of the grid file to be read, i.e., the global byte size; then, combined with a byte-level fully equal-division algorithm, the byte range of the file data segment that each target process needs to read in the grid file to be read is determined.

[0037] Specifically, first, the file size of the grid file to be read can be determined, denoted as n bytes, and the number of target processes as m. The theoretical read volume of the file data segment required by each target process is n / m bytes. Here, when n is not divisible by m, the remainder r can be redistributed equally among the target processes to ensure load balancing. For example, when n=100 bytes and m=4, each target process reads 25 bytes, with no remainder requiring additional allocation; while when n=102 bytes and m=4, each target process reads 25 bytes, but r=2 bytes remain. In this case, the remaining 2 bytes can be redistributed equally among the target processes, meaning each target process ultimately receives 25.5 bytes.

[0038] In some embodiments, after determining the amount of data that each target process needs to read, the byte range of the target data segment corresponding to each target process in the grid file to be read can be accurately calculated based on the global number of each target process. The byte range can be represented by an interval (start, end), where the start position start and the end position end can both be located in bytes.

[0039] For example, given a 100-byte file and four target processes: target process 0 reads data in the range (0, 25), target process 1 reads data in the range (25, 50), target process 2 reads data in the range (50, 75), and target process 3 reads data in the range (75, 100). This achieves a continuous, non-overlapping partition of the entire grid file to be read. Here, the global ID of each target process refers to a unique, continuous integer identifier assigned by the runtime system to each participating target process in a parallel computing environment. This global ID is used to uniquely distinguish and locate different target processes globally, and is the foundation for inter-process communication, data partitioning, and collaborative work.

[0040] In the parallel reading process based on byte partitioning, the end position of the target data segment may happen to be in the middle of a certain line of data, which will result in the loss of header information in the target data segment. Figure 2 This is a schematic diagram illustrating a structure for reading data segments from a grid file, as provided in an embodiment of this disclosure. Figure 2 As shown, the data segment includes a first target data segment 201 to be read by target process 1, a buffer data segment 202 of target process 1, and a second target data segment 203 to be read by target process 2. The first target data segment 201 ends between the characters "10" and "1" in the data "(12 (6f 1 10 1 3 ))", causing the data to break between "10" and "1", resulting in two incomplete parts. One part is located in the first target data segment 201, and the other part is located in the second target data segment 203. If this truncation is not handled, the header information or topology of target process 1 and target process 2 will be unrecognizable, leading to subsequent data parsing errors.

[0041] In some embodiments, to eliminate the problem of header information truncation in file data segments, this disclosure embodiment can add a buffer data segment within the file data segment that each target process needs to read. In this case, the file data segment can include the original target data segment to be read and the additional buffer data segment. Here, the buffer data segment can be a segment of data of a specific length that continues to be read after the original target data segment that the target process needs to read. Thus, when the header information of the target process is incomplete, the missing header information can be directly obtained from the corresponding buffer data segment without frequent inter-process communication, thereby completing the header information of the current target process. It is evident that the design of the buffer data segment not only ensures the integrity of the header information in the grid file to be read but also reduces the communication overhead incurred between target processes to coordinate the integrity of file data segments, improving the overall reading efficiency of the grid file to be read. Here, the header information is a continuous byte sequence with a fixed or variable length in the grid file to be read.

[0042] Specifically, in this embodiment, a buffered data segment is introduced for each target process. That is, at the end of the target data segment of each target process, an additional buffered data segment of length 'a' is read, creating overlapping data segments of length 'a' between adjacent target processes. Here, the data length 'a' of the buffered data segment can be greater than the maximum number of bytes in a single line of data in the target file; typically, 'a' ≥ 100 can be set to meet the reading requirements of most grid files. It should be understood that for the target process with the last global number, since the file data segment to be read by the last target process has reached the end of the file, a buffered data segment is not required.

[0043] Based on this, after determining the range of bytes that each target process needs to read, a buffer data segment can be extended backward by 'a' bytes to construct a file data segment for each target process, consisting of the target data segment and the buffer data segment. For example, when a grid file of total size n is read by four target processes, target process 0 reads (0, n / 4 + a), target process 1 reads (n / 4, 2n / 4 + a), target process 2 reads (2n / 4, 3n / 4 + a), and target process 3 reads (3n / 4, n).

[0044] As can be seen, the embodiments of this disclosure can significantly reduce the communication overhead between target processes while ensuring the integrity of the header information of the grid file to be read through the byte-level equalization and buffering data segment mechanism, thereby achieving efficient and scalable parallel reading.

[0045] S102, control multiple target processes to read the corresponding file data segments, identify the header information in the multiple file data segments, and store the identified header information into a preset data structure.

[0046] In some embodiments, the grid file to be read can be opened using MPIIO. Through a single large block of continuous I / O read operations, each target process acquires and stores the file data segment it is responsible for processing. After a target process reads its corresponding file data segment, the successfully read segment is stored in its corresponding temporary storage area. Here, the temporary storage area is not the final data structure used for computation; its content is an unparsed raw byte sequence, which needs to be further parsed and stored in a data array with a defined logical structure. The temporary storage area can consist of two parts: a local area and a buffer. The local area stores the target data segments that the target processes need to process, and its size is the file size of the grid file to be read divided by the total number of processes. The buffer stores buffered data segments, and its size is 'a'. Based on this, the total number of bytes actually read by each target process can be determined to be n / m + a. Here, MPIIO refers to a set of programming interfaces and specifications that allow multiple target processes to perform parallel and collaborative read and write operations on the same or different files in a parallel computing environment (such as a high-performance computing cluster or a supercomputer).

[0047] After the target process completes data reading, all subsequent data processing is performed in memory space. Each target process first parses the raw bytes of the target data segment in the temporary storage area of ​​its corresponding memory space to identify and extract the header information within the target data segment. This header information may include: the total number of points, the total number of faces, the total number of volumes, and the starting position information of point data blocks, face data blocks, and volume data blocks. This information, along with the mesh dimension information, constitutes the header information. Here, each target process corresponds to an independent, private memory space, which includes a temporary storage area for data. A target process may or may not include header information.

[0048] Figure 3 This is a schematic diagram illustrating the structure of a Fluent mesh file according to an embodiment of this disclosure. Figure 3 The diagram illustrates the basic structure of a Fluent mesh file, specifically showcasing four key data types and their storage formats: string comments (301), dimensions (302), scale information (303), basic point data information (304), cell topology information (305), polygon topology information (306), and polygon topology identification information (307). Scale information (303) includes three header entries, each beginning with an opening bracket and ending with a closing bracket. The first number in each header entry identifies the corresponding data type, which can include point coordinate markers (10), polygon topology markers (12), and volume topology markers (13). The core function of these header entries is to define the data type and structure of subsequent data blocks.

[0049] In some embodiments, when each target process parses the target data segment in its temporary storage area, it may identify zero or at least one header information. In this case, all header information possessed by the target process can be collected and encapsulated into a data structure named Patterns. This data structure includes the target process number to which the header information belongs, the data type of the header information, the byte range of the header information in the target data file, whether the header information is truncated due to partitioning, the specific location where the header information truncation occurs, and the byte range read from the buffer to complete the header information. Here, one target process corresponds to one Pattern.

[0050] like Figure 2 As shown, target process 1 first identifies multiple complete header information from the first target data segment 201 and records them in Patterns. However, when parsing to the end of the first target data segment 201, target process 1 finds that one of the header information is truncated, resulting in incomplete header information in target process 1. At this time, corresponding data can be obtained from the buffer data segment 202 of target process 1 to complete the incomplete header information. For example, continuing to read "1 3 ))" makes the header information in the first target data segment 201 complete. At this time, Patterns will mark this header information as truncated and record the range of bytes read for completion in the buffer data segment to ensure that subsequent data processing can correctly identify and concatenate the data segment.

[0051] In some embodiments, special handling is required when the header information being processed is the starting position information of specific information. Here, the starting position information of specific information is a special type of header information whose function is to declare the data type that immediately follows it, and the specific format can be (a (bc d...)). In this case, if the truncation occurs within the starting position information of specific information, the data parsing process should not terminate immediately upon encountering a newline character.

[0052] Figure 4 This is a schematic diagram illustrating header information truncation according to an embodiment of this disclosure. Figure 4 As shown, different versions of the mesh file may insert newline characters between parentheses. If only the newline character is read, the complete structural information may not be obtained. Therefore, the correct approach is to continue reading the buffered data segment in the buffer until the opening parenthesis of the next header information is captured, and mark all content before the opening parenthesis as the completion part of the current header information. At the same time, record its position in the buffer to ensure the integrity of the header information structure and the accuracy of parsing.

[0053] Once the target process successfully parses and obtains the starting position information of at least one complete point, surface, or volume, it can determine the data type of subsequent data segments based on this starting position information. To this end, each target process creates three associated containers for point information, surface topology information, and volume topology information. Based on this, all clearly identifiable data segments in the temporary storage area can be parsed and stored into the corresponding containers according to the determined header information type.

[0054] The specific data can be read by taking a complete line in the mesh file to be read as the smallest unit. Point information corresponds to the two-dimensional or three-dimensional coordinates of a point; surface topology information includes the indexes of all points constituting the surface and the identifiers of its adjacent left and right units; volume topology information requires differentiation between two cases: Case 1, if the mesh type is explicitly specified in the header information, the topology data mapping can be directly generated based on its declaration; Case 2, if not specified, each line of data must be parsed sequentially within the local area. If a line of data is truncated at a partition boundary, it must enter the buffer and continue reading until a newline character is encountered at the end of the line to complete the data item, and the truncation position is recorded. It should be understood that the processing rules here differ from the header information completion strategy; the reading of data lines ends with a newline character as the explicit termination boundary.

[0055] S103 controls multiple target processes to communicate with each other, and when no header information is identified in the file data segment of a target process, the file data segment of the target process is identified based on the communication results.

[0056] In some embodiments, after all target processes have completed the collection of their corresponding Patterns data, they can be controlled to communicate with each other to construct a global header set AllPatterns based on each target process's Patterns. This ensures that all target processes can synchronously acquire all header information. Given that the number of headers contained in each target process varies, the communication can be divided into two steps: first, the lengths of the Patterns data structures of each target process are exchanged, and then the actual data body is transmitted synchronously.

[0057] After communication is complete, each target process can parse the remaining, undetermined data in the temporary storage area based on AllPatterns. Specifically, this involves traversing AllPatterns to find locations where the data type of the header information changes, thereby determining the data type within that range. If the current file data segment is truncated at a partition by the previous target process, reading must begin from the truncation point recorded in the previous target process's Patterns to ensure data continuity. If the data segment is complete, parsing begins directly from the beginning of the current target process's file data segment. During parsing, the program automatically skips format characters such as spaces and newlines, directly extracting valid numeric information.

[0058] For example, suppose target process 2 has a piece of residual data of an undetermined type in its temporary storage area, with its byte range corresponding to bytes 150-170 of the global file. After obtaining AllPatterns, target process 2 iterates through the AllPattern and finds that the last header information of target process 1 (ending at byte 149) is marked as face topology data, while the first header information of target process 3 (starting at byte 171) is marked as volume topology data. Therefore, it is determined that the residual data between bytes 150-170 should be information indicating the transition from face topology to volume topology. Simultaneously, since target process 1's Patterns record that its last data is truncated at byte 149, target process 2 starts parsing from byte 150 (i.e., after the truncation point), automatically skipping newline characters, continuously reading the numerical sequence, and correctly parsing it into the topological connection information of a face cell based on the context, ultimately filling it into the face topology data structure.

[0059] Finally, each target process fills the mesh data corresponding to the identified header information into pre-defined point topology containers, face topology containers, or volume topology containers, according to their categories. After data filling, each target process has completed all analysis work in its local temporary storage area, and the global mesh topology is formed by the aggregation of local data from all processes. Subsequently, each target process writes the organized file data segments in memory to the normalized target file in one go through batch aggregation I / O operations. This process ensures that the output data has good structure and read / write efficiency, facilitating direct and efficient access for subsequent computational tasks. Here, the header information marks the starting position and data type of each data segment in the mesh file.

[0060] S104: Based on the communication results and the global header information set, the grid data of each target process is integrated to obtain the target grid file.

[0061] In some embodiments, before integrating the grid data of each target process, the number of point data, area data, and volume data held by each target process can be aggregated from AllPatterns. Since the data types and quantities processed by each target process are unevenly distributed, the above information can be synchronized through inter-process communication to ensure that all target processes are aware of the global data distribution status, which forms the basis for subsequent parallel write operations.

[0062] Before parallel writing, the write offset of each target process in different data files can be calculated. The offset depends on the total amount of data to be written by all processes preceding that target process. The calculation is based on the inherent format of each data type. For example, point data contains 2 or 3 coordinate values ​​per record; volume topology contains only one type identifier per record; and polygon topology has 4 values ​​in 2D (2 point indices and left / right cell numbers) and a maximum of 6 values ​​in 3D (4 point indices and left / right cell numbers, padded with -1 if necessary). By standardizing the length of each record, the write position and total data volume of each target process can be accurately calculated, laying the foundation for subsequent parallel I / O. Based on this, according to the calculated offset, file data segments from multiple target processes can be simultaneously and in parallel written to target files via MPIIO, according to data type, to obtain a target mesh file containing point mesh files, polygon mesh files, and volume mesh files. Thus, when reading this mesh file again, the target mesh file can be read directly to generate point mesh files, polygon mesh files, and volume mesh files.

[0063] As can be seen, in the initial stage, the grid file to be read is divided into multiple file data segments according to the file size and allocated to each target process. Each target process only needs to read the corresponding file data segment through a single read operation. Subsequently, each target process identifies the header information of its own file data segment and completes the incomplete header information. Then, a global header information set is constructed by combining the communication results between the target processes. Finally, based on the global header information set, the grid data in the file data segment of each target process is parsed to obtain the grid data of each file data segment. The grid data of each file data segment is then integrated to obtain the target grid file. In this way, multiple target processes can be controlled to read their respective file data segments in parallel at one time, thereby significantly reducing the disk access frequency and the overall data throughput pressure. Especially in large-scale grid file processing scenarios, this mechanism of avoiding repeated traversal not only greatly shortens the data loading time, but also effectively avoids the risk of file system rate limiting and network congestion caused by intensive operations. While ensuring data integrity and parsing accuracy, it achieves near-linear parallel scalability, greatly improving the preprocessing efficiency of engineering applications such as aero-engine combustion chamber simulation in supercomputing environments.

[0064] The parallel processing method for grid files provided in this disclosure can be executed by a terminal or by a chip applied to the terminal.

[0065] For example, the aforementioned terminals may include one or more of the following: mobile phones, tablets, wearable devices, in-vehicle devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, handheld computers (PDAs), and wearable devices based on augmented reality (AR) and / or virtual reality (VR) technologies. They may also include, but are not limited to, remote control devices, wearable devices, streetlights, home appliances, and other smart terminals. This disclosure does not impose specific limitations on these aspects.

[0066] Figure 5 This is a flowchart illustrating another parallel processing method for mesh files provided in an embodiment of this disclosure. Figure 5 As shown, it specifically includes:

[0067] S501, obtain the file size of the grid file to be read, and determine the file data segments to be read by multiple target processes according to the file size.

[0068] In some embodiments, the main process first obtains the total size of the grid file to be read through file system calls. For example, for a 1.2GB grid file to be read (the Fluent grid file mentioned above), based on the preset number of 4 target processes, the file size can be divided by the number of processes to obtain a theoretical load of 300MB for each target process. To handle the remainder (200MB) that cannot be divided evenly, a dynamic allocation strategy is adopted: the first two target processes (target process 0 and target process 1) are each allocated an additional 50MB, that is, they read 350MB respectively, with starting offsets of 0 and 350MB; while the latter two target processes (target process 2 and target process 3) read 250MB respectively, with starting offsets of 700MB and 950MB respectively, ensuring that all data is completely covered and without overlap.

[0069] In this way, by distributing the remainders across preceding target processes, the performance bottleneck of processing too much residual data in a single target process is avoided, and the I / O operation times of each target process tend to be consistent. At the same time, clearly dividing the file into contiguous blocks eliminates read conflicts and significantly improves parallel read efficiency. Especially for large grid files, it can greatly shorten the overall data loading time, laying an efficient data foundation for subsequent parallel computing.

[0070] S502 controls multiple target processes to read corresponding file data segments, and after successful reading, controls each target process to identify the header information contained in the read file data segment.

[0071] In some embodiments, the master process can broadcast its respective file data segment partitioning information (the byte range of the file data segment) to the four target processes, and then concurrently control each target process to read the specified file data segment. For example, after target process 0 reads 0-350MB of data, it immediately identifies the fixed-format header information (such as grid dimensions, number of variables, and other metadata) at the beginning of the file data segment in the corresponding memory; the same operation is also performed synchronously for the other target processes to ensure that each target process can correctly understand the structure of the data segment it is responsible for.

[0072] In this way, each target process can independently verify the validity of the data without waiting for the entire grid file to be read, significantly reducing I / O blocking time. At the same time, localizing the processing header information avoids frequent queries to the main process, which not only improves processing efficiency but also lays the foundation for each target process to independently process the main grid data based on metadata, forming a highly efficient staged parallel pipeline.

[0073] S503: For any file data segment, when it is determined from the header information recognition result that the header information in the file data segment is incomplete, the header information is completed.

[0074] In some embodiments, assuming that when target process 2 reads a data segment starting at 700MB, it finds that the header information is truncated because it is exactly at the boundary of the data segment (e.g., it only contains grid dimensions but lacks variable definitions), target process 2 will immediately send a request to the adjacent target process 1 (responsible for the previous data segment) to obtain a copy of the complete header information contained at the end of target process 1's file data segment. Subsequently, target process 2 will concatenate the received supplementary information with its own incomplete header information to reconstruct the complete file header structure, thereby ensuring the correct parsing of subsequent grid data.

[0075] In this way, information completion is achieved through lightweight communication between neighboring target processes, avoiding the bottleneck caused by centralizing all header information processing in the main process, and maintaining the overall efficiency of parallel reading. This design enhances the robustness of parallel file reading, ensuring that even if the data segmentation is not ideal, each target process can obtain the complete metadata required for parsing, thus improving the system's fault tolerance and adaptability.

[0076] S504 controls communication between multiple target processes and constructs a global header information set based on the communication results and the header information identification results of each target process.

[0077] In some embodiments, under the coordination of the main process, each target process can send its identified header information to all other target processes through communication operations. For example, target process 0 reports that its file data segment contains grid points 1-100 on the X-axis, and target process 1 reports points 101-200. After collecting this information, the main process integrates it to generate a global header information set describing the entire grid domain (points 1-400 on the X-axis) and broadcasts it to all target processes to ensure that each target process obtains a unified global data view.

[0078] In this way, each target process can not only process local file data segments, but also have a clear understanding of the overall grid structure, laying the foundation for computational operations that require global information. At the same time, communication operations avoid the serial bottleneck of the main process and achieve efficient information synchronization. Although a small amount of communication overhead is introduced, it is exchanged for a significant improvement in the correctness and coordination of the entire parallel program, which is a key step in building efficient large-scale parallel applications.

[0079] S505 parses the grid data in the file data segment of each target process according to the global header information set to obtain the grid data of each file data segment, and integrates the grid data of each file data segment to obtain the target grid file.

[0080] In some embodiments, during the aircraft flow simulation, the header information defines a complete aerodynamic mesh (including wing surface mesh, far-field boundary mesh, and spatial flow volume mesh). At this point, target process 0 parses its corresponding file data segment to obtain the surface mesh points (points) of the wing leading edge region and the triangular mesh surfaces they form; target process 1 parses the wing trailing edge and part of the near-field space, including both surface mesh and spatial volume mesh elements; target process 2 parses the volume mesh of the far-field region. After each target process completes its parsing, the main process stitches together the wing leading edge surface mesh of target process 0 and the trailing edge surface mesh of target process 1 according to global coordinates to restore the complete wing surface mesh. Then, it assembles the spatial volume meshes of target process 1 and target process 2 according to topological relationships, ultimately integrating them into a target mesh file containing complete surface and spatial volume meshes for CFD calculations.

[0081] In this way, through systematic parsing and integration operations, grid data scattered in file data segments of different target processes can be efficiently and completely reconstructed into target grid files with a unified structure and standardized format. This not only significantly improves the utilization and sharing of data resources, but also greatly facilitates subsequent reading and application. Since the data has been integrated into a single file and follows a standard structure, subsequent software or analysis programs can directly and quickly read and parse it, avoiding the complexity of processing multi-source heterogeneous data, thereby improving the efficiency and reliability of the overall data processing workflow.

[0082] As can be seen, in the initial stage, the grid file to be read is divided into multiple file data segments according to the file size and allocated to each target process. Each target process only needs to read the corresponding file data segment through a single read operation. Subsequently, each target process identifies the header information of its own file data segment and completes the incomplete header information. Then, a global header information set is constructed by combining the communication results between the target processes. Finally, based on the global header information set, the grid data in the file data segment of each target process is parsed to obtain the grid data of each file data segment. The grid data of each file data segment is then integrated to obtain the target grid file. In this way, multiple target processes can be controlled to read their respective file data segments in parallel at one time, thereby significantly reducing the disk access frequency and the overall data throughput pressure. Especially in large-scale grid file processing scenarios, this mechanism of avoiding repeated traversal not only greatly shortens the data loading time, but also effectively avoids the risk of file system rate limiting and network congestion caused by intensive operations. While ensuring data integrity and parsing accuracy, it achieves near-linear parallel scalability, greatly improving the preprocessing efficiency of engineering applications such as aero-engine combustion chamber simulation in supercomputing environments.

[0083] In some embodiments, after controlling each target process to identify the header information contained in the read file data segment, the method further includes: classifying multiple file data segments of multiple target processes into identifiable file data segments and unidentified data segments according to the header information identification result; for identifiable file data segments, when it is determined according to the header information identification result that the header information in the identifiable file data segment is incomplete, completing the header information; for unidentified data segments, determining the header information of the unidentified data segment according to the header information identification result of the file data segment adjacent to the unidentified data segment.

[0084] Specifically, after header information recognition is completed, the file data segments of each target process can be classified according to the recognition results. For example, for target processes 0, 1, 2, and 3, target processes 0, 1, and 3 are marked as recognizable file data segments because key metadata such as grid dimensions were successfully identified; while the data segment of target process 2 is marked as unrecognized because its starting position falls exactly at the gap between two grid blocks and no valid header information was parsed. Regarding the incomplete variable type information found in target process 1, the missing descriptor can be requested from target process 0 and completed. For target process 2, based on the header information of its adjacent target processes 1 and 3, it can be inferred that the segment should be pure grid node coordinate data, and the correct header information can be assigned accordingly.

[0085] In some embodiments, the header information recognition results of each target process for its file data segment can vary: if a target process fails to recognize any header information, the header information corresponding to the file data segment of this target process can be directly inferred and determined based on the header information recognized by the previous target process; if a target process successfully recognizes header information in the middle of the file data segment, the data before the header information and located between the beginning and end of the file data segment may be structurally incomplete due to being at the partition boundary, making it impossible to independently determine its corresponding header information. In this case, it is still necessary to rely on the header information record at the end of the previous target process to determine the actual type of this incomplete data, thereby ensuring that when the header information appears in the middle of the data segment, the residual data before it can also be correctly parsed. Here, specific implementation methods can also refer to the relevant content of S102 and S103, which will not be elaborated here.

[0086] In this way, by distinguishing between "identifiable" and "unidentifiable" file data segments and employing "nearest neighbor completion" and "adjacent neighbor inference" strategies respectively, various complex file segmentation situations can be intelligently handled, especially when the partition boundary falls on the metadata area or the junction of different data types. This not only avoids the interruption of the entire process due to the failure to identify individual file data segments, but also reduces the dependence on perfect data segment partitioning, ensuring that data can be correctly reconstructed even under non-ideal file partitioning, thus improving the fault tolerance and adaptability of the entire method.

[0087] In some embodiments, based on the communication results, the header information in the identifiable file data segment after the header information is completed and the header information in the unidentified data segment with the identified header information can be integrated to construct a global header information set.

[0088] Specifically, assuming that target process 1 completes the header information for target process 0, and target process 2 identifies its own data segment header information from target processes 1 and 3, the main process initiates global communication to collect the updated header information from all target processes. This header information includes the complete mesh parameters of target processes 0 and 1, the newly inferred node coordinate data descriptions from target process 2, and the boundary condition identifiers from target process 3. Based on this, the main process can integrate the completed information from identifiable file data segments and the newly identified information from unidentified file data segments to ultimately construct a complete and consistent global header information set. This set accurately describes the organizational structure of the entire mesh file, all variable types, and the physical meaning of each data segment. Here, the global header information set is AllPatterns mentioned above. The specific construction process of AllPatterns can be found in the relevant description above and will not be elaborated upon here.

[0089] In this way, the results of the previous classification process are fully utilized, and the reliable information after completion is integrated with the reasonable inference, ensuring the accuracy and completeness of the global header information set. This design makes the construction process of the global set highly fault-tolerant. Even if the initial partitioning causes some data segments to be unable to be identified independently, a unified global understanding can still be formed in the end, laying a solid foundation for the correct parsing and integration of subsequent data and ensuring the final quality of the parallel processing flow.

[0090] In some embodiments, a file data segment includes a target data segment and a cached data segment, wherein, in the grid file to be read, the cached data segment is located at the end of the target data segment and partially overlaps with the next adjacent file data segment; for any file data segment, when it is determined from the header information identification result that the header information in the file data segment is incomplete, the header information is completed, including: when it is determined from the header information identification result that the header information in the file data segment is incomplete, the header information is completed based on the buffered data segment of the file data segment.

[0091] Specifically, the file data segment read by each target process is designed to consist of two parts: a core target data segment and a cached data segment located at the end, overlapping with the beginning of the next target process's segment. For example, target process 1 is responsible for reading data from 350MB to 700MB. 350-699MB is target process 1's target data segment, while 699-701MB (2MB) is designated as a cached data segment, its content completely overlapping with the 0-2MB beginning of target process 2's target data segment. In this case, when target process 1 finds its header information truncated at the end of its target data segment, it does not need to request it from target process 2. It can directly extract the header of the next data block from its local cached data segment (i.e., the 699-701MB region) to complete the missing header information. For related details, please refer to the description in S102 above; further elaboration is omitted here.

[0092] In some embodiments, the cached data segment is used not only to complete truncated header information but also to complete truncated data bodies. Specifically, when the target process parses the data body following a complete header (e.g., (66 2)), if the data body is truncated because it is located at the end of the file data segment (e.g., the header information indicates that there should be 66 face unit data, but the actual file data segment only contains 64), the target process will continue to read data from the cached data segment until it obtains the complete 66 data units, thereby ensuring the integrity of a single data body (e.g., a complete topological sequence of faces).

[0093] In this way, by sacrificing a small amount of redundant disk reads (each target process reads a small segment of overlapping data), the synchronous communication wait and delay caused by incomplete header information during the parsing phase are completely avoided. This space-for-time strategy is particularly suitable for high-performance computing environments, significantly reducing the coupling and coordination costs between target processes, enabling each target process to independently and efficiently complete header information parsing and completion, greatly improving the efficiency and smoothness of overall parallel reading.

[0094] In some embodiments, the header information includes an information start identifier and an information end identifier. For any file data segment, when it is determined that the header information in the file data segment is incomplete based on the header information identification result, the header information is completed. The method includes: when only the information start identifier or the information end identifier of the header information exists in the file data segment, it is determined that the header information in the file data segment is incomplete.

[0095] Specifically, suppose the header information of a grid file begins with a specific character (START) and ends with (END). Target process 1 finds (START) in its file data segment, but does not find the corresponding (END) in the data following (START). In this case, it can be clearly determined that the header information in the file data segment is incomplete. For the incomplete header information, it can be completed from the cached data segment corresponding to target process 1. See the above text for details. Figure 4 The relevant content will not be elaborated here.

[0096] This allows for accurate identification of the specific circumstances under which the header information is truncated, providing a precise basis for subsequent targeted completion strategies, avoiding misjudgments based on complex content parsing, and enhancing the robustness and automation of the entire recognition and completion process.

[0097] In some embodiments, the grid data in the file data segment of each target process is parsed according to a global header information set to obtain the grid data of each file data segment, and the grid data of each file data segment is integrated to obtain a target grid file, including: determining the data type and data range of the grid data included in each file data segment according to the global header information set; for each file data segment, parsing the grid data in the file data segment into structured data according to the data type and data range of the grid data included in the file data segment, wherein the structured data includes at least one of point coordinate data, surface topology data, and volume topology data; and integrating the structured data of multiple file data segments to obtain a target grid file.

[0098] Specifically, in aero-engine simulation, the global header information set defines the composition of the entire model: fan blade surface mesh (surface topology data), compressor channel volume mesh (volume topology data), and point coordinate data of the combustion chamber walls. At this point, it can be assumed that target process 0 parses the surface mesh of the 1st to 3rd stage fan blades based on the data range of its allocated file data segment; target process 1 parses the volume mesh elements of the 4th to 6th stage compressors based on the data range of its allocated file data segment; and target process 2 parses the point cloud data of the combustion chamber region based on the data range of its allocated file data segment. After each target process completes its local structured data conversion, the main process integrates the blade surface mesh, compressor volume mesh, and combustion chamber point cloud data according to the engine aerodynamic path and their assembly relationships, ultimately generating a complete engine flow channel target mesh file that can be used for CFD calculations.

[0099] In this way, by performing parallel analysis of heterogeneous mesh components (points, surfaces, volumes) according to global information, the loading efficiency of multi-component, multi-precision hybrid meshes is greatly improved. Based on the integration of data semantics and spatial range, the geometric continuity and topological consistency of the mesh data of each component at the interface are ensured, providing an accurate mesh model for high-fidelity engine flow field simulation and significantly shortening the data preparation time for large-scale parallel computing.

[0100] In some embodiments, obtaining the file size of the grid file to be read and determining the file data segments to be read by multiple target processes based on the file size includes: dividing the file size equally according to the number of multiple target processes to obtain the theoretical number of bytes to be read by each target process; and determining the file data segments to be read by each target process in the grid file to be read based on the preset order of each target process and the theoretical number of bytes to be read.

[0101] In some embodiments, the overall mesh file of the aircraft engine to be read is 8GB in size, and there are 4 target processes. The system first divides the file size evenly, resulting in a theoretical read count of 2GB for each target process. Then, the file data segments are allocated strictly according to the preset order of the target process IDs (0, 1, 2, 3): target process 0 reads the data segment from offset 0 to 2GB, target process 1 reads the data segment from 2GB to 4GB, target process 2 reads the data segment from 4GB to 6GB, and target process 3 reads the data segment from 6GB to 8GB, ensuring that the entire file is continuously and non-overlapping.

[0102] This achieves basic load balancing, ensuring that the I / O data volume of each target process is basically consistent, thus avoiding idle waiting caused by uneven data distribution. Furthermore, this method is logically simple, has minimal overhead, and is suitable for files with uniformly distributed grid data.

[0103] Table 1. Time consumed when processing grid files with different numbers of cores.

[0104]

[0105] As shown in Table 1, the parallel processing method for grid files provided in this disclosure has significant advantages over related technologies. For example, for a grid of 15.8 million cells, as the number of cores increases from 8 to 48, the processing time of the method provided in this disclosure can be reduced from 11.71 seconds to 2.509 seconds, demonstrating good parallel scalability. In contrast, the processing time of related technologies consistently remains above 21 seconds, failing to effectively utilize multi-core resources. More importantly, when processing an ultra-large-scale grid of 840 million cells, the methods of related technologies cannot complete the reading in a supercomputing environment due to I / O and memory issues, while the method provided in this disclosure, using 1000 cores, successfully processes the data in just 50.13 seconds, demonstrating superior large-scale processing capabilities and reliability.

[0106] As shown in Table 1, the method provided in this disclosure shows a near-linear decrease in processing time as the number of cores increases when processing 15.8 million cells, and demonstrates excellent performance when processing ultra-large-scale meshes of 840 million cells. It is evident that the method provided in this disclosure effectively solves the readability and performance bottleneck problems caused by I / O and memory issues in related technologies.

[0107] As can be seen, in the initial stage, the grid file to be read is divided into multiple file data segments according to the file size and allocated to each target process. Each target process only needs to read the corresponding file data segment through a single read operation. Subsequently, each target process identifies the header information of its own file data segment and completes the incomplete header information. Then, a global header information set is constructed by combining the communication results between the target processes. Finally, based on the global header information set, the grid data in the file data segment of each target process is parsed to obtain the grid data of each file data segment. The grid data of each file data segment is then integrated to obtain the target grid file. In this way, multiple target processes can be controlled to read their respective file data segments in parallel at one time, thereby significantly reducing the disk access frequency and the overall data throughput pressure. Especially in large-scale grid file processing scenarios, this mechanism of avoiding repeated traversal not only greatly shortens the data loading time, but also effectively avoids the risk of file system rate limiting and network congestion caused by intensive operations. While ensuring data integrity and parsing accuracy, it achieves near-linear parallel scalability, greatly improving the preprocessing efficiency of engineering applications such as aero-engine combustion chamber simulation in supercomputing environments.

[0108] The foregoing primarily describes the solutions provided by the embodiments of this disclosure from the perspective of the server. It is understood that, in order to implement the above functions, the server includes the corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, this disclosure can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0109] This disclosure embodiment can divide the server into functional units according to the above method example. For example, it can divide each function into separate functional modules, or it can integrate two or more functions into one management module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this disclosure embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0110] In the case of dividing each functional module according to its corresponding function, an exemplary embodiment of this disclosure provides a parallel processing apparatus for grid files, which can be a server or a chip applied to a server. Figure 6 This is a schematic diagram of the structure of a parallel processing apparatus for grid files provided in one embodiment of the present disclosure. Figure 6 As shown, the parallel processing device 600 for the grid file includes:

[0111] The acquisition module 601 is used to acquire the file size of the grid file to be read, and determine the file data segments to be read by multiple target processes according to the file size;

[0112] The control module 602 is used to control multiple target processes to read the corresponding file data segments respectively, and after successful reading, control each target process to identify the header information contained in the read file data segment;

[0113] The completion module 603 is used to complete the header information of any file data segment when it is determined from the header information recognition result that the header information in the file data segment is incomplete.

[0114] The construction module 604 is used to control communication between multiple target processes and to construct a global header information set based on the communication results and the header information identification results of each target process.

[0115] The parsing module 605 is used to parse the grid data in the file data segment of each target process according to the global header information set, so as to obtain the grid data of each file data segment, and to integrate the grid data of each file data segment to obtain the target grid file.

[0116] In an alternative approach, after controlling each target process to identify the header information contained in the read file data segment, the method further includes:

[0117] Based on the header information recognition results, multiple file data segments of multiple target processes are classified into identifiable file data segments and unidentified data segments; for the identifiable file data segments, if it is determined from the header information recognition results that the header information in the identifiable file data segment is incomplete, the header information is completed; for the unidentified data segments, the header information of the unidentified data segments is determined based on the header information recognition results of the file data segments adjacent to the unidentified data segments.

[0118] In one alternative approach, based on the communication results, the header information in the identifiable file data segment after the header information is completed is integrated with the header information in the unidentified data segment whose header information has been identified, in order to construct a global header information set.

[0119] In one optional embodiment, the file data segment includes a target data segment and a cached data segment, wherein, in the grid file to be read, the cached data segment is located at the end of the target data segment and partially overlaps with the next adjacent file data segment; the step of completing the header information for any file data segment when it is determined from the header information identification result that the header information in the file data segment is incomplete includes: when it is determined from the header information identification result that the header information in the file data segment is incomplete, completing the header information based on the cached data segment of the file data segment.

[0120] In one optional approach, the header information includes an information start identifier and an information end identifier. For any file data segment, when it is determined that the header information in the file data segment is incomplete based on the header information identification result, the header information is completed. The method includes: determining that the header information in the file data segment is incomplete when only the information start identifier or the information end identifier of the header information exists in the file data segment.

[0121] In one optional approach, the step of parsing the grid data in the file data segment of each target process according to the global header information set to obtain the grid data of each file data segment, and integrating the grid data of each file data segment to obtain a target grid file, includes: determining the data type and data range of the grid data included in each file data segment according to the global header information set; for each file data segment, parsing the grid data in the file data segment into structured data according to the data type and data range of the grid data included in the file data segment, wherein the structured data includes at least one of point coordinate data, surface topology data, and volume topology data; and integrating the structured data of multiple file data segments to obtain the target grid file.

[0122] In one optional approach, obtaining the file size of the grid file to be read and determining the file data segments to be read by multiple target processes based on the file size includes: dividing the file size equally according to the number of multiple target processes to obtain the theoretical number of bytes to be read by each target process; and determining the file data segments to be read by each target process in the grid file to be read based on the preset order of each target process and the theoretical number of bytes to be read.

[0123] This disclosure also provides an electronic device, including: at least one processor; a memory for storing at least one processor-executable instruction; wherein the at least one processor is used to execute the instruction to implement the steps of the method disclosed in this disclosure.

[0124] Figure 7 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present disclosure. Figure 7 As shown, the electronic device 700 includes at least one processor 701 and a memory 702 coupled to the processor 701, which can perform the corresponding steps in the methods disclosed in the embodiments of this disclosure.

[0125] The processor 701 described above can also be called a Central Processing Unit (CPU), which can be an integrated circuit chip with signal processing capabilities. Each step in the method disclosed in this embodiment can be implemented by the integrated logic circuitry in the processor 701's hardware or by software instructions. The processor 701 can be a general-purpose processor, a digital signal processor (DSP), an ASIC, a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this embodiment can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can be located in the memory 702, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The processor 701 reads information from the memory 702 and, in conjunction with its hardware, completes the steps of the method described above.

[0126] Furthermore, various operations / processes according to this disclosure, implemented via software and / or firmware, can be transmitted from a storage medium or network to a computer system with a dedicated hardware architecture, for example, Figure 8 The computer system 800 shown is equipped with the programs that constitute the software. When various programs are installed, the computer system is able to perform various functions, including those mentioned above. Figure 8 This is a schematic diagram of the structure of a computer system provided in an embodiment of the present disclosure.

[0127] Computer system 800 is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0128] like Figure 8As shown, the computer system 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. The RAM 803 may also store various programs and data required for the operation of the computer system 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0129] Multiple components in the computer system 800 are connected to the I / O interface 805, including: an input unit 806, an output unit 807, a storage unit 808, and a communication unit 809. The input unit 806 can be any type of device capable of inputting information into the computer system 800. The input unit 806 can receive input numerical or character information and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 807 can be any type of device capable of presenting information and may include, but is not limited to, a monitor, speaker, video / audio output terminal, vibrator, and / or printer. The storage unit 808 may include, but is not limited to, a hard disk and an optical disk. The communication unit 809 allows the computer system 800 to exchange information / data with other devices via a network, such as the Internet, and may include, but is not limited to, a modem, network card, infrared communication device, wireless communication transceiver, and / or chipset, such as Bluetooth™ devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0130] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above. For example, in some embodiments, the methods disclosed in this disclosure can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on an electronic device via ROM 802 and / or communication unit 809. In some embodiments, the computing unit 801 can be configured to perform the methods disclosed in this disclosure by any other suitable means (e.g., by means of firmware).

[0131] This disclosure also provides a computer-readable storage medium, wherein when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is able to perform the methods disclosed in this disclosure.

[0132] The computer-readable storage medium in this disclosure can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. The aforementioned computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specifically, the aforementioned computer-readable storage medium may include electrical connections based on one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0133] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0134] Figure 9 This is a schematic diagram of a computer program product provided according to an embodiment of the present disclosure. Figure 9 As shown, the computer program product 900 includes a computer program 901, which, when executed by a processor, implements the methods disclosed in the embodiments of this disclosure.

[0135] In embodiments of this disclosure, computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include, but are not limited to, object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)), or it can be connected to an external computer.

[0136] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0137] The modules, components, or units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the modules, components, or units do not necessarily constitute a limitation on the module, component, or unit itself.

[0138] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary hardware logic components that can be used include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0139] The above description is merely an illustration of some embodiments of this disclosure and the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0140] While specific embodiments of this disclosure have been described in detail by way of example, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of this disclosure. Those skilled in the art should understand that modifications can be made to the above embodiments without departing from the scope and spirit of this disclosure. The scope of this disclosure is defined by the appended claims.

Claims

1. A method for parallel processing of a grid file, characterized by, The method comprises: acquiring a file size of a grid file to be read, and determining file data segments to be read by a plurality of target processes according to the file size respectively; controlling the plurality of target processes to read the corresponding file data segments respectively, and after reading successfully, controlling each target process to identify header information contained in the read file data segment; for any file data segment, when it is determined according to the header information identification result that the header information in the file data segment is incomplete, the header information is completed; controlling communication between the plurality of target processes, and constructing a global header information set according to the communication result and the header information identification result of each target process; according to the global header information set, parsing the grid data in the file data segment of each target process to obtain the grid data of each file data segment, and integrating the grid data of each file data segment to obtain a target grid file.

2. The method of claim 1, wherein, After the control of each target process to identify the header information contained in the read file data segment, the method further comprises: according to the header information identification result, classifying the plurality of file data segments of the plurality of target processes into identifiable file data segments and unidentified data segments; for the identifiable file data segments, when it is determined according to the header information identification result that the header information in the identifiable file data segment is incomplete, the header information is completed; for the unidentified data segments, determining the header information of the unidentified data segment according to the header information identification result of the file data segment adjacent to the unidentified data segment.

3. The method of claim 2, wherein, The method further comprises: according to the communication result, integrating the header information in the identifiable file data segment after the header information is completed and the unidentified data segment whose header information has been identified, to construct a global header information set.

4. The method of claim 1, wherein, The file data segment comprises a target data segment and a cache data segment, wherein, in the grid file to be read, the cache data segment is located at the end of the target data segment, and overlaps with part of the next adjacent file data segment; The method for any file data segment, when it is determined according to the header information identification result that the header information in the file data segment is incomplete, the header information is completed, comprises: when it is determined according to the header information identification result that the header information in the file data segment is incomplete, the header information is completed according to the buffer data segment of the file data segment.

5. The method of claim 1, wherein, The header information comprises information start identifier and information end identifier, and the method for any file data segment, when it is determined according to the header information identification result that the header information in the file data segment is incomplete, the header information is completed, comprises: when only the information start identifier or the end identifier of the header information exists in the file data segment, it is determined that the header information in the file data segment is incomplete.

6. The method of claim 1, wherein, The parsing, according to the global header information set, of mesh data in the file data segment of each target process to obtain mesh data of each file data segment, and the integration of the mesh data of each file data segment to obtain a target mesh file, comprises: According to the global header information set, determine the data type and data range of the mesh data included in each file data segment; For each file data segment, according to the data type and data range of the mesh data included in the file data segment, parse the mesh data in the file data segment into structured data, wherein the structured data includes at least one of point coordinate data, face topology data and volume topology data; Integrate the structured data of multiple file data segments to obtain the target mesh file.

7. The method of claim 1, wherein, The file size of the mesh file to be read is obtained, and according to the file size, the file data segments to be read by multiple target processes are determined respectively, comprising: Divide the file size by the number of multiple target processes to obtain the theoretical reading byte number of each target process; According to the preset order of each target process and the theoretical reading byte number, determine the file data segment to be read in the mesh file to be read by each target process.

8. A parallel processing device for a mesh file, characterized by Comprise: An acquisition module is configured to acquire a file size of a mesh file to be read, and determine file data segments to be read by multiple target processes according to the file size respectively; A control module is configured to control multiple target processes to read corresponding file data segments respectively, and control each target process to identify header information included in the read file data segment after successful reading; A completion module is configured to, for any file data segment, when it is determined according to the header information identification result that the header information in the file data segment is incomplete, complete the header information; A construction module is configured to control communication between multiple target processes, and construct a global header information set according to the communication result and the header information identification result of each target process; An analysis module is configured to, according to the global header information set, parse mesh data in the file data segment of each target process to obtain mesh data of each file data segment, and integrate the mesh data of each file data segment to obtain a target mesh file.

9. A computer device comprising a memory, a processor, and a computer program stored on the memory, wherein the computer program comprises instructions that, when executed by the processor, cause the processor to perform the method of any one of claims 1-8. The processor executes the computer program to realize the steps of the method of any one of claims 1-6.

10. A computer readable storage medium having stored thereon computer programs / instructions, characterized in that, The computer program / instructions are executed by the processor to realize the steps of the method of any one of claims 1-6.

Images