Performance evaluation method and device for geoscience parallel file system, equipment and medium

Abstract

The invention provides a performance evaluation method, device and equipment for a geoscience parallel file system and a medium, and relates to the technical field of computers.The method comprises the steps that a plurality of input and output operations and a plurality of synchronous operations corresponding to the geoscience parallel file system are obtained, and an operation tracking log is determined based on all the input and output operations and all the synchronous operations; generating analog input and output operations corresponding to the input and output operations and analog synchronous operations corresponding to the synchronous operations based on the operation tracking log; and determining a performance evaluation result of the geoscience parallel file system based on all the analog input and output operations and all the analog synchronous operations. According to the technical scheme, performance evaluation is carried out on the geoscience parallel file system on the basis of the simulation operation, operation of the geoscience parallel file system is not involved, and therefore the performance evaluation result of the geoscience parallel file system can be obtained under the condition that application execution on the geoscience parallel file system is not interfered.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a performance evaluation method, device, equipment and medium for a geospatial parallel file system. Background Art

[0002] Traditional file systems have significant limitations in parallel processing and data access, so data-intensive applications often use efficient geospatial parallel file systems for fast data access and processing.

[0003] Because the Geoscience Parallel File System involves large-scale hardware and software design, such as input / output (I / O) request scheduling, metadata management, data striping, caching mechanism, and network data transmission and backup, it is a huge challenge to fully understand and optimize the performance of the Geoscience Parallel File System. In the existing large-scale production environment, there are many inconveniences in testing the architecture and configuration of different Geoscience Parallel File Systems, which will cause performance interference to other applications on the Geoscience Parallel File System and even prevent other applications from continuing to execute.

[0004] Therefore, a performance evaluation method for geospatial parallel file systems is urgently needed to solve the above problems. Summary of the invention

[0005] The present invention provides a performance evaluation method, device, equipment and medium for a geoscientific parallel file system, which are used to solve the defect in the prior art that testing the architecture and configuration of different geoscientific parallel file systems will cause performance interference to other applications on the geoscientific parallel file system and even hinder the continued execution of other applications. The technical solution of the present invention performs performance evaluation on the geoscientific parallel file system based on simulated operations, and does not involve the operation of the geoscientific parallel file system itself. Therefore, the performance evaluation result of the geoscientific parallel file system can be obtained without interfering with the execution of applications on the geoscientific parallel file system, so that the architecture and configuration of the geoscientific parallel file system can also be tested and studied according to the performance evaluation result.

[0006] The present invention provides a performance evaluation method for a geospatial parallel file system, comprising the following steps.

[0007] Acquire multiple input / output operations and multiple synchronization operations corresponding to the geospatial parallel file system, and determine an operation tracking log based on all the input / output operations and all the synchronization operations; Generate, based on the operation tracking log, simulated input and output operations corresponding to the input and output operations and simulated synchronization operations corresponding to the synchronization operations; A performance evaluation result of the geoscientific parallel file system is determined based on all the simulated input and output operations and all the simulated synchronization operations.

[0008] According to a performance evaluation method of a geoscientific parallel file system provided by the present invention, the performance evaluation result of the geoscientific parallel file system is determined based on all the simulated input and output operations and all the simulated synchronization operations, including: For each of the simulated input and output operations, determining a previous simulated input and output operation of the simulated input and output operation, wherein the previous simulated input and output operation has the same simulation thread number as the simulated input and output operation; determining a first simulation timestamp of the simulated input and output operation based on an operation time interval corresponding to the simulated input and output operation, and a first simulation timestamp and simulation parameters of the previous simulated input and output operation; the operation time interval is determined based on a difference between a timestamp of an input and output operation corresponding to the simulated input and output operation and a timestamp of an input and output operation previous to the input and output operation; Determine respective second simulation timestamps of all the simulated synchronization operations based on the execution status of all the simulated synchronization operations and all the simulated input and output operations and the simulation thread numbers; The performance evaluation result is determined based on all of the first simulation timestamps and all of the second simulation timestamps.

[0009] According to a performance evaluation method of a geospatial parallel file system provided by the present invention, the first simulation timestamp of the simulated input / output operation is determined based on the operation time interval corresponding to the simulated input / output operation, and the first simulation timestamp and simulation parameters of the previous simulated input / output operation, including: Determining the operation execution time of the previous simulated input and output operation based on the simulation parameters of the previous simulated input and output operation; The operation execution duration and the first simulation timestamp of the previous simulation input / output operation and the operation time interval corresponding to the simulation input / output operation are summed, and the sum result is determined as the first simulation timestamp of the simulation input / output operation.

[0010] According to a performance evaluation method of a geospatial parallel file system provided by the present invention, the simulation parameters include a simulation operation type, a simulation offset, and a simulation access data volume; The determining, based on the simulation parameter of the previous simulation input / output operation, the operation execution duration of the previous simulation input / output operation comprises: In the case where the hardware configuration corresponding to the geospatial parallel file system is a mechanical hard disk, determining the head addressing duration based on the rotation speed corresponding to the mechanical hard disk and the simulated offset of the previous simulated input and output operation; Determining a unit byte read bandwidth corresponding to the mechanical hard disk based on the simulation operation type of the previous simulation input and output operation; Determine the medium reading duration based on the unit byte reading bandwidth and the simulated access data volume of the previous simulated input and output operation; The sum of the head addressing time and the medium reading time is determined as the operation execution time of the previous analog input and output operation.

[0011] According to a performance evaluation method of a geoscience parallel file system provided by the present invention, the simulation parameters include simulation operation type and simulation access data volume; The determining, based on the simulation parameter of the previous simulation input / output operation, the operation execution duration of the previous simulation input / output operation comprises: In a case where the hardware configuration corresponding to the geospatial parallel file system is a solid state drive, determining a unit byte read bandwidth corresponding to the solid state drive based on a simulation operation type of the previous simulation input and output operation; Determining the medium reading duration based on the unit byte reading bandwidth corresponding to the solid state drive and the simulated access data volume of the previous simulated input and output operation; The medium reading duration is determined as the operation execution duration of the previous simulated input and output operation.

[0012] According to a performance evaluation method of a geospatial parallel file system provided by the present invention, the operation tracking log is determined based on all the input and output operations and all the synchronization operations, including: The thread numbers, timestamps, operation types, accessed data volumes and offsets of all the input and output operations, and the thread numbers and timestamps corresponding to all the synchronization operations are recorded to determine the operation tracking log.

[0013] According to a performance evaluation method of a geospatial parallel file system provided by the present invention, generating simulated input and output operations corresponding to each of the input and output operations and simulated synchronization operations corresponding to each of the synchronization operations based on the operation tracking log includes: For each of the input and output operations, the thread number, operation type, access data volume and offset of the input and output operation are respectively determined as the simulated thread number, simulated operation type, simulated access data volume and simulated offset of the corresponding simulated input and output operation, to obtain the simulated input and output operation corresponding to the input and output operation; For each of the synchronization operations, the thread number corresponding to the synchronization operation is determined as the simulation thread number corresponding to the simulation synchronization operation, and the simulation synchronization operation corresponding to the synchronization operation is obtained.

[0014] The present invention also provides a performance evaluation device for a geospatial parallel file system, comprising the following modules: An acquisition module, used for acquiring a plurality of input and output operations and a plurality of synchronization operations corresponding to the geospatial parallel file system, and determining an operation tracking log based on all the input and output operations and all the synchronization operations; A simulation module, used for generating, based on the operation tracking log, simulated input and output operations corresponding to each of the input and output operations and simulated synchronization operations corresponding to each of the synchronization operations; An evaluation module is used to determine a performance evaluation result of the geoscientific parallel file system based on all the simulated input and output operations and all the simulated synchronization operations.

[0015] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the performance evaluation method of the geospatial parallel file system as described in any one of the above is implemented.

[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the performance evaluation method of the geospatial parallel file system as described in any one of the above.

[0017] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-mentioned methods for evaluating the performance of a geospatial parallel file system.

[0018] The performance evaluation method, device, equipment and medium of the geoscientific parallel file system provided by the present invention first obtain multiple input and output operations and multiple synchronization operations corresponding to the geoscientific parallel file system, and determine the operation tracking log based on all the input and output operations and all the synchronization operations; generate simulated input and output operations corresponding to each input and output operation and simulated synchronization operations corresponding to each synchronization operation based on the operation tracking log; and determine the performance evaluation result of the geoscientific parallel file system based on all the simulated input and output operations and all the simulated synchronization operations. The technical solution of the present invention determines the operation tracking log based on the acquired input and output operations and synchronization operations, and then determines the simulated input and output operations and simulated synchronization operations according to the operation tracking log, and performs performance evaluation on the geoscientific parallel file system based on the simulated operation, which does not involve the operation of the geoscientific parallel file system itself, so that the performance evaluation result of the geoscientific parallel file system can be obtained without interfering with the execution of applications on the geoscientific parallel file system, so that the architecture and configuration of the geoscientific parallel file system can also be tested and studied according to the performance evaluation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 It is a flow chart of the performance evaluation method of the geoscience parallel file system provided by the present invention.

[0021] Figure 2 It is a structural schematic diagram of the performance evaluation device of the geoscience parallel file system provided by the present invention.

[0022] Figure 3 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] In view of the above problems in the prior art, the present invention provides a performance evaluation method for a geospatial parallel file system. Figure 1 FIG. 1 is a flow chart of a method for evaluating the performance of a geoscience parallel file system provided by the present invention. Figure 1 As shown, the method includes the following: Step 110: Acquire multiple input / output operations and multiple synchronization operations corresponding to the geospatial parallel file system, and determine an operation tracking log based on all the input / output operations and all the synchronization operations.

[0025] Specifically, the present invention can pre-set a tracking tool, for example, a traditional recorder tracking tool can be customized and modified to adapt to a variety of input and output operations (I / O operations). The tracking tool can be used to record the function calls of the application corresponding to the geoscientific parallel file system, that is, the tracking tool can be used as a preload library to intercept I / O operations in real time and redirect them to the tracking tool, so that multiple input and output operations and multiple synchronization operations of different threads corresponding to the geoscientific parallel file system can be obtained. After obtaining multiple input and output operations and multiple synchronization operations corresponding to the geoscientific parallel file system, the operation tracking log can be determined based on all input and output operations and all synchronization operations.

[0026] In one embodiment, determining the operation tracking log based on all the input and output operations and all the synchronization operations includes: The thread numbers, timestamps, operation types, accessed data volumes and offsets of all the input and output operations, and the thread numbers and timestamps corresponding to all the synchronization operations are recorded to determine the operation tracking log.

[0027] Specifically, for input and output operations, the thread number, timestamp, operation type, access data volume, and offset of all input and output operations can be recorded. In addition, since synchronization operations write the data in the buffer back to the disk, and synchronization operations also represent the dependency between I / O operations between different threads, if the I / O scheduler lacks fairness between processes, the synchronization barrier in the application will reduce performance. Therefore, for synchronization operations, the thread number and timestamp corresponding to all synchronization operations can be recorded.

[0028] Further, an operation tracking log may be determined, that is, the operation tracking log records the above simulation parameters of the input and output operations and the above simulation parameters of the synchronization operations. The operation tracking log may include: Hierarchical Data Format version 5 (HDF5), Message Passing Interface Input / Output (MPI-IO) and Portable Operating System Interface (POSIX).

[0029] In the above embodiment, an operation tracking log is determined by recording the thread number, timestamp, operation type, accessed data volume and offset of the input and output operations, as well as the thread number and timestamp corresponding to the synchronization operation. The operation tracking log lays the foundation for generating simulated input and output operations and generating simulated synchronization operations.

[0030] Step 120: Generate, based on the operation tracking log, simulated input / output operations corresponding to the input / output operations and simulated synchronization operations corresponding to the synchronization operations.

[0031] Specifically, the operation tracking log can be read by the I / O load generator, and then the simulated input and output operations corresponding to each input and output operation and the simulated synchronization operations corresponding to each synchronization operation can be generated.

[0032] In one embodiment, generating the simulated input / output operations corresponding to the input / output operations and the simulated synchronization operations corresponding to the synchronization operations based on the operation tracking log includes: For each of the input and output operations, the thread number, operation type, access data volume and offset of the input and output operation are respectively determined as the simulated thread number, simulated operation type, simulated access data volume and simulated offset of the corresponding simulated input and output operation, to obtain the simulated input and output operation corresponding to the input and output operation; For each of the synchronization operations, the thread number corresponding to the synchronization operation is determined as the simulation thread number corresponding to the simulation synchronization operation, and the simulation synchronization operation corresponding to the synchronization operation is obtained.

[0033] Specifically, for each input and output operation, the thread number, operation type, access data volume and offset of the input and output operation can be respectively determined as the simulated thread number, simulated operation type, simulated access data volume and simulated offset of the corresponding simulated input and output operation, and the simulated input and output operation corresponding to the input and output operation is obtained. In addition, for each synchronization operation, the thread number corresponding to the synchronization operation can also be determined as the simulated thread number of the corresponding simulated synchronization operation, and the simulated synchronization operation corresponding to the synchronization operation is obtained. It is easy to understand that the input and output operation corresponds to the simulated input and output operation one-to-one, and the thread number, operation type, access data volume and offset of the two are the same, but the timestamp is different. The timestamp of the simulated input and output operation is used to evaluate the performance of the geoscientific parallel file system, so it needs to be determined by calculation. Similarly, the synchronization operation corresponds to the simulated synchronization operation one-to-one, and the thread number of the two is the same, but the timestamp is different. The timestamp of the simulated synchronization operation is also used to evaluate the performance of the geoscientific parallel file system.

[0034] In the above embodiment, simulated input and output operations corresponding to each input and output operation and simulated synchronization operations corresponding to each synchronization operation are generated based on the operation tracking log. Through the simulated operations, the performance of the geoscientific parallel file system can be evaluated without affecting the execution of other applications by the geoscientific parallel file system, which is beneficial to the research on the geoscientific parallel file system.

[0035] Step 130: Determine a performance evaluation result of the geoscientific parallel file system based on all the simulated input and output operations and all the simulated synchronization operations.

[0036] Specifically, the performance evaluation result of the geospatial parallel file system may be determined based on all simulated input and output operations and all simulated synchronization operations.

[0037] It should be noted that all simulated input and output operations and all simulated synchronization operations are performed on the simulator of the Geoscience Parallel File System, which may include multiple storage nodes, a storage controller, multiple metadata nodes and a metadata controller. The storage controller is used to control the storage nodes, and the metadata controller is used to control the metadata nodes. Multiple storage nodes can store user file data in a distributed manner, and multiple metadata nodes can store metadata information corresponding to user file data. The metadata information may include creation time, access control information and file size. For some small files, their file data and metadata information may be stored in metadata nodes. In addition, storage nodes and metadata nodes can be configured with different storage parameters, and the storage parameters may include parameters such as read / write rate, overhead and stripe size.

[0038] Among them, the storage node, storage controller, metadata node and metadata controller can all be composed of multiple logical processing units (LP). The logical processing units LP communicate with each other through events. Each event has its own independent timestamp, sender, receiver, event type and execution time. Each logical processing unit LP corresponds to an event queue, that is, a priority queue sorted by timestamp. Each logical processing unit LP also corresponds to a virtual time. As each event is executed, the virtual time jumps to the virtual timestamp of the event. An event may trigger another event, and the event is inserted into the priority queue according to the timestamp. The simulator of the GeoParallel File System can use a sequential scheduler so that only one event is executed at a time in the entire simulator.

[0039] In addition, when an I / O operation requests to open a file, the request is processed by the metadata controller. The metadata controller has multiple worker threads, each of which processes a metadata request. The metadata controller can look up metadata information in the metadata node. When the metadata controller has no available worker threads, new requests will be queued. The metadata controller is only involved in operations such as opening and closing files, but will not participate in subsequent file read and write operations. The read and write requests corresponding to the I / O operation will be read and written by the storage controller in the storage node.

[0040] In one embodiment, determining the performance evaluation result of the geospatial parallel file system based on all the simulated input and output operations and all the simulated synchronization operations includes: For each of the simulated input and output operations, determining a previous simulated input and output operation of the simulated input and output operation, wherein the previous simulated input and output operation has the same simulation thread number as the simulated input and output operation; determining a first simulation timestamp of the simulated input and output operation based on an operation time interval corresponding to the simulated input and output operation, and a first simulation timestamp and simulation parameters of the previous simulated input and output operation; the operation time interval is determined based on a difference between a timestamp of an input and output operation corresponding to the simulated input and output operation and a timestamp of an input and output operation previous to the input and output operation; Determine respective second simulation timestamps of all the simulated synchronization operations based on the execution status of all the simulated synchronization operations and all the simulated input and output operations and the simulation thread numbers; The performance evaluation result is determined based on all of the first simulation timestamps and all of the second simulation timestamps.

[0041] Specifically, the input and output operations can first be classified according to their corresponding simulation thread numbers. For each simulation input and output operation, the previous simulation input and output operation of the simulation input and output operation can be determined, so the previous simulation input and output operation has the same simulation thread number as the simulation input and output operation.

[0042] Furthermore, the first simulation timestamp of the simulated input / output operation can be determined based on the operation time interval corresponding to the simulated input / output operation, and the first simulation timestamp and simulation parameters of the previous simulated input / output operation. The operation time interval is determined based on the difference between the timestamp of the input / output operation corresponding to the simulated input / output operation and the previous input / output operation in the same thread of the input / output operation. Exemplarily, the operation time interval It can be expressed by the following formula:

[0043] in, Indicates Thread timestamps of input and output operations, Indicates Thread The timestamp of each input and output operation.

[0044] Furthermore, the second simulation timestamps of all simulated synchronization operations can be determined based on the execution status of all simulated synchronization operations and all simulated input and output operations and the simulation thread numbers. Taking the barrier operation as an example, the completion time of the barrier depends on the time when the last thread reaches the barrier, so the execution time of the synchronization operation is It can be expressed by the following formula:

[0045] in, Indicates the total number of threads (can be determined based on the thread number), Indicates Thread The timestamp of the simulation operation, Indicates Thread The timestamp of the simulation operation.

[0046] Then, respective second simulation timestamps of all simulated synchronization operations can be determined.

[0047] Furthermore, a performance evaluation result may be determined based on all first simulation timestamps and all second simulation timestamps. The performance evaluation result may include operation processing throughput and operation average processing delay, and the above two parameters may better reflect the performance of the geospatial parallel file system.

[0048] In the above embodiment, the first simulation timestamp of all simulated input and output operations and the second simulation timestamp of all simulated synchronization operations are determined respectively, and then the performance evaluation result is determined based on all the first simulation timestamps and all the second simulation timestamps. The determination of the performance evaluation result no longer depends on the executing geoscientific parallel file system, but can be completed through the simulated timestamp. The research on geoscientific parallel file system no longer depends on expensive hardware environment, which helps to accelerate the research and development of geoscientific parallel file system and provide more optimized storage solutions for large-scale data processing applications.

[0049] In one embodiment, determining the first simulation timestamp of the simulated input / output operation based on the operation time interval corresponding to the simulated input / output operation, and the first simulation timestamp and simulation parameters of the previous simulated input / output operation includes: Determining the operation execution time of the previous simulated input and output operation based on the simulation parameters of the previous simulated input and output operation; The operation execution duration and the first simulation timestamp of the previous simulation input / output operation and the operation time interval corresponding to the simulation input / output operation are summed, and the sum result is determined as the first simulation timestamp of the simulation input / output operation.

[0050] Specifically, the operation execution time of the previous simulated input / output operation may be determined based on the simulation parameters of the previous simulated input / output operation.

[0051] Furthermore, the operation execution duration and the first simulation timestamp of the previous simulation input / output operation and the operation time interval corresponding to the simulation input / output operation may be summed, and the summed result may be determined as the first simulation timestamp of the simulation input / output operation. Exemplarily, the first simulation timestamp may be determined by the following formula:

[0052] in, Indicates Thread The first simulation timestamp of the simulated input and output operation, Indicates Thread The first simulation timestamp of the simulation input and output operation is In the case of For the preset timestamp, Indicates Thread The execution time of a simulated input and output operation.

[0053] It is easy to understand that the process of determining the second simulation timestamp is similar to the process of determining the first simulation timestamp, that is, the operation execution duration and simulation timestamp of the previous simulation operation, and the operation time interval corresponding to the simulated synchronization operation are summed, and the sum result is determined as the second simulation timestamp of the simulated synchronization operation.

[0054] In the above embodiment, firstly, based on the simulation parameters of the previous simulated input / output operation, the operation execution duration of the previous simulated input / output operation is determined, and then the operation execution duration of the previous simulated input / output operation and the first simulation timestamp, as well as the operation time interval corresponding to the simulated input / output operation are summed, and the sum result is determined as the first simulation timestamp of the simulated input / output operation, which lays a foundation for evaluating the performance of the geoscientific parallel file system.

[0055] In one embodiment, the simulation parameters include a simulation operation type, a simulation offset, and a simulation access data amount; The determining, based on the simulation parameter of the previous simulation input / output operation, the operation execution duration of the previous simulation input / output operation comprises: In the case where the hardware configuration corresponding to the geospatial parallel file system is a mechanical hard disk, determining the head addressing duration based on the rotation speed corresponding to the mechanical hard disk and the simulated offset of the previous simulated input and output operation; Determining a unit byte read bandwidth corresponding to the mechanical hard disk based on the simulation operation type of the previous simulation input and output operation; Determine the medium reading duration based on the unit byte reading bandwidth and the simulated access data volume of the previous simulated input and output operation; The sum of the head addressing time and the medium reading time is determined as the operation execution time of the previous analog input and output operation.

[0056] Specifically, the simulation parameters may include the simulation operation type, simulation offset and simulation access data volume of the simulated input and output operation. When the hardware configuration corresponding to the Geoscience Parallel File System is a mechanical hard disk, the head addressing duration may be determined based on the rotation speed corresponding to the mechanical hard disk, and the simulation operation type and simulation offset of the previous simulated input and output operation. Among them, head addressing duration = simulation offset - head position / rotation speed. The unit byte read bandwidth corresponding to the mechanical hard disk may also be determined based on the simulation operation type of the previous simulated input and output operation, and then the medium read duration may be determined based on the unit byte read bandwidth corresponding to the mechanical hard disk and the simulated access data volume of the previous simulated input and output operation. Among them, the medium read duration = simulated access data volume / unit byte read bandwidth corresponding to the mechanical hard disk. Further, the sum of the head addressing duration and the medium read duration may be determined as the operation execution duration of the previous simulated input and output operation. That is, the operation execution duration = head addressing duration + medium read duration.

[0057] In the above embodiment, when the hardware configuration corresponding to the Geoscience Parallel File System is a mechanical hard disk, the head addressing time and the medium reading time are determined respectively, and finally the operation execution time of the previous simulated input and output operation can be determined more accurately, and the subsequent solution of the first simulation timestamp is facilitated.

[0058] In one embodiment, the simulation parameters include a simulation operation type and a simulation access data volume; The determining, based on the simulation parameter of the previous simulation input / output operation, the operation execution duration of the previous simulation input / output operation comprises: In a case where the hardware configuration corresponding to the geospatial parallel file system is a solid state drive, determining a unit byte read bandwidth corresponding to the solid state drive based on a simulation operation type of the previous simulation input and output operation; Determining the medium reading duration based on the unit byte reading bandwidth corresponding to the solid state drive and the simulated access data volume of the previous simulated input and output operation; The medium reading duration is determined as the operation execution duration of the previous simulated input and output operation.

[0059] Specifically, the simulation parameters may include the simulation operation type and the simulated access data volume of the simulated input / output operation. When the hardware configuration corresponding to the Geoscience Parallel File System is a solid-state hard disk, the unit byte read bandwidth corresponding to the solid-state hard disk may be determined based on the simulation operation type of the previous simulated input / output operation, and the medium read duration may be further determined based on the unit byte read bandwidth corresponding to the solid-state hard disk and the simulated access data volume of the previous simulated input / output operation. Among them, the medium read duration = simulated access data volume / unit byte read bandwidth corresponding to the solid-state hard disk. The medium read duration may be further determined as the operation execution duration of the previous simulated input / output operation. That is, the operation execution duration = medium read duration.

[0060] In the above embodiment, when the hardware configuration corresponding to the Geoscience Parallel File System is a solid-state hard disk, the medium reading time is determined, and ultimately the operation execution time of the previous simulated input and output operation can be determined more accurately, and the subsequent solution of the first simulation timestamp is facilitated.

[0061] The performance evaluation method of the geoscientific parallel file system provided by the present invention first obtains multiple input and output operations and multiple synchronization operations corresponding to the geoscientific parallel file system, and determines the operation tracking log based on all the input and output operations and all the synchronization operations; generates the simulated input and output operations corresponding to each input and output operation and the simulated synchronization operations corresponding to each synchronization operation based on the operation tracking log; and determines the performance evaluation result of the geoscientific parallel file system based on all the simulated input and output operations and all the simulated synchronization operations. The technical solution of the present invention determines the operation tracking log based on the acquired input and output operations and synchronization operations, and then determines the simulated input and output operations and the simulated synchronization operations according to the operation tracking log, and performs performance evaluation on the geoscientific parallel file system based on the simulated operations, which does not involve the operation of the geoscientific parallel file system itself, so that the performance evaluation result of the geoscientific parallel file system can be obtained without interfering with the execution of applications on the geoscientific parallel file system, so that the architecture and configuration of the geoscientific parallel file system can also be tested and studied according to the performance evaluation results.

[0062] The performance evaluation device of the geoscientific parallel file system provided by the present invention is described below. The performance evaluation device of the geoscientific parallel file system described below and the performance evaluation method of the geoscientific parallel file system described above can be referenced to each other.

[0063] Figure 2 is a schematic diagram of the structure of the performance evaluation device of the geoscience parallel file system provided by the present invention, such as Figure 2 As shown, the performance evaluation device 200 of the geospatial parallel file system includes: An acquisition module 210 is used to acquire multiple input and output operations and multiple synchronization operations corresponding to the geospatial parallel file system, and determine an operation tracking log based on all the input and output operations and all the synchronization operations; A simulation module 220, configured to generate, based on the operation tracking log, simulated input / output operations corresponding to the input / output operations and simulated synchronization operations corresponding to the synchronization operations; The evaluation module 230 is used to determine the performance evaluation result of the geoscientific parallel file system based on all the simulated input and output operations and all the simulated synchronization operations.

[0064] In one embodiment, the evaluation module 230 is specifically configured to: For each of the simulated input and output operations, determining a previous simulated input and output operation of the simulated input and output operation, wherein the previous simulated input and output operation has the same simulation thread number as the simulated input and output operation; determining a first simulation timestamp of the simulated input and output operation based on an operation time interval corresponding to the simulated input and output operation, and a first simulation timestamp and simulation parameters of the previous simulated input and output operation; the operation time interval is determined based on a difference between a timestamp of an input and output operation corresponding to the simulated input and output operation and a timestamp of an input and output operation previous to the input and output operation; Determine respective second simulation timestamps of all the simulated synchronization operations based on the execution status of all the simulated synchronization operations and all the simulated input and output operations and the simulation thread numbers; The performance evaluation result is determined based on all of the first simulation timestamps and all of the second simulation timestamps.

[0065] In one embodiment, the evaluation module 230 is further configured to: Determining the operation execution time of the previous simulated input and output operation based on the simulation parameters of the previous simulated input and output operation; The operation execution duration and the first simulation timestamp of the previous simulation input / output operation and the operation time interval corresponding to the simulation input / output operation are summed, and the sum result is determined as the first simulation timestamp of the simulation input / output operation.

[0066] In one embodiment, the simulation parameters include a simulation operation type, a simulation offset, and a simulation access data amount; the evaluation module 230 is further configured to: In the case where the hardware configuration corresponding to the geospatial parallel file system is a mechanical hard disk, determining the head addressing duration based on the rotation speed corresponding to the mechanical hard disk and the simulated offset of the previous simulated input and output operation; Determining a unit byte read bandwidth corresponding to the mechanical hard disk based on the simulation operation type of the previous simulation input and output operation; Determine the medium reading duration based on the unit byte reading bandwidth and the simulated access data volume of the previous simulated input and output operation; The sum of the head addressing time and the medium reading time is determined as the operation execution time of the previous analog input and output operation.

[0067] In one embodiment, the simulation parameters include the simulation operation type and the amount of simulated access data; the evaluation module 230 is further used to: In a case where the hardware configuration corresponding to the geospatial parallel file system is a solid state drive, determining a unit byte read bandwidth corresponding to the solid state drive based on a simulation operation type of the previous simulation input and output operation; Determining the medium reading duration based on the unit byte reading bandwidth corresponding to the solid state drive and the simulated access data volume of the previous simulated input and output operation; The medium reading duration is determined as the operation execution duration of the previous simulated input and output operation.

[0068] In one embodiment, the acquisition module 210 is specifically used for: The thread numbers, timestamps, operation types, accessed data volumes and offsets of all the input and output operations, and the thread numbers and timestamps corresponding to all the synchronization operations are recorded to determine the operation tracking log.

[0069] In one embodiment, the simulation module 220 is specifically used for: For each of the input and output operations, the thread number, operation type, access data volume and offset of the input and output operation are respectively determined as the simulated thread number, simulated operation type, simulated access data volume and simulated offset of the corresponding simulated input and output operation, to obtain the simulated input and output operation corresponding to the input and output operation; For each of the synchronization operations, the thread number corresponding to the synchronization operation is determined as the simulation thread number corresponding to the simulation synchronization operation, and the simulation synchronization operation corresponding to the synchronization operation is obtained.

[0070] The performance evaluation device of the geoscientific parallel file system provided by the present invention first obtains multiple input-output operations and multiple synchronization operations corresponding to the geoscientific parallel file system, and determines the operation tracking log based on all the input-output operations and all the synchronization operations; generates the simulated input-output operations corresponding to each input-output operation and the simulated synchronization operations corresponding to each synchronization operation based on the operation tracking log; and determines the performance evaluation result of the geoscientific parallel file system based on all the simulated input-output operations and all the simulated synchronization operations. The technical solution of the present invention determines the operation tracking log based on the obtained input-output operations and synchronization operations, and then determines the simulated input-output operations and the simulated synchronization operations according to the operation tracking log, and performs performance evaluation on the geoscientific parallel file system based on the simulated operations, which does not involve the operation of the geoscientific parallel file system itself, so that the performance evaluation result of the geoscientific parallel file system can be obtained without interfering with the execution of applications on the geoscientific parallel file system, so that the architecture and configuration of the geoscientific parallel file system can also be tested and studied according to the performance evaluation results.

[0071] Figure 3 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 3 As shown, the electronic device may include: a processor 310, a communications interface 320, a memory 330 and a communication bus 340, wherein the processor 310, the communications interface 320 and the memory 330 communicate with each other through the communication bus 340. The processor 310 may call the logic instructions in the memory 330 to execute the performance evaluation method of the geospatial parallel file system, and the method includes: Acquire multiple input / output operations and multiple synchronization operations corresponding to the geospatial parallel file system, and determine an operation tracking log based on all the input / output operations and all the synchronization operations; Generate, based on the operation tracking log, simulated input and output operations corresponding to the input and output operations and simulated synchronization operations corresponding to the synchronization operations; A performance evaluation result of the geoscientific parallel file system is determined based on all the simulated input and output operations and all the simulated synchronization operations.

[0072] In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0073] On the other hand, the present invention further provides a computer program product, the computer program product comprising a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the performance evaluation method of the geoscientific parallel file system provided by the above methods, the method comprising: Acquire multiple input / output operations and multiple synchronization operations corresponding to the geospatial parallel file system, and determine an operation tracking log based on all the input / output operations and all the synchronization operations; Generate, based on the operation tracking log, simulated input and output operations corresponding to the input and output operations and simulated synchronization operations corresponding to the synchronization operations; A performance evaluation result of the geoscientific parallel file system is determined based on all the simulated input and output operations and all the simulated synchronization operations.

[0074] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the performance evaluation method of the geoscientific parallel file system provided by the above methods, the method comprising: Acquire multiple input / output operations and multiple synchronization operations corresponding to the geospatial parallel file system, and determine an operation tracking log based on all the input / output operations and all the synchronization operations; Generate, based on the operation tracking log, simulated input and output operations corresponding to the input and output operations and simulated synchronization operations corresponding to the synchronization operations; A performance evaluation result of the geoscientific parallel file system is determined based on all the simulated input and output operations and all the simulated synchronization operations.

[0075] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0076] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0077] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A performance evaluation method for a geospatial parallel file system, characterized in that: include: Acquire multiple input / output operations and multiple synchronization operations corresponding to the geospatial parallel file system, and determine an operation tracking log based on all the input / output operations and all the synchronization operations; Generate, based on the operation tracking log, simulated input and output operations corresponding to the input and output operations and simulated synchronization operations corresponding to the synchronization operations; A performance evaluation result of the geoscientific parallel file system is determined based on all the simulated input and output operations and all the simulated synchronization operations.

2. The performance evaluation method of the geospatial parallel file system according to claim 1, characterized in that: The step of determining the performance evaluation result of the geoscientific parallel file system based on all the simulated input and output operations and all the simulated synchronization operations includes: For each of the simulated input and output operations, determining a previous simulated input and output operation of the simulated input and output operation, wherein the previous simulated input and output operation has the same simulation thread number as the simulated input and output operation; determining a first simulation timestamp of the simulated input and output operation based on an operation time interval corresponding to the simulated input and output operation, and a first simulation timestamp and simulation parameters of the previous simulated input and output operation; the operation time interval is determined based on a difference between a timestamp of an input and output operation corresponding to the simulated input and output operation and a timestamp of an input and output operation previous to the input and output operation; Determine respective second simulation timestamps of all the simulated synchronization operations based on the execution status of all the simulated synchronization operations and all the simulated input and output operations and the simulation thread numbers; The performance evaluation result is determined based on all of the first simulation timestamps and all of the second simulation timestamps.

3. The performance evaluation method of the geoscientific parallel file system according to claim 2, characterized in that: The determining the first simulation timestamp of the simulated input / output operation based on the operation time interval corresponding to the simulated input / output operation, and the first simulation timestamp and simulation parameters of the previous simulated input / output operation comprises: Determining the operation execution time of the previous simulated input and output operation based on the simulation parameters of the previous simulated input and output operation; The operation execution duration and the first simulation timestamp of the previous simulation input / output operation and the operation time interval corresponding to the simulation input / output operation are summed, and the sum result is determined as the first simulation timestamp of the simulation input / output operation.

4. The performance evaluation method of the geospatial parallel file system according to claim 3 is characterized in that: The simulation parameters include simulation operation type, simulation offset and simulation access data volume; The determining, based on the simulation parameter of the previous simulation input / output operation, the operation execution duration of the previous simulation input / output operation comprises: In the case where the hardware configuration corresponding to the geospatial parallel file system is a mechanical hard disk, determining the head addressing duration based on the rotation speed corresponding to the mechanical hard disk and the simulated offset of the previous simulated input and output operation; Determining a unit byte read bandwidth corresponding to the mechanical hard disk based on the simulation operation type of the previous simulation input and output operation; Determine the medium reading duration based on the unit byte reading bandwidth and the simulated access data volume of the previous simulated input and output operation; The sum of the head addressing time and the medium reading time is determined as the operation execution time of the previous analog input and output operation.

5. The performance evaluation method of the geospatial parallel file system according to claim 3, characterized in that: The simulation parameters include simulation operation type and simulation access data volume; The determining, based on the simulation parameter of the previous simulation input / output operation, the operation execution duration of the previous simulation input / output operation comprises: In a case where the hardware configuration corresponding to the geospatial parallel file system is a solid state drive, determining a unit byte read bandwidth corresponding to the solid state drive based on a simulation operation type of the previous simulation input and output operation; Determining the medium reading duration based on the unit byte reading bandwidth corresponding to the solid state drive and the simulated access data volume of the previous simulated input and output operation; The medium reading duration is determined as the operation execution duration of the previous simulated input and output operation.

6. The performance evaluation method of the geospatial parallel file system according to claim 4 or 5, characterized in that: The determining of the operation tracking log based on all the input and output operations and all the synchronization operations includes: The thread numbers, timestamps, operation types, accessed data volumes and offsets of all the input and output operations, and the thread numbers and timestamps corresponding to all the synchronization operations are recorded to determine the operation tracking log.

7. The performance evaluation method of the geospatial parallel file system according to claim 6, characterized in that: The generating, based on the operation tracking log, the simulated input / output operations corresponding to the input / output operations and the simulated synchronization operations corresponding to the synchronization operations respectively, comprises: For each of the input and output operations, the thread number, operation type, access data volume and offset of the input and output operation are respectively determined as the simulated thread number, simulated operation type, simulated access data volume and simulated offset of the corresponding simulated input and output operation, to obtain the simulated input and output operation corresponding to the input and output operation; For each of the synchronization operations, the thread number corresponding to the synchronization operation is determined as the simulation thread number corresponding to the simulation synchronization operation, and the simulation synchronization operation corresponding to the synchronization operation is obtained.

8. A performance evaluation device for a geospatial parallel file system, characterized in that: include: An acquisition module, used for acquiring a plurality of input and output operations and a plurality of synchronization operations corresponding to the geospatial parallel file system, and determining an operation tracking log based on all the input and output operations and all the synchronization operations; A simulation module, used for generating, based on the operation tracking log, simulated input and output operations corresponding to each of the input and output operations and simulated synchronization operations corresponding to each of the synchronization operations; An evaluation module is used to determine a performance evaluation result of the geoscientific parallel file system based on all the simulated input and output operations and all the simulated synchronization operations.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the performance evaluation method of the geospatial parallel file system as described in any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the performance evaluation method of the geospatial parallel file system as described in any one of claims 1 to 7 is implemented.

Images