A method and device for evaluating the performance of a discrete event simulation engine

Abstract

The present invention belongs to the field of simulation computing technology, and relates to a discrete event simulation engine performance evaluation method and device. The method includes: through a series of discrete events occurring simultaneously, testing the CPU time required for the simulation engine to complete all discrete events, and then calculating the average time consumption for discrete event processing as an engine performance evaluation index; testing the maximum number of discrete events processed per unit time when the simulation engine faces a large number of discrete events, as an engine load evaluation index; analyzing and testing the processing capacity of the simulation engine when facing abnormal conditions, and obtaining an engine robustness evaluation index; normalizing the engine performance evaluation index, engine load evaluation index, and engine robustness evaluation index of the simulation engine to form an engine performance radar chart, and obtaining an engine performance comprehensive evaluation index. The present invention can scientifically evaluate the performance of discrete event simulation engines from multiple perspectives, and provide guidance for the improvement of discrete event simulation engines.

Description

Technical Field

[0001] The invention belongs to the technical field of simulation computing, and in particular relates to a discrete event simulation engine performance evaluation method and device. Background Art

[0002] Discrete events are a common type of activity in many fields such as military, transportation, logistics, production, and services. Simulation for discrete events has always been an important means of conducting analysis and research in related fields. The simulation calculation of discrete events depends on the discrete event simulation engine, and its performance directly determines the efficiency of the simulation operation. Scientific and comprehensive evaluation of the simulation engine performance helps to discover the characteristics and deficiencies of the engine, which is crucial for the further development and improvement of the engine.

[0003] Currently, the performance evaluation of discrete event simulation engines mainly uses single performance indicators, such as simulation time and acceleration ratio. These indicators can reflect the overall efficiency of the engine when dealing with specific problems, but cannot comprehensively reflect the performance of all aspects of the engine. They are highly related to the simulation scenario and computer performance scale, and cannot effectively guide developers to optimize the engine. In addition, some researchers have established a test evaluation system for the engine event scheduling strategy, but have not considered event processing and other related performance influencing factors. Some researchers also collect the processing time of each discrete event during testing to quickly locate models with excessive runtime loss and help optimize event models.

[0004] At present, there is no comprehensive performance evaluation method for discrete event simulation engines, and the existing mature engine testing tools are not suitable for discrete event simulation. Therefore, it is necessary to build a new discrete event simulation engine performance evaluation method that can scientifically evaluate the simulation engine performance from multiple perspectives and provide guidance for engine improvement. Summary of the invention

[0005] The purpose of the present invention is to provide a discrete event simulation engine performance evaluation method and device, which comprehensively evaluates the performance of the simulation engine from two dimensions: computing performance and robustness.

[0006] To achieve this purpose, the present invention discloses a discrete event simulation engine performance evaluation method, the method comprising:

[0007] Step S1: Engine performance evaluation: Through a series of discrete events occurring simultaneously, the CPU time required for the simulation engine to complete all events is tested, and then the average processing time of discrete events is calculated based on the number of events.

[0008] Step S2: Engine load evaluation. Test the maximum number of discrete events that the simulation engine can process per unit time. max , used to measure the load capacity of the engine.

[0009] Step S3: Engine robustness evaluation. Analyze and test the processing ability of the simulation engine when facing abnormal conditions. The specific test content includes abnormal conditions such as the inability to load resources normally due to too many events / data, errors in the time of new events, errors in event execution, and conflicts between events of the same object at the same time.

[0010] Step S4: Comprehensive evaluation of engine performance: According to the evaluation results of engine performance, load and robustness, various indicators of the simulation engine are normalized to form an engine performance radar chart, and the comprehensive evaluation indicators of engine performance are obtained.

[0011] Furthermore, the step S1 includes:

[0012] In response to the needs of simulation engine performance evaluation, simulation engine test discrete event sets of different scales are established, including various discrete events that occur simultaneously. Then the simulation engine is used to test the total time required to complete all discrete event processing under different simulation scales, and the average discrete event processing time is calculated based on the number of events in the test set, that is:

[0013]

[0014] Where, T mean is the average time required for discrete event processing, T total is the total processing time of the discrete event test set, N total is the number of discrete events in the test set.

[0015] After completing three tests, the average discrete event processing time T of different scale (small scale, medium scale and large scale) test sets was obtained. mean S 、T mean M 、T mean L Then, according to the scale and frequency of the simulation problems processed by the engine during actual operation, the final average discrete event processing time is weighted and calculated, that is:

[0016]

[0017] Among them, α1, α2, and α3 are weighted coefficients of three types of test scenarios, and α1+α2+α3=1.

[0018] Furthermore, the step S2 comprises:

[0019] Construct a discrete event test set containing various typical events, including sequential events and simultaneous events, and use continuous massive discrete events to test the extreme processing performance of the simulation engine under high load. In order to accurately evaluate the engine load capacity, multiple rounds of tests are required, and statistical analysis is performed on the maximum number of discrete events processed per unit time in each test. max, repeat the test multiple times and calculate the mean As a measure of engine load capacity.

[0020]

[0021] Among them, E max 1 +E max 2 +···+E max n The maximum number of discrete events processed per unit time obtained from n tests.

[0022] Further, the step S3 includes a memory overflow test, a timing disorder test, an event anomaly test, and an event conflict test, wherein:

[0023] The memory overflow test uses a large amount of event data to artificially trigger a computing resource disaster, making the engine unable to complete the loading of all data, and evaluates the engine's ability to maintain normal operation or cope with this situation;

[0024] The timing disorder test generates new timing error events at multiple specified simulation time nodes. The design execution time of such events is earlier than the current simulation time, and the engine's processing ability in the face of timing sequence conflicts is evaluated;

[0025] The event exception test analyzes the engine's ability to monitor and handle abnormal events by adding abnormal events such as infinite loop events, input and output error events, and unexecuted events.

[0026] The event conflict test evaluates the engine's ability to handle conflict events by artificially introducing simulated conflicts, such as adding multiple simultaneous actions to the same object, moving multiple objects to the same location, etc.

[0027] Furthermore, in order to quantitatively analyze the robustness of the simulation engine, the step S3 first establishes a stepwise quantitative index for each robustness test, namely:

[0028] 0——No processing capability;

[0029] 1——The exception is found and reported, but cannot be processed;

[0030] 2——Discover the abnormality and report it, which can be partially handled;

[0031] 3——Discover the anomaly and report it, handle it completely, and continue the simulation.

[0032] On this basis, various robustness indicators are integrated and normalized to obtain the engine robustness evaluation index R total ',Right now:

[0033]

[0034] Among them, R int , R seq , R abn , R cla Represent four types of robustness test indicators respectively.

[0035] Further, the step S4 includes:

[0036] The min-max normalization method is used to process the consistency of the performance evaluation index and load evaluation index of the simulation engine to obtain a new performance evaluation index. and load assessment metrics Then, a three-dimensional radar chart is used to graphically represent the normalized indicators of the engine, which facilitates comparative analysis of the performance of the engine.

[0037] In addition, based on the constructed three types of normalized performance indicators, namely Forming a comprehensive evaluation index for discrete event simulation engines perf ,Right now:

[0038]

[0039] Among them, β1, β2, and β3 are weighted coefficients of the three types of simulation engine evaluation indicators, β1+β2+β3=1.

[0040] Another embodiment of the present invention provides a discrete event simulation engine performance evaluation device, comprising:

[0041] The engine performance evaluation module is used to test the CPU time required for the simulation engine to complete all discrete events through a series of discrete events that occur simultaneously, and then calculate the average processing time of discrete events based on the number of events as an engine performance evaluation indicator;

[0042] The engine load evaluation module is used to test the maximum number of discrete events that can be processed per unit time when the simulation engine is faced with massive discrete events. It is used to measure the engine's load capacity as an engine load evaluation indicator.

[0043] The engine robustness evaluation module is used to analyze and test the processing capability of the simulation engine when facing abnormal conditions and obtain the engine robustness evaluation index;

[0044] The engine performance comprehensive evaluation module is used to normalize the engine performance evaluation index, engine load evaluation index and engine robustness evaluation index of the simulation engine, form an engine performance radar chart, and obtain the engine performance comprehensive evaluation index.

[0045] Compared with the existing method, the present invention has the following beneficial effects:

[0046] 1. The proposed average discrete event processing time index takes into account the event processing efficiency of the engine under different simulation scales. Compared with the general total time index, this index reduces the impact of simulation scale on engine evaluation to a certain extent and is more conducive to the performance comparison of the engine under different simulation environments;

[0047] 2. The proposed engine load evaluation index can show the upper limit of the engine's ability to handle discrete events, which is of great significance for the comprehensive analysis and evaluation of engine performance.

[0048] 3. The proposed engine robustness evaluation index takes the engine's ability to handle abnormal situations into account, providing a new perspective for engine performance evaluation.

[0049] 4. Form standardized evaluation indicators and visual display methods to facilitate comparative analysis of the comprehensive performance of different simulation engines and different simulation scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a flow chart of a method and device for comprehensive evaluation of discrete event simulation engine performance disclosed in an embodiment of the present invention;

[0051] Figure 2 This is the event processing timing diagram during engine load evaluation;

[0052] Figure 3 It is a radar chart for comprehensive evaluation of engine performance that integrates three types of evaluation indicators. DETAILED DESCRIPTION

[0053] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those skilled in the art, several changes and improvements can be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0054] like Figure 1 As shown, the present invention discloses a discrete event simulation engine performance evaluation method, the method comprising the following steps:

[0055] Step S1: Engine performance evaluation: Through a series of discrete events occurring simultaneously, the CPU time required for the simulation engine to complete all events is tested, and then the average processing time of discrete events is calculated based on the number of events.

[0056] Step S2: Engine load evaluation. Test the maximum number of discrete events that the simulation engine can process per unit time. max , used to measure the load capacity of the engine.

[0057] Step S3: Engine robustness evaluation. Analyze and test the processing ability of the simulation engine when facing abnormal conditions. The specific test content includes abnormal conditions such as the inability to load resources normally due to too many events / data, errors in the time of new events, errors in event execution, and conflicts between events of the same object at the same time.

[0058] Step S4: Comprehensive evaluation of engine performance: According to the results of engine performance evaluation, engine load evaluation and engine robustness evaluation, various indicators of the simulation engine are normalized to form an engine performance radar chart, and the comprehensive evaluation indicators of engine performance are obtained.

[0059] In one embodiment of the present invention, step S1 comprises:

[0060] In order to evaluate the computational efficiency of the simulation engine at different simulation scales, the above tests are conducted in small-scale, medium-scale and large-scale simulation scenarios to obtain the average processing time of discrete events at different simulation scales. S 、SOE M 、SOE L , including various discrete events that occur simultaneously. Then the simulation engine is used to test the total time required to complete all discrete event processing under different simulation scales, and the average discrete event processing time is calculated based on the number of events in the test set, that is:

[0061]

[0062] Where, T mean is the average time required for discrete event processing, T total is the total processing time of the discrete event test set, N total is the number of discrete events in the test set.

[0063] After completing three tests, the average discrete event processing time T under different test sets is obtained. mean S 、T mean M 、T mean L Then, according to the scale and frequency of the simulation problems processed by the engine during actual operation, the final average discrete event processing time is weighted and calculated, that is:

[0064]

[0065] Among them, α1, α2, and α3 are weighted coefficients of the three types of test scenarios, and α1+α2+α3=1. The weighted coefficients are determined according to the frequency of occurrence of different workloads of the simulation engine. For example, for an engine that generally works under a low load state, the configurable test scenario weighted coefficients are: α1=0.5, α2=0.3, and α3=0.2.

[0066] In one embodiment of the present invention, step S2 comprises:

[0067] Construct a discrete event test set containing various typical events, including sequential events and simultaneous events, and use continuous massive discrete events to test the extreme processing performance of the simulation engine under high load. Specifically, when performing engine load testing, record the start time and completion time of each event processed by the engine. After the test is completed, all discrete event processing information is drawn into an event processing timing diagram according to the event sequence, and the maximum number of discrete events processed per unit time (ΔT) E is statistically analyzed. max .like Figure 2 The event processing sequence diagram for engine load evaluation is shown in the figure.

[0068] Figure 2 The horizontal lines of varying lengths represent each discrete event being processed. The horizontal axis corresponding to the starting point of the short horizontal line represents the start time of the event execution, and the horizontal axis corresponding to the end point represents the completion time of the event execution. Its length represents the time taken to process the event. In order to count the number of discrete events processed per unit time, for discrete events in the [T0, T1] time interval, only events with a start time after T0 and a completion time before T1 will be counted as processed events in the ΔT time period.

[0069] Repeat the test multiple times and calculate the mean As a measure of engine load capacity.

[0070]

[0071] Among them, E max 1 +E max 2 +···+E max n The test results obtained for n tests are respectively, that is, the maximum number of discrete events processed per unit time.

[0072] In one embodiment of the present invention, step S3 includes a memory overflow test, a timing disorder test, an event anomaly test, and an event conflict test, wherein:

[0073] The memory overflow test uses a large amount of event data to artificially cause a computing resource disaster, making the engine unable to complete the loading of all data, and evaluates the engine's ability to maintain normal operation or cope with this situation.

[0074] The timing violation test generates new timing error events at multiple specified simulation time nodes. The design execution time of such events is earlier than the current simulation time, and the engine's processing capabilities in the face of timing sequence conflicts are evaluated.

[0075] The event exception test analyzes the engine's ability to monitor and handle abnormal events by adding abnormal events such as infinite loop events, input and output error events, and unexecution events.

[0076] The event conflict test evaluates the engine's ability to handle conflict events by artificially introducing simulated conflicts, such as adding multiple simultaneous actions to the same object, moving multiple objects to the same location, etc.

[0077] In order to quantitatively analyze the robustness of the simulation engine, we first establish a step-by-step quantitative index for each robustness test, namely:

[0078] 0——No processing capability;

[0079] 1——The exception is found and reported, but cannot be processed;

[0080] 2——Discover the abnormality and report it, which can be partially handled;

[0081] 3——Discover the anomaly and report it, handle it completely, and continue the simulation.

[0082] On this basis, various robustness indicators are combined and normalized to obtain the engine robustness evaluation index Right now:

[0083]

[0084] Among them, R int , R seq , R abn , R cla They represent the step-wise quantitative indicators of memory overflow test, timing disorder test, event anomaly test, and event conflict test respectively; 3 is the maximum value of the step-wise quantitative indicator; and 4 is the number of step-wise quantitative indicators.

[0085] In one embodiment of the present invention, step S4 comprises:

[0086] The min-max normalization method is used to make the performance evaluation index and load evaluation index of the simulation engine consistent. The engine performance evaluation index, the average time consumption of discrete event processing, is normalized to obtain a new performance evaluation index. Right now:

[0087]

[0088] Among them, T max Indicates the maximum time consumption of discrete event processing in engine performance test, T min Indicates the minimum discrete event processing time in the engine performance test.

[0089] Normalize the engine load evaluation index - the maximum number of discrete events processed per unit time, and get a new load evaluation index Right now:

[0090]

[0091] like Figure 3 As shown in the engine performance comprehensive evaluation radar chart, the present invention uses normalized three types of engine evaluation indicators to graphically represent the evaluation results of the simulation engine. The radar chart can intuitively display the engine performance characteristics, which is convenient for comparative analysis of engine performance.

[0092] Based on the three types of normalized performance indicators that have been constructed, namely Forming a comprehensive evaluation index for discrete event simulation engines perf ,Right now:

[0093]

[0094] Among them, β1, β2, and β3 are weighted coefficients of the three types of simulation engine evaluation indicators, β1+β2+β3=1. The weighted coefficients should be designed according to the user's emphasis on different engine evaluation indicators. For example, if the engine load is not high and the engine's ability to run smoothly is more important, the values ​​of β1, β2, and β3 can be set to 0.3, 0.2, and 0.5 respectively; if the engine's ability to run under high load is more important, the values ​​of β1, β2, and β3 can be set to 0.35, 0.5, and 0.15 respectively.

[0095] Another embodiment of the present invention provides a discrete event simulation engine performance evaluation device, comprising:

[0096] The engine performance evaluation module is used to test the CPU time required for the simulation engine to complete all discrete events through a series of discrete events that occur simultaneously, and then calculate the average processing time of discrete events based on the number of events as an engine performance evaluation indicator;

[0097] The engine load evaluation module is used to test the maximum number of discrete events that can be processed per unit time when the simulation engine is faced with massive discrete events. It is used to measure the engine's load capacity as an engine load evaluation indicator.

[0098] The engine robustness evaluation module is used to analyze and test the processing capability of the simulation engine when facing abnormal conditions and obtain the engine robustness evaluation index;

[0099] The engine performance comprehensive evaluation module is used to normalize the engine performance evaluation index, engine load evaluation index and engine robustness evaluation index of the simulation engine, form an engine performance radar chart, and obtain the engine performance comprehensive evaluation index.

[0100] The specific implementation process of each module refers to the above description of the method of the present invention.

[0101] Another embodiment of the present invention provides a computer device (computer, server, smart phone, etc.), which includes a memory and a processor, the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program includes instructions for executing each step in the method of the present invention.

[0102] Another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, magnetic disk, optical disk), wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a computer, the steps of the method of the present invention are implemented.

[0103] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A discrete event simulation engine performance evaluation method, characterized in that: The following steps are involved: Through a series of discrete events that occur simultaneously, the CPU time required for the simulation engine to complete all discrete events is tested, and then the average processing time of discrete events is calculated based on the number of events as an engine performance evaluation indicator; The maximum number of discrete events processed per unit time when the test simulation engine faces a large number of discrete events is used as the engine load evaluation indicator; Analyze and test the processing capabilities of the simulation engine when facing abnormal conditions, and obtain the engine robustness evaluation index; Normalize the engine performance evaluation index, engine load evaluation index, and engine robustness evaluation index of the simulation engine to form an engine efficiency radar chart and obtain the engine efficiency comprehensive evaluation index; The processing capability of the analysis and testing simulation engine when facing abnormal conditions includes: Memory overflow test: Use a large amount of event data to artificially cause a computing resource disaster, so that the engine cannot complete the loading of all data, and evaluate the engine's ability to maintain normal operation or cope with this situation; Timing disorder test: Generate new time error events at multiple specified simulation time nodes. The design execution time of such events is earlier than the current simulation time. The engine's processing ability in the face of time sequence conflicts is evaluated. Event anomaly testing: By adding abnormal events, the engine's monitoring and processing capabilities for abnormal events are analyzed; Event conflict test: By artificially introducing simulated conflicts, the engine's ability to handle conflict events is evaluated.

2. The method according to claim 1, characterized in that The engine performance evaluation index is calculated using the following steps: To meet the needs of simulation engine performance evaluation, discrete event sets of simulation engine tests of different sizes are established, including various discrete events occurring simultaneously; The simulation engine is used to test the total time required to complete all discrete event processing at different scales, and the average discrete event processing time is calculated based on the number of events in the test set, that is: Where, T mean is the average time required for discrete event processing, T total is the total processing time of the discrete event test set, N total is the number of discrete events in the test set; After completing three tests, the average discrete event processing time T for small-scale, medium-scale, and large-scale test sets was obtained. mean S 、T mean M 、T mean L Then, according to the scale and frequency of the simulation problems processed by the engine during actual operation, the final average discrete event processing time is weighted and calculated, that is: Among them, α1, α2, and α3 are weighted coefficients of three types of test scenarios, and α1+α2+α3=1.

3. The method according to claim 1, characterized in that The engine load evaluation index is calculated using the following steps: Construct a discrete event test set containing various typical events, including sequential events and simultaneous events, and use continuous massive discrete events to test the extreme processing performance of the simulation engine under high load; Statistical analysis of the maximum number of discrete events processed per unit time in each test, repeated multiple tests, and calculated the mean As an engine load assessment indicator: Among them, E max 1 +E max 2 +···+E max n The maximum number of discrete events processed per unit time obtained from n tests.

4. The method according to claim 1, characterized in that The abnormal events include infinite loop events, input / output error events, and unexecution events; the simulation conflicts include adding multiple simultaneous execution actions to the same object and moving multiple objects to the same position.

5. The method according to claim 1, characterized in that: The engine robustness evaluation index is calculated using the following steps: Conduct robustness testing, including memory overflow testing, timing disorder testing, event anomaly testing, and event conflict testing; A step-by-step quantitative index is established for each robustness test, namely: 0 - no processing capability; 1 - abnormality is found and reported, but cannot be processed; 2 - abnormality is found and reported, and can be partially processed; 3 - abnormality is found and reported, and can be fully processed and simulation continues; Comprehensively analyze and normalize various robustness indicators to obtain engine robustness evaluation indicators Right now: Among them, R int , R seq , R abn , R cla They represent the step-wise quantitative indicators of memory overflow test, timing disorder test, event anomaly test, and event conflict test respectively; 3 is the maximum value of the step-wise quantitative indicator; and 4 is the number of step-wise quantitative indicators.

6. The method according to claim 5, characterized in that The engine performance comprehensive evaluation index is obtained, including: The min-max normalization method is used to process the consistency of the performance evaluation index and load evaluation index of the simulation engine to obtain a new performance evaluation index. and load assessment metrics Based on the three types of normalized performance indicators that have been constructed, namely Forming a comprehensive evaluation index for discrete event simulation engines perf ,Right now: Among them, β1, β2, and β3 are weighted coefficients of the three types of simulation engine evaluation indicators, β1+β2+β3=1.

7. A discrete event simulation engine performance evaluation device, characterized in that: include: The engine performance evaluation module is used to test the CPU time required for the simulation engine to complete all discrete events through a series of discrete events that occur simultaneously, and then calculate the average processing time of discrete events based on the number of events as an engine performance evaluation indicator; The engine load evaluation module is used to test the maximum number of discrete events processed per unit time when the simulation engine faces a large number of discrete events, which serves as an engine load evaluation indicator; The engine robustness evaluation module is used to analyze and test the processing capability of the simulation engine when facing abnormal conditions and obtain the engine robustness evaluation index; The engine performance comprehensive evaluation module is used to normalize the engine performance evaluation index, engine load evaluation index and engine robustness evaluation index of the simulation engine, form an engine performance radar chart, and obtain the engine performance comprehensive evaluation index; The processing capability of the analysis and testing simulation engine when facing abnormal conditions includes: Memory overflow test: Use a large amount of event data to artificially cause a computing resource disaster, so that the engine cannot complete the loading of all data, and evaluate the engine's ability to maintain normal operation or cope with this situation; Timing disorder test: Generate new time error events at multiple specified simulation time nodes. The design execution time of such events is earlier than the current simulation time. The engine's processing ability in the face of time sequence conflicts is evaluated. Event anomaly testing: By adding abnormal events, the engine's monitoring and processing capabilities for abnormal events are analyzed; Event conflict test: By artificially introducing simulated conflicts, the engine's ability to handle conflict events is evaluated.

8. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a computer, the method according to any one of claims 1 to 6 is implemented.

Images