Real-time data acquisition, processing and storage system and method for simulation process

Abstract

This invention relates to a real-time acquisition, processing, and storage system and method for simulation process data. The system includes: a data acquisition module for caching continuously pushed simulation process data from outside the system as messages and achieving parallel processing of the simulation process data through multiple acquisition and processing processes; a data sorting module for real-time sorting of the simulation process data according to timestamps and periodically reading, forwarding, and deleting sorted data accumulated in the sorted heap; a data storage and query analysis module for storing the simulation process data and providing fast query and analysis functions for the stored simulation process data; and a sensing and scheduling module for detecting the working status of the real-time acquisition, processing, and storage system for simulation process data and coordinating the overall operation of each module based on the detection results and the simulation scenario's operating status. This invention can automatically sense changes in the output of simulation process data and dynamically adjust the acquisition and processing capabilities.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of aerospace equipment system simulation, and in particular to a simulation process data real-time acquisition processing and storage system and method. BACKGROUND

[0002] The system simulation process data real-time acquisition processing and storage management system is an important component of system simulation. It is used to realize real-time acquisition processing and storage of system simulation process data, support business applications with real-time simulation process data as input parameters, and provide efficient query and reading functions for data utilization.

[0003] With the continuous development of system simulation towards service and scale, the traditional system simulation data support system has short board problems in data real-time acquisition processing capacity and query efficiency. In terms of real-time data acquisition processing, due to the changes of simulation deduction scene acceleration ratio and the number of parallel simulation deduction scenes, the output data volume changes dramatically. The traditional method realizes the acquisition processing program through dynamic multi-threading, which corresponds to an independent acquisition processing thread for each scene, and to some extent can solve the problem of changes in the number of parallel simulation deduction scenes. However, due to the operation of a single server node, when the number of parallel simulation deduction scenes is large and when a large number of concurrent simulation deduction test scenes are running, there is still a problem of processing capacity bottleneck. The traditional method realizes horizontal expansion by combining distributed technology based on dynamic multi-threading, which can fully utilize the computing power of multiple nodes to improve the real-time processing capacity of simulation process data. However, this brings about other problems, such as the dynamic reduction of the number of entities in the simulation process, which reduces the data acquisition processing volume, and the inability to effectively release the occupied resources, resulting in resource overuse. When the simulation scene is a complex system containing a large number of entities, and when the simulation scene is running at a high acceleration ratio, data congestion and acquisition processing capacity lag are still likely to occur, which further leads to the lag of subsequent real-time efficiency evaluation calculation and real-time situation display business applications. SUMMARY

[0004] To solve the technical problems existing in the prior art, the present application aims to provide a simulation process data real-time acquisition processing and storage system and method, which automatically senses the output change volume of simulation process data and realizes dynamic adjustment of acquisition processing capacity.

[0005] To achieve the above-mentioned application purpose, the present application provides a simulation process data real-time acquisition processing and storage system, comprising:

[0006] A data acquisition module for buffering simulation process data continuously pushed from outside the system in the form of a message, and realizing parallel processing of simulation process data through multiple acquisition processing processes;

[0007] The data sorting module is used to sort the simulation process data in real time according to the timestamp order, and to periodically read, forward, and delete the sorted data accumulated in the sort heap.

[0008] The data storage and query analysis module is used to store simulation process data and provide fast query and analysis functions for the stored simulation process data;

[0009] The perception and scheduling module is used to detect the working status of the simulation process data acquisition, processing and storage system in real time, and to coordinate the work of each module in an overall manner based on the detection results and the simulation scenario operation status.

[0010] According to one aspect of the present invention, the data acquisition module includes:

[0011] The data caching submodule caches messages in the form of topics and partitions, providing a message push interface for simulation process data to the simulation engine.

[0012] The data acquisition and processing submodule is used for parallel acquisition and processing of simulation process data. It packages the processed simulation process data into groups with the same timestamp and sends the group of data and the pointer position information of the last sorting insertion to the data sorting module.

[0013] The data caching submodule is also used to provide a message extraction interface for the acquisition and processing submodule to extract cached messages.

[0014] According to one aspect of the present invention, the data sorting module includes:

[0015] The sorting and insertion submodule is used to sort the grouped simulation process data packaged by the acquisition and processing submodule. Based on the previous pointer position information, it finds the sorting and insertion position according to the timestamp, completes the real-time sorting operation according to the timestamp order, and returns the pointer position information of the current sorting and insertion to the acquisition and processing submodule.

[0016] The data sequential reading and deletion submodule is used to periodically detect whether the number of sorted data accumulated in the sorted heap has reached a preset threshold h. When the preset threshold h is reached, the first h data are read and sent to the storage and query analysis module, and at the same time to the external situation display software.

[0017] According to one aspect of the present invention, the data storage and query analysis module includes:

[0018] The data storage submodule is built on the Hive data warehouse tool and adopts a distributed data storage architecture;

[0019] The data query and analysis submodule is built on Presto tool components and is used for fast querying and efficient computational analysis of Hive data.

[0020] According to one aspect of the present invention, the sensing and scheduling module includes:

[0021] The status awareness submodule periodically monitors the status parameters of the data acquisition and processing module, the data sorting module, and the data storage and query analysis module, and calculates the average data processing rate p based on the data acquisition and processing rate v, providing input parameters for the calculation and splitting of topics and partitions;

[0022] The dynamic scheduling submodule is used to read the scenario file when the simulation engine is about to start a scenario, and calculate and split the Kafka topics and partitions based on the total number of output ports in the scenario;

[0023] The dynamic scheduling submodule is also used to send sorting process scheduling commands to the data sorting module and data storage table creation commands to the data storage and query analysis module.

[0024] The dynamic scheduling submodule is also used to monitor the backlog of messages in each partition during the simulation engine's operation. When the backlog of messages exceeds the threshold t for a duration of s1, and when there is an idle region exceeding 1 for a duration of s2, the module sends messages to the simulation engine to indicate the partitions exceeding the threshold and the idle partitions. Based on the agreed feedback results, the module also sends an adjustment command for the parallel acquisition strategy to the data acquisition and processing module.

[0025] According to one aspect of the present invention, the data caching submodule is further configured to provide the dynamic scheduling submodule with a partition message backlog status reading interface, a topic and partition modification interface;

[0026] The acquisition and processing submodule is also used to provide an acquisition and processing process scheduling interface for the dynamic scheduling submodule.

[0027] According to one aspect of the present invention, a method for real-time acquisition, processing, and storage of simulation process data based on a simulation process data real-time acquisition, processing, and storage system as described in any of the above technical solutions is proposed, comprising:

[0028] Before the simulation process, complete the initialization preparation;

[0029] After the simulation simulation task is started, the data acquisition module caches the simulation process data continuously pushed from outside the system in the form of messages, and realizes parallel processing of the simulation process data through multiple acquisition and processing processes.

[0030] The data sorting module is used to sort the simulation process data in real time according to timestamp order, and to periodically read, forward, and delete the sorted data accumulated in the sort heap.

[0031] The data storage and query analysis module is used to store the simulation process data and provide a fast query and analysis function for the stored simulation process data.

[0032] The perception and scheduling module is used to detect the working state of the real-time acquisition, processing and storage system of the simulation process data, and overall coordinate the work of each module according to the detection results and the operation state of the simulation scenario.

[0033] According to one aspect of the present invention, during the simulation deduction process, the working state of the real-time acquisition, processing and storage system of the simulation process data is detected, logical judgment is performed according to the detection results and the operation state of the simulation scenario, and the overall coordination of each module is carried out based on the logical judgment results. The specific logical judgment includes:

[0034] When d≥t, w≥s1, and there is an idle partition, notify the simulation engine, and the simulation engine will match half of the number of ports in the port set of the message squeezing partition to the non-backlogged partition.

[0035] When d≥t, w≥s1, and there is no idle partition, create a new partition and create an acquisition and processing process corresponding to this partition, notify the simulation engine, and the simulation engine will match half of the number of ports in the port set of the message squeezing partition to the non-backlogged partition.

[0036] When d<t, w≥s2, and the number of idle partitions>1, send the idle partition set information to the simulation engine. The simulation engine will rematch the ports of all idle partitions to one of the idle partitions and feedback the matching result to the perception and scheduling module. The perception and scheduling module deletes the remaining idle partitions and the corresponding acquisition and processing processes.

[0037] Where, d is the backlog amount of partition messages, t is the backlog threshold of partition messages, w is the duration, s1 is the duration when the backlog amount of messages is greater than the threshold t, and s2 is the duration when the idle area appears greater than 1.

[0038] According to one aspect of the present invention, when the simulation deduction task is paused, the perception and scheduling module pauses the logical judgment process and controls the parameters of the partition, the acquisition and processing process, and the sorting process to remain unchanged.

[0039] When the simulation deduction task is started, the logical judgment process is enabled.

[0040] According to one aspect of the present invention, when the simulation deduction task is stopped, the perception and scheduling module stops the logical judgment process and enables the following judgment conditions:

[0041] When the backlog status of all partitions is empty and w≥s2, delete the topic, stop the acquisition and processing process, and stop the sorting process.

[0042] Where w is the duration and s2 is the duration when the idle region has a duration greater than 1.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] According to one aspect of the present invention, by constructing four major modules—a data acquisition module, a data sorting module, a data storage and query analysis module, and a sensing and scheduling module—the parameters closely related to the amount of data in the simulation scenario are comprehensively considered, including the number of ports of the entity components in the simulation scenario and the speedup ratio used to run the simulation scenario. The number of initial acquisition and processing processes and the number of partitions are calculated using the above parameters, thereby avoiding the problem of large amounts of data accumulation in the simulation process.

[0045] By monitoring the data backlog in each partition and coordinating with the simulation engine, the matching between entity component ports and partitions is dynamically adjusted, thereby achieving continuous dynamic optimization of the data acquisition and processing capabilities during the parallel data acquisition and processing process in the simulation.

[0046] This invention provides a system and method for real-time acquisition and processing of system simulation data, which can handle massive amounts of entities, high concurrency, and multiple samples. It is adapted to the service-oriented and large-scale development trend of system simulation and provides technical support for real-time situation display and real-time online analysis and evaluation. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0048] Figure 1 This diagram illustrates the flowchart of the simulation process data real-time acquisition, processing, and storage system method in an embodiment of the present invention.

[0049] Figure 2 This illustration illustrates the pre-start preparation sub-process for the simulation scenario in this embodiment of the invention.

[0050] Figure 3 This illustration illustrates the sub-process for dynamically optimizing data acquisition and processing capabilities in an embodiment of the present invention.

[0051] Figure 4 This illustration illustrates a simulation scenario-based pause subprocess in an embodiment of the present invention.

[0052] Figure 5 This illustration illustrates a simulated scenario-based stopping subprocess in an embodiment of the present invention.

[0053] Figure 6 This illustration illustrates the data acquisition, processing, and storage sub-processes in an embodiment of the present invention.

[0054] Figure 7 The illustrations represent the functional modules of the simulation process data real-time acquisition, processing, and storage system in this embodiment of the invention, as well as the interfaces between modules and the external system. Detailed Implementation

[0055] The description of the embodiments in this specification should be taken in conjunction with the accompanying drawings, which should form part of the complete specification. In the drawings, the shape or thickness of the embodiments may be exaggerated and may be indicated in a simplified or convenient manner. Furthermore, parts of the various structures in the drawings will be described separately; it is worth noting that elements not shown in the figures or not described in words are in a form known to those skilled in the art.

[0056] The descriptions of the embodiments herein, including any references to directions and orientations, are for ease of description only and should not be construed as limiting the scope of the invention. The following description of preferred embodiments involves combinations of features, which may exist independently or in combination; the invention is not particularly limited to the preferred embodiments. The scope of the invention is defined by the claims.

[0057] like Figures 1 to 7 As shown, the present invention provides a simulation process data real-time acquisition, processing, and storage system, comprising:

[0058] The data acquisition module is used to cache the simulation process data continuously pushed from outside the system in the form of messages, and to realize parallel processing of the simulation process data through multiple acquisition and processing processes;

[0059] The data sorting module is used to sort the simulation process data in real time according to the timestamp order, and to periodically read, forward, and delete the sorted data accumulated in the sort heap.

[0060] The data storage and query analysis module is used to store simulation process data and provide fast query and analysis functions for the stored simulation process data;

[0061] The perception and scheduling module is used to detect the working status of the simulation process data acquisition, processing and storage system in real time, and to coordinate the work of each module in an overall manner based on the detection results and the simulation scenario operation status.

[0062] In this embodiment, by constructing four major modules—a data acquisition module, a data sorting module, a data storage and query analysis module, and a perception and scheduling module—it can fully take into account factors closely related to the amount of simulation data, such as the number of entities within the simulation scenario, the number of parallel simulation scenarios, and the speedup ratio. This enables on-demand, dynamically scalable data acquisition and processing capabilities, as well as efficient simulation process data query and retrieval capabilities. It solves the problem of data backlog caused by the inability of traditional system simulation data support systems to adapt to the real-time data processing needs generated by service-oriented and large-scale system simulations. It also avoids the problem of lag in subsequent real-time performance evaluation calculations and real-time situation display, ensuring the real-time performance of situation display and performance evaluation calculations.

[0063] Furthermore, through technologies such as multi-process parallel processing, dynamic monitoring and scheduling, and big data, the system can automatically sense changes in the data output during the simulation process and dynamically adjust the acquisition and processing capabilities. This ensures both the real-time performance of acquisition and processing when the amount of output data increases during the simulation process and timely release of surplus operating resources when the amount of data decreases.

[0064] In one embodiment of the present invention, preferably, the data acquisition module includes:

[0065] The data caching submodule caches messages in the form of topics and partitions, providing a message push interface for simulation process data to the simulation engine.

[0066] The data acquisition and processing submodule is used for parallel acquisition and processing of simulation process data. It packages the processed simulation process data into groups with the same timestamp and sends the group of data and the pointer position information of the last sorting insertion to the data sorting module.

[0067] The data caching submodule is also used to provide a message extraction interface for the acquisition and processing submodule to extract cached messages.

[0068] In one embodiment of the present invention, preferably, the data sorting module includes:

[0069] The sorting and insertion submodule is used to sort the grouped simulation process data packaged by the acquisition and processing submodule. Based on the previous pointer position information, it finds the sorting and insertion position according to the timestamp, completes the real-time sorting operation according to the timestamp order, and returns the pointer position information of the current sorting and insertion to the acquisition and processing submodule.

[0070] The data sequential reading and deletion submodule is used to periodically detect whether the number of sorted data accumulated in the sorted heap has reached a preset threshold h. When the preset threshold h is reached, the first h data are read and sent to the storage and query analysis module, and at the same time to the external situation display software.

[0071] In one embodiment of the present invention, preferably, the data storage and query analysis module includes:

[0072] The data storage submodule is built on the Hive data warehouse tool and adopts a distributed data storage architecture to provide high-concurrency data writing capabilities.

[0073] The data query and analysis submodule is built on the Presto tool components and has a standard SQL interaction interface. It uses memory acceleration technology to achieve fast querying and efficient calculation and analysis of Hive data.

[0074] In one embodiment of the present invention, preferably, the sensing and scheduling module includes:

[0075] The status awareness submodule periodically monitors the status parameters of the data acquisition and processing module, data sorting module, and data storage and query analysis module, such as message backlog, data acquisition and processing rate, number of acquisition and processing processes, number of data sorting processes, data storage volume, number of data storage tables, etc. It provides a human-machine interface for displaying each parameter and calculates the average data processing rate p based on the data acquisition and processing rate v, providing input parameters for the calculation and splitting of topics and partitions.

[0076] The dynamic scheduling submodule is used to read the scenario file when the simulation engine is about to start a scenario, and parse out all equipment, components and output port data in the scenario file. Since the equipment, components and output ports are contained in descending order, and the number of output ports is one of the direct positive correlation factors of the data output rate in the simulation process, a unique identifier ID is established for each output port. The number of Kafka topics and partitions is calculated and split according to the total number of output ports, and the unique identifier ID of the output port is mapped to the topic and partition to establish a mapping relationship, thus creating conditions for parallel data acquisition and processing.

[0077] The dynamic scheduling submodule is also used to send a sorting process scheduling command to the data sorting module to start a new sorting process for the simulation scenario to be run; and to send a data storage table creation command to the data storage and query analysis module to create a table for the simulation scenario to be run.

[0078] The dynamic scheduling submodule is also used to monitor the backlog of messages in each partition during the simulation engine's operation. When the backlog of messages exceeds the threshold t for a duration of s1, and when there is an idle region exceeding 1 for a duration of s2, the module sends messages to the simulation engine to indicate the partitions exceeding the threshold and the idle partitions. Based on the agreed feedback results, the module also sends an adjustment command for the parallel acquisition strategy to the data acquisition and processing module.

[0079] In one embodiment of the present invention, preferably, the data caching submodule is further configured to provide the dynamic scheduling submodule with a partition message backlog status reading interface, a topic and partition modification interface;

[0080] The acquisition and processing submodule is also used to provide an acquisition and processing process scheduling interface for the dynamic scheduling submodule.

[0081] like Figures 1 to 6 As shown, according to one aspect of the present invention, a method for real-time acquisition, processing, and storage of simulation process data based on a simulation process data real-time acquisition, processing, and storage system as described in any of the above technical solutions is proposed, comprising:

[0082] Before the simulation process, complete the initialization preparation;

[0083] After the simulation simulation task is started, the data acquisition module caches the simulation process data continuously pushed from outside the system in the form of messages, and realizes parallel processing of the simulation process data through multiple acquisition and processing processes.

[0084] The data sorting module is used to sort the simulation process data in real time according to timestamp order, and to periodically read, forward, and delete the sorted data accumulated in the sort heap.

[0085] The system utilizes a data storage and query analysis module to store simulation process data and provides fast query and analysis capabilities for the stored simulation process data.

[0086] The sensing and scheduling module is used to detect the working status of the simulation process data acquisition, processing and storage system in real time, and to coordinate the work of each module in an overall manner based on the detection results and the simulation scenario operation status.

[0087] In one embodiment of the present invention, preferably, during the simulation process, the working status of the real-time data acquisition, processing, and storage system is detected, and logical judgments are made based on the detection results and the simulated operating status. Based on the logical judgment results, the various modules are coordinated as a whole. The logical judgments specifically include:

[0088] When d≥t, w≥s1, and there is a free partition, notify the simulation engine. The simulation engine will match half of the ports in the port set of the squeezed partition that match the message to the non-backlogged partition.

[0089] When d≥t, w≥s1, and there is no free partition, create a new partition and create a corresponding acquisition and processing process for the partition. Notify the simulation engine, and the simulation engine will match half of the ports in the port set of the squeezed partition that match the message to the non-backlogged partition.

[0090] When d < t, w ≥ s2, and the number of idle partitions > 1, send the information of the set of idle partitions to the simulation engine. The simulation engine rematches the ports of all idle partitions to one of the idle partitions and feeds back the matching result to the perception and scheduling module. The perception and scheduling module deletes the remaining idle partitions and the corresponding acquisition and processing processes.

[0091] Where d is the backlog of partition messages, t is the threshold of partition message backlog, w is the duration, s1 is the duration when the message backlog is greater than the threshold t, and s2 is the duration when the idle area appears greater than 1.

[0092] In an embodiment of the present invention, preferably, when the simulation deduction task is paused, the perception and scheduling module pauses the logical judgment process and controls the parameters of partitions, acquisition and processing processes, and sorting processes unchanged.

[0093] When the simulation deduction task is started, the logical judgment process is enabled.

[0094] In an embodiment of the present invention, preferably, when the simulation deduction task stops, the perception and scheduling module stops the logical judgment process and enables the following judgment conditions:

[0095] When the backlog status of all partitions is empty and w ≥ s2, delete the topic, stop the acquisition and processing process, and stop the sorting process.

[0096] Where w is the duration and s2 is the duration when the idle area appears greater than 1.

[0097] In an embodiment of the present invention, preferably, the method for real-time acquisition, processing, and storage of simulation process data includes the following processes: the sub-process of preparation before the start of simulation scenario (initialization preparation), the sub-process of data acquisition, processing, and storage after the start of simulation scenario, the sub-process of stopping simulation scenario, the sub-process of pausing simulation scenario, and the sub-process of dynamically optimizing acquisition and processing capabilities.

[0098] Specifically as follows:

[0099] (1) Before the simulation deduction engine runs the simulation scenario, obtain the scenario file for preparing the simulation deduction, parse the parameters in the simulation scenario file, including the ports of each component in all simulation entities (at least including simulation entity ID, component ID, output port ID), and the acceleration ratio k used in the simulation deduction. Statistically calculate the total number of ports m, and calculate the initial number of partitions n that Kafka needs to create for this simulation scenario to receive the simulation process data continuously output when this simulation scenario runs on the simulation deduction engine. The specific formula is as follows:

[0100]

[0101] Where p is the average processing rate of a single acquisition and processing process, and ε is an adjustment coefficient, which defaults to 1, providing users with a means to manually intervene in the initial number of partitions.

[0102] (2) Add one topic to the simulation task, create n partitions, and start the acquisition and processing process of n corresponding partitions.

[0103] (3) Start a sorting process for the simulation task to realize the task of sorting the data output by multiple acquisition and processing processes according to the timestamp.

[0104] (4) Create HIVE table storage space for this simulation task to provide conditions for the storage and query of real-time simulation data.

[0105] (5) After completing the above preparations, the scheduling module sends the new topics and partition information that Kafka has added for the simulation task to the simulation engine.

[0106] The above steps constitute the preparatory sub-process (initialization preparation) before starting the simulation scenario. Figure 2 As shown.

[0107] (6) After the simulation task is started, the ports of each component in all simulation entities output simulation process data and push it to each partition in the form of messages. The port ID is the partition key, and data is pushed to different partitions respectively. In the initial state, the number of ports corresponding to each partition is the same.

[0108] (7) During the simulation and deduction task, detect various parameters of the data processing process, obtain the message backlog d of each Kafka partition, the message processing time c of each partition, and calculate the average processing rate p of a single acquisition and processing process based on the processing rate v of the data acquisition and processing process of each partition, so as to provide input parameters for calculating the number of processing partitions for the next simulation scenario.

[0109] During the simulation process, the working status of the real-time data acquisition, processing, and storage system is monitored. Based on the monitoring results and the simulated operating status, logical judgments are made, and the overall coordination of each module is performed based on the logical judgment results. The logical judgments specifically include:

[0110] ① When d≥t, w≥s1, and there is a free partition, notify the simulation engine. The simulation engine will match half of the ports in the port set of the squeezed partition that match the message to the non-backlogged partition.

[0111] ② When d≥t, w≥s1, and there is no free partition, create a new partition and create a corresponding acquisition and processing process for the partition. Notify the simulation engine, and the simulation engine will match half of the ports in the port set of the squeezed partition that match the message to the non-backlogged partition.

[0112] ③ When d < t, w ≥ s2, and the number of free partitions > 1, send the free partition set information to the simulation engine. The simulation engine rematches the ports of all free partitions to one of the free partitions and feeds back the matching result to the perception and scheduling module. The perception and scheduling module deletes the remaining free partitions and the corresponding acquisition and processing processes.

[0113] Among them, t is the partition message backlog threshold, w is the duration, s1 is the duration when the message backlog is greater than the threshold t, and s2 is the duration when the free area appears greater than 1.

[0114] The above steps are the sub-process of dynamically optimizing the acquisition and processing capabilities, as Figure 3 shown.

[0115] (8) All acquisition and processing processes start to acquire data. The acquisition and processing process reads the message set in the corresponding interval, analyzes the data in the message set, and takes a group of data with continuous same timestamps. The group of data and the pointer position information (pos) of the previous sorted insertion are sent to the sorting module.

[0116] (9) The sorting module finds the insertion position according to the pointer position information (pos), performs sorting processing, returns the pointer position information (pos) after sorting, and updates the sorting insertion pointer information (pos) of the acquisition and processing process after sorting.

[0117] (10) When the number of sorted data entries accumulated in the sorting heap is greater than h, the sorting module reads the first h pieces of data in the sorting heap and deletes the h pieces of data from the sorting heap.

[0118] (11) After the sorting module reads the first h pieces of data, on the one hand, it sends the data to the front-end display. On the other hand, it stores the data in the Hive database of the data storage and query analysis module.

[0119] (12) The data storage and query analysis module improves the query and analysis capabilities of the Hive database through distributed memory acceleration-related technologies.

[0120] The above is the sub-process of data acquisition, processing and storage, as Figure 6 shown.

[0121] (13) When the simulation and deduction task is paused, the perception and scheduling module pauses the judgment logic in step (7). The parameters such as partitions, acquisition and processing processes, and sorting processes remain unchanged. When the simulation and deduction task starts, the judgment logic in step (7) is enabled.

[0122] The above is the sub-process of simulation scenario pause, as Figure 4 shown.

[0123] (14) When the simulation task stops, the perception and scheduling module stops the judgment condition in step (7). The following judgment condition is enabled: when all partition backlog states are empty and w≥s2, delete the topic, stop the acquisition and processing process, and stop the sorting process.

[0124] The above is the simulation scenario's stopping sub-process, such as... Figure 5 As shown.

[0125] This invention discloses a real-time data acquisition, processing, and storage system and method for simulation processes. By constructing four major modules—a data acquisition module, a data sorting module, a data storage and query analysis module, and a perception and scheduling module—it can fully consider factors closely related to the amount of simulation data, such as the number of entities within the simulation scenario, the number of parallel simulation scenarios, and the speedup ratio. It achieves on-demand, dynamically scalable data acquisition and processing capabilities, as well as efficient simulation process data query and retrieval capabilities. This solves the problem of data backlog caused by the inability of traditional system simulation data support systems to adapt to the real-time data processing needs of service-oriented and large-scale system simulations. It also avoids the lag in subsequent real-time performance evaluation calculations and real-time situation display, ensuring the real-time performance of situation display and performance evaluation calculations.

[0126] Furthermore, through technologies such as multi-process parallel processing, dynamic monitoring and scheduling, and big data, the system can automatically sense changes in the data output during the simulation process and dynamically adjust the acquisition and processing capabilities. This ensures both the real-time performance of acquisition and processing when the amount of output data increases during the simulation process and timely release of surplus operating resources when the amount of data decreases.

[0127] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A real-time acquisition, processing and storage system for simulation process data, characterized in that, The system comprises: a data collection module for buffering simulation process data continuously pushed outside the system in the form of messages and realizing parallel processing of the simulation process data through multiple collection and processing processes; the data collection module comprises a data buffering sub-module for buffering the messages in the form of topics and partitions; a data sorting module for real-time sorting of the simulation process data according to timestamps, and periodic reading, forwarding and deleting of the sorted data accumulated in a sorting heap; a data storage and query analysis module for storing the simulation process data and providing quick query and analysis functions for the stored simulation process data; a perception and scheduling module for detecting the working state of the simulation process data real-time collection and processing system, and overall coordinating the modules according to the detection results and the running state of the simulation scenario; the perception and scheduling module comprises: a state perception sub-module for periodically monitoring the state parameters of the data collection and processing module, the data sorting module and the data storage and query analysis module, and calculating an average data processing rate p according to a data collection and processing rate v to provide input parameters for the calculation and splitting of topics and partitions; a dynamic scheduling sub-module for reading a scenario file when a scenario in the simulation engine is about to be started, and calculating and splitting the topics and partitions of Kafka according to the total number of output ports in the scenario; the dynamic scheduling sub-module is further used for sending sorting process scheduling commands to the data sorting module and data storage table creation commands to the data storage and query analysis module; in the simulation deduction process, the working state of the simulation process data real-time collection and processing system is detected, and logical judgment is made according to the detection results and the running state of the simulation scenario, and overall coordination of the modules is made based on the logical judgment results, and the logical judgment specifically comprises: when d≥t, w≥s1, and there is an idle partition, the simulation engine is notified, and the simulation engine matches half the number of ports in the port set of the message backlog partition to a non-backlog partition; when d≥t, w≥s1, and there is no idle partition, a new partition is created, a collection and processing process corresponding to the partition is created, and the simulation engine is notified, and the simulation engine matches half the number of ports in the port set of the message backlog partition to a non-backlog partition; when d wherein d is the partition message backlog, t is the partition message backlog threshold, w is the continuous duration, s1 is the continuous duration when the message backlog is greater than the threshold t, and s2 is the continuous duration when the number of idle regions is greater than 1; after the simulation deduction task is started, the port outputs of all components in the simulation entities output simulation process data in the form of messages to the partitions, the port ID is the partition key, and the data is pushed to different partitions; in the initial state, the number of ports corresponding to each partition is the same.

2. The simulation process data real-time acquisition processing and storage system according to claim 1, characterized in that the data caching submodule provides a message push interface for the simulation deduction engine to provide simulation process data; the data acquisition module further comprises an acquisition processing submodule for parallel acquisition and processing of simulation process data, and for packing the processed simulation process data into groups according to continuous same time stamps, and for sending the group data and pointer position information of the last sorting insertion to the data sorting module; the data caching submodule is further configured to provide a message extraction interface for the acquisition processing submodule to extract cached messages. the data sorting module comprises:

3. The real-time acquisition processing and storage system for simulation process data according to claim 2, characterized in that, a sorting insertion submodule configured to find a sorting insertion position according to time stamps on the basis of previous pointer position information, to complete real-time sorting according to time stamps, and to return the pointer position information of the current sorting insertion to the acquisition processing submodule; a data sequential reading and deletion submodule configured to periodically detect whether the number of sorted data accumulated in the sorting heap reaches a preset threshold h, and to read the first h data and send them to the storage and query analysis module and to an external situation display software when the preset threshold h is reached. the data storage and query analysis module comprises:

4. The real-time acquisition processing and storage system for simulation process data according to claim 1, characterized in that, a data storage submodule constructed based on a Hive data warehouse tool and adopting a distributed data storage architecture; a data query analysis submodule constructed based on a Presto tool component and configured to quickly query and efficiently calculate and analyze Hive data.

5. The simulation process data real-time acquisition processing and storage system according to claim 2, characterized in that the dynamic scheduling submodule is further configured to monitor the message backlog of each partition during the running of the simulation engine, to send information about partitions with a message backlog greater than a threshold and idle partitions to the simulation engine when the message backlog is greater than the threshold for a duration of s1 and when there is more than one idle region for a duration of s2, and to send an adjustment command for the parallel acquisition strategy to the data acquisition processing module according to the agreed feedback result. the data caching submodule is further configured to provide a message backlog state reading interface for the dynamic scheduling submodule, a topic and partition changing interface; the acquisition processing submodule is further configured to provide a process scheduling interface for the dynamic scheduling submodule.

6. The real-time acquisition processing and storage system of simulation process data according to claim 5, characterized in that, including: completing initialization before the simulation deduction process; 7. A simulation process data real-time acquisition processing storage method based on the simulation process data real-time acquisition processing storage system according to any one of claims 1 to 6, characterized in that, after the simulation deduction task is started, using the data acquisition module to cache simulation process data continuously pushed from outside the system in the form of messages, and using multiple acquisition processing processes to realize parallel processing of simulation process data, using the data sorting module to sort simulation process data in real time according to time stamps, and periodically reading, forwarding and deleting sorted data accumulated in the sorting heap, using the data storage and query analysis module to store simulation process data and provide quick query and analysis functions for the stored simulation process data, using the sensing and scheduling module to detect the working state of the simulation process data real-time acquisition processing and storage system, and to coordinate the work of each module according to the detection result and the running state of the simulation scenario. ​ ​ 8. The method of claim 7, wherein, When the simulation reasoning task is paused, the perception and scheduling module suspends the logical judgment process and controls the partition, collection processing process, and ordering process parameters to remain unchanged. When the simulation reasoning task is started, the logical judgment process is enabled.

9. The method of claim 7, wherein, When the simulation reasoning task is stopped, the perception and scheduling module stops the logical judgment process and enables the following judgment conditions: When all partition backlog states are empty, and w > s2, delete the topic, stop the collection processing process, and stop the ordering process, where w is the duration, and s2 is the duration of the appearance of more than 1 idle area.

Images