Simulation data acquisition and storage system and method

Abstract

The embodiment of the invention discloses a simulation data acquisition and storage system and method. In a specific implementation mode, the system comprises a simulation test data acquisition unit which is used for acquiring simulation data of a simulation test engine according to an interface description file provided by a tester and is also used for writing the simulation data into a database table; and the processor is used for establishing a database table in the database according to the interface description file provided by the tester. Data are processed and written into a database, and efficient data collection and storage are completed. Meanwhile, through introduction of an interface description file, a data structure of simulation data needing to be collected and processed is determined in advance, and the automation degree of the process is improved.

Description

Technical Field

[0001] The present invention relates to the field of simulation technology, and more specifically, to a system and method for collecting and storing simulation data. Background Art

[0002] Large sample simulation often generates massive amounts of data. The huge amount of data not only increases the difficulty of storage, but also puts higher requirements on the speed and efficiency of data processing. Therefore, there are many related requirements for data recording in simulation. First, in large sample simulation, when multiple simulation systems are simulated in parallel, each system will generate data at the same time. Since these data come from different systems, their data structure contents are different. It is necessary to distinguish and process the received simulation data to facilitate subsequent parsing and storage. Secondly, although a large amount of data is generated in large sample simulation, only a small part of the large amount of data can be used by other systems and needs to be preserved permanently. Therefore, valuable data must be extracted from the large amount of data. Finally, the data types and structures in large sample simulation become very diverse with the changes in simulation experiments, which increases the complexity of system development. If the test data processing program is rewritten for all types of experiments, a large amount of redundant workload will occur. Summary of the invention

[0003] The object of the present invention is to provide a system and method for collecting and storing simulation data to solve at least one of the problems existing in the prior art.

[0004] In order to achieve the above object, the present invention adopts the following technical solutions:

[0005] A first aspect of the present invention provides a system for collecting and storing simulation data, comprising:

[0006] A simulation test data collector, used to collect simulation data of the simulation test engine according to the interface description file provided by the test personnel, and also used to write the simulation data into a database table in the database;

[0007] The processor is used to establish a database table in the database according to the interface description file provided by the experimenter.

[0008] Optionally, the simulation test data collector includes:

[0009] The first parsing unit is used to parse the interface description file provided by the test personnel to obtain the simulation test collection instruction;

[0010] A consumption unit, used for collecting simulation data of the simulation test engine according to the instruction;

[0011] A data writing unit is used to write the simulation data into the database.

[0012] Optionally, the write data unit includes

[0013] A parsing module, used for parsing the simulation data to obtain database data;

[0014] A database writing module is used to write the database data into the database.

[0015] Optionally, the database data includes

[0016] The attribute name, attribute type, name of the simulation experiment data type, experiment ID, sample ID, and name of the simulation model.

[0017] Optionally, the processor includes

[0018] A format conversion unit, used to describe the simulation test parameters in the interface description file provided by the test personnel using the CSV format to obtain a CSV file;

[0019] A second parsing unit is used to parse the CSV file to obtain the attribute name, attribute type, name of the simulation test data type, test ID, sample ID and name of the simulation model of the simulation test;

[0020] A data processing unit, used to generate a test data table name according to the name of the simulation test type, the test ID, the sample ID and the name data of the simulation model; and to generate corresponding fields in the test data table according to the attribute name and attribute type;

[0021] A database table generation module is used to generate SQL command statements for building a database table according to the test data table name and fields.

[0022] The database is used to establish a database table according to the SQL command statement.

[0023] A second aspect of the present invention provides a method for collecting and storing simulation data, comprising:

[0024] Collect simulation data from the simulation test engine according to the interface description file provided by the test personnel.

[0025] Establish a database table in the database according to the interface description file provided by the test personnel;

[0026] The simulation data is written into a database table in the database.

[0027] Optionally, the step of collecting simulation data of the simulation test engine according to the interface description file provided by the test personnel includes:

[0028] Parse the interface description file provided by the test personnel to obtain the simulation test acquisition instructions;

[0029] In response to the simulation test collection instruction, the simulation data of the simulation test engine is collected, and the simulation data is written into the database.

[0030] Optionally, writing the simulation data into the database includes:

[0031] The simulation data is parsed to obtain database data, and the database data is written into the database.

[0032] Optionally, the database data includes

[0033] The attribute name, attribute type, name of the simulation experiment data type, experiment ID, sample ID, and name of the simulation model.

[0034] Optionally, the step of establishing a database table in a database according to the interface description file provided by the experimenter includes:

[0035] Describe the simulation test parameters in the interface description file provided by the test personnel in CSV format to obtain a CSV file;

[0036] Parse the CSV file to obtain the attribute name, attribute type, name of the simulation test data type, test ID, sample ID and name of the simulation model of the simulation test;

[0037] Generate a test data table name according to the name of the simulation test type, the test ID, the sample ID and the name data of the simulation model; generate corresponding fields in the test data table according to the attribute name and attribute type;

[0038] Generate a SQL command statement for building a database table according to the test data table name and fields;

[0039] The database establishes a database table according to the SQL command statement.

[0040] The beneficial effects of the present invention are as follows:

[0041] In the process of large sample simulation, the present invention uses a consumer unit to orderly receive and manage the massive test data, process the data and write it into the database, and complete efficient data collection and storage. At the same time, through the introduction of the interface description file, the data structure of the simulation data to be collected and processed is determined in advance, which improves the automation of this process. Part of the code development and database establishment database table work can be automatically generated, which can reduce the complexity of system development and enable developers to focus on the function development of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The specific implementation modes of the present invention are further described in detail below in conjunction with the accompanying drawings.

[0043] Figure 1 A flow chart showing an automated collection and storage system for simulation data to which an embodiment of the present invention may be applied.

[0044] Figure 2 A schematic diagram showing an interface description file to which an embodiment of the present invention can be applied. DETAILED DESCRIPTION

[0045] In order to explain the present invention more clearly, the present invention is further described below in conjunction with embodiments and drawings. Similar components in the drawings are represented by the same reference numerals. It should be understood by those skilled in the art that the content specifically described below is illustrative rather than restrictive, and the protection scope of the present invention should not be limited thereto.

[0046] like Figure 1 As shown, a system for collecting and storing simulation data includes:

[0047] A simulation test data collector, used to collect simulation data of the simulation test engine according to the interface description file provided by the test personnel, and also used to write the simulation data into a database table in the database;

[0048] The processor is used to establish a database table in the database according to the interface description file provided by the experimenter.

[0049] Simulation test data collector, including

[0050] The first parsing unit is used to parse the interface description file provided by the test personnel to obtain the simulation test collection instruction;

[0051] A consumption unit, used for collecting simulation data of the simulation test engine according to the instruction;

[0052] A data writing unit is used to write the simulation data into the database.

[0053] The write data unit includes

[0054] A parsing module, used for parsing the simulation data to obtain database data;

[0055] A database writing module is used to write the database data into the database.

[0056] Database data includes

[0057] The attribute name, attribute type, name of the simulation test data type, test ID, sample ID, and name of the simulation model;

[0058] Processors include

[0059] A format conversion unit, used to describe the simulation test parameters in the interface description file provided by the test personnel using the CSV format to obtain a CSV (Comma-Separated Values) file;

[0060] A second parsing unit is used to parse the CSV file to obtain the attribute name, attribute type, name of the simulation test data type, test ID, sample ID and name of the simulation model of the simulation test;

[0061] A data processing unit, used to generate a test data table name according to the name of the simulation test type, the test ID, the sample ID and the name data of the simulation model; and to generate corresponding fields in the test data table according to the attribute name and attribute type;

[0062] A database table generation module is used to generate SQL (Structured Query Language, which is the abbreviation of Structured Query Language and is a standardized language command statement for managing and operating relational databases) for database table construction according to the test data table name and fields.

[0063] The database is used to establish a database table according to the SQL command statement.

[0064] Specifically, in the test preparation stage, the processor parses the data structure that the user needs to use in the simulation test and establishes the corresponding database table in the database. The simulation test data collector parses the data structure that the user needs to use in the simulation test and generates data collection and storage related codes in the database and compiles them into plug-ins; in the simulation operation stage, the simulation test data collector writes the data generated by the simulation test engine into the database table created in the test preparation stage in real time to complete the automatic data collection and storage of large sample simulations. The display system, evaluation system and other systems that need to use the test data can find the corresponding test data table according to the test scheme ID (Identity document, exclusive number) and sample ID they want to find, and perform corresponding functions such as real-time situation display and analysis and evaluation. In the simulation, by setting different topics for each type of simulation test, the data of different tests can be isolated to ensure that they do not interfere with each other, solving the problem of data isolation and classification.

[0065] In one embodiment, a method for collecting and storing simulation data includes: a test preparation phase and a simulation operation phase;

[0066] During the test preparation phase, a simulation test collection instruction for collecting simulation data of a simulation test engine is established according to the interface description file provided by the test personnel; a database table is established in a database according to the interface description file provided by the test personnel;

[0067] During the simulation operation phase, the simulation data is written into a database table in the database in real time according to the simulation test collection instruction.

[0068] In one embodiment, the simulation data of the simulation test engine is collected according to the interface description file provided by the test personnel, including

[0069] Parse the data interface description file provided by the test personnel to obtain the simulation test collection instructions for the simulation data that needs to be collected and processed in the simulation test;

[0070] The simulation data of the simulation test engine is collected according to the simulation test collection instruction, and the simulation data is written into the database.

[0071] Optionally, writing the simulation data into the database includes:

[0072] The simulation data is parsed to obtain database data, and the database data is written into the database. The database data includes the attribute name of the simulation test, the attribute type, the name of the simulation test data type, the test ID, the sample ID and the name of the simulation model.

[0073] In one embodiment, a database table is established in the database according to the interface description file provided by the experimenter, including:

[0074] Describe the simulation test parameters in the interface description file provided by the test personnel in CSV format to obtain a CSV file;

[0075] Parse the CSV file to obtain the attribute name, attribute type, name of the simulation test data type, test ID, sample ID and name of the simulation model of the simulation test;

[0076] Generate a test data table name according to the name of the simulation test type, the test ID, the sample ID and the name data of the simulation model; generate corresponding fields in the test data table according to the attribute name and attribute type;

[0077] Generate a SQL command statement for building a database table according to the test data table name and fields;

[0078] The database establishes a database table according to the SQL command statement.

[0079] The present invention uses a multi-process + multi-thread parallel method to process data and write it into a database to complete efficient data collection and storage. At the same time, through the introduction of the interface description file, the data structure of the simulation data to be collected and processed is determined in advance, which improves the automation of this process. Part of the code development and database establishment database table work can be automatically generated, which can reduce the complexity of system development and enable developers to focus their main energy on system function development.

[0080] In one embodiment, a method for collecting and storing large sample simulation data includes:

[0081] S1, database table generation;

[0082] S2, automatic generation and compilation of data logging plug-ins;

[0083] S3, data recording plug-in loading and automatic data collection.

[0084] In one embodiment, S1 includes

[0085] S11. The simulation test personnel provide the interface description file required for this test, describe the structured data in the interface description file using the CSV format, and compile the text-based communication protocol content into structured data described in a table format;

[0086] S12, parsing the interface description file, obtaining the test ID and sample ID of the simulation test and the name data of all simulation models;

[0087] S13. Generate SQL statements. Generate a globally unique test data table name based on the name of the test data type, test ID, sample ID and other data. Generate corresponding fields in the table based on the name and type of the attributes. Based on this information, assemble the SQL command statements for database table creation, send them to the database and execute them, and complete the automatic database table creation.

[0088] In one embodiment, S2 includes

[0089] S21. In the test preparation stage, the data interface description file provided by the test personnel is parsed to obtain the data structure of the test data to be collected and processed in this test, and the simulation test collection instruction is obtained;

[0090] S22. Generate Kafka data consumption code, data parsing code and database writing code according to the obtained simulation test collection instructions. Automatic code generation can avoid repeated development on the one hand and errors that may be caused by handwritten programs on the other hand, thereby improving the efficiency of software development.

[0091] In one embodiment, S3 includes

[0092] S31. In the test preparation stage, a simulation test data acquisition tool is generated;

[0093] S32. During the test operation phase, when the simulation data is sent to Kafka, the different simulation test data are diverted through the topic, and the simulation data is consumed from Kafka in real time;

[0094] S33. Using a multi-process + multi-threaded parallel approach, the simulation data obtained from Kafka is parsed according to the data structure in the interface description file, and the parsed field values ​​are spliced ​​into SQL command statements inserted into the database and then executed, and the data is written into the database to complete efficient data collection and storage.

[0095] The present invention can send and receive a large amount of data in parallel by using Kafka distributed clusters and partitions, and then process the data in a multi-process + multi-threaded parallel manner, write the cached data into the database, complete efficient data collection and storage, and solve the problem of large data volume. The present invention automatically generates data processing code through the interface description file provided by the large sample test. After Kafka receives the data sent by the test, it only extracts and processes the data recorded in the file, solving the problems of data quality and redundant development.

[0096] The data in the database table can be queried and used by other systems that need to use test data, such as display systems and evaluation systems, to achieve corresponding functions such as situation display, analysis and evaluation.

[0097] The present invention uses a multi-process + multi-thread parallel approach to process data and write it into a database, completing efficient data collection and storage. At the same time, by introducing an interface description file, the data structure of the simulation data to be collected and processed is determined in advance, which improves the automation of the process. Part of the code development and database table building work can be automatically generated, which can reduce the complexity of system development and enable developers to focus their main energy on system function development.

[0098] In one embodiment, in view of the data recording requirements of large sample digital simulation, taking the maritime combat test as an example, the specific implementation method includes the following three steps:

[0099] First, in the development phase, an interface description file (CSV format) is compiled based on the test plan, and the text-based communication protocol content is compiled into structured data described in a table format. At the same time, the test plan and the interface description file are associated in the database. Since the models and model parameters of each experiment are different, each experiment requires a corresponding interface file. The CSV file for the maritime combat test is as follows: Figure 2As shown. The first line in the interface description file is the header line, which includes all the complete data attributes and attribute types of the model, including ModelName (model name), ModelID (model ID), Latitude (longitude), Longitude (dimension), Height (height), direction (direction), Action (action), Time (simulation time), TestID (test ID), SampleID (sample ID). Starting from the second line, each line represents the data segment contained in the simulation model. If the simulation model contains the data, the value of the column is the data type of the attribute. If the simulation model does not contain the data, it is empty. For example, the Height of a warship moving on the sea surface always remains at 0, so there is no need to transmit the height parameters of the model in the test, so the value of the Height column corresponding to the ship model in the CSV file is empty, while the values ​​of longitude and latitude need to be updated in real time, and the corresponding values ​​of the column are double. Among them, if the model has output of action attributes in the simulation test, the action type corresponding to the action parameters of the model needs to be defined according to the specific test. For example, radar has event attributes. During the simulation preparation process, it is agreed in advance with other systems that the corresponding values ​​of no target, detected target, and target lost events are 0\1\2, and the corresponding int value is directly sent during the simulation run.

[0100] During the experiment preparation phase, it includes automatic table creation in the database and automatic generation of Kafka programs for consuming data, parsing data, and writing to the database.

[0101] The interface description file is parsed through an automatic table creation application. According to the name of the interface description file and the content of the structured data therein, after receiving the initialization data from the test, the corresponding table and field are automatically created in the database table, and a simulation test data collection plug-in is generated.

[0102] In order to distinguish the simulation model data of different tests and samples, the name definition of the database table includes the name of the simulation model, the test ID and the sample ID, that is, the name format of the table is ModelName_TestID_SampleID.

[0103] The first sample in the first test of the maritime combat test will automatically create a total of 7 database tables, namely warship_1_1, submarine_1_1, aircraft_1_1, radar_1_1, missile_1_1, torpedo_1_1 and buoy_1_1. The fields and field types of the tables and the data structure of the data parsing program are matched according to the attributes and attribute types in the corresponding rows of the model in the CSV file. At the same time, the interface description file is parsed through the data record plug-in generation method to automatically generate the code for consuming data from Kafka, parsing data, and writing to the database. Among them, most of the parameters in the Kafka data consumption code part, such as the consumer group name group.id, the client identifier client.id, and the service short end address bootstrap.servers.broker, are consistent or default configurations. The main difference is the different consumer subscription topics. The topic configured in the maritime combat test is SY_HSZZ. The data parsing code part first converts the received String type data into Json type data, then finds the corresponding row in the interface description file according to the model name, and parses it according to the attribute field in the row. For example, consume a Sting type message sent by a simulation system in Kafka, and then convert it to Json type;

[0104] {"ModelName":"buoy","ModelID":1,"TestID":1,"SampleID":1}, after obtaining the message ModelName is buoy, then obtain the values ​​of the ModelID, TestID, SampleID and other fields of the model according to the fields of the corresponding row of buoy (buoy) in the interface description file. The part of writing the database code is to splice the parsed values ​​of each field into an SQL statement to insert data, and send it to the database to insert data. For example, the message is spliced ​​into INSERT INTO BUOY_1_1(ModelName,ModelID,TestID,SampleID)VALUES(buoy,1,1,1). Finally, the automatically generated Kafka data consumption code, data parsing code, and database writing code are developed and compiled in a multi-process + multi-threaded parallel manner to generate a simulation test data collection method.

[0105] During the test run, the simulation data will be sent by the producer of the simulation test engine to a topic named SY_HSZZ, and then the Kafka consumer part in the simulation test data collection method will subscribe to receive the test data. The data parsing part will parse and process the received simulation data, and store it in the corresponding database table through the write database part, thereby realizing automatic data collection and storage for large sample simulations.

[0106] The display system, evaluation system and other systems that need to use the test data can find the corresponding test data table according to the test plan ID and sample ID they want to find, and perform corresponding functions such as real-time situation display and analysis and evaluation.

[0107] The present invention provides an automatic table building method, which realizes automatic table building in a database by defining an interface description file and a simulation test initialization file in advance, and does not need to repeat the table building operation in the database for data of different samples of different tests. A data record plug-in generation method is provided, which automatically generates a program for consuming data from Kafka, parsing data, and writing data into a database, thereby reducing repeated development and improving coding efficiency; a simulation test data collection method is provided, which uses Kafka middleware to decouple simulation test programs and display evaluation programs, thereby improving system development efficiency.

[0108] In the description of the present invention, it should be noted that the terms "upper", "lower", etc. indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific position, be constructed and operated in a specific position, and therefore cannot be understood as a limitation on the present invention. Unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be a connection between the two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances.

[0109] It should also be noted that, in the description of the present invention, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0110] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not limitations on the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is impossible to list all the embodiments here. All obvious changes or modifications derived from the technical solution of the present invention are still within the protection scope of the present invention.

Claims

1. A system for collecting and storing simulation data, characterized in that: include: A simulation test data collector, used to collect simulation data of the simulation test engine according to the interface description file provided by the test personnel, and also used to write the simulation data into a database table in the database; The processor is used to establish a database table in the database according to the interface description file provided by the experimenter.

2. The system according to claim 1, characterized in that The simulation test data collector comprises The first parsing unit is used to parse the interface description file provided by the test personnel to obtain the simulation test collection instruction; A consumption unit, used for collecting simulation data of the simulation test engine according to the instruction; A data writing unit is used to write the simulation data into the database.

3. The system according to claim 2, characterized in that The data writing unit includes A parsing module, used for parsing the simulation data to obtain database data; A database writing module is used to write the database data into the database.

4. The system according to claim 3, characterized in that The database data includes The attribute name, attribute type, name of the simulation test data type, test ID, sample ID, and name of the simulation model.

5. The system according to claim 4, characterized in that The processor includes A format conversion unit, used to describe the simulation test parameters in the interface description file provided by the test personnel using the CSV format to obtain a CSV file; A second parsing unit is used to parse the CSV file to obtain the attribute name, attribute type, name of the simulation test data type, test ID, sample ID and name of the simulation model of the simulation test; A data processing unit, used to generate a test data table name according to the name of the simulation test type, the test ID, the sample ID and the name data of the simulation model; and to generate corresponding fields in the test data table according to the attribute name and attribute type; A database table generation module is used to generate SQL command statements for building a database table according to the test data table name and fields. The database is used to establish a database table according to the SQL command statement.

6. A method for collecting and storing simulation data, characterized in that: include: Collect simulation data from the simulation test engine according to the interface description file provided by the test personnel. Establish a database table in the database according to the interface description file provided by the test personnel; The simulation data is written into a database table in the database.

7. The method according to claim 6, characterized in that The simulation data of the simulation test engine is collected according to the interface description file provided by the test personnel, including Parse the interface description file provided by the test personnel to obtain the simulation test acquisition instructions; According to the simulation test collection instruction, the simulation data of the simulation test engine is collected, and the simulation data is written into the database.

8. The method according to claim 7, characterized in that Writing the simulation data into the database includes The simulation data is parsed to obtain database data, and the database data is written into the database.

9. The system according to claim 8, characterized in that The database data includes The attribute name, attribute type, name of the simulation experiment data type, experiment ID, sample ID, and name of the simulation model.

10. The system according to claim 9, characterized in that The step of establishing a database table in a database according to the interface description file provided by the experimenter includes: Describe the simulation test parameters in the interface description file provided by the test personnel in CSV format to obtain a CSV file; Parse the CSV file to obtain the attribute name, attribute type, name of the simulation test data type, test ID, sample ID and name of the simulation model of the simulation test; Generate a test data table name according to the name of the simulation test type, the test ID, the sample ID and the name data of the simulation model; generate corresponding fields in the test data table according to the attribute name and attribute type; Generate a SQL command statement for building a database table according to the test data table name and fields; The database establishes a database table according to the SQL command statement.

Images