An emergency analysis and disposal method based on Netcdf data files

Abstract

The application discloses an emergency analysis and disposal method based on a Netcdf data file, and the method takes a Netcdf file as a standard input data source format, analyzes and processes dimensions, variables, attributes and data items in the Netcdf file, establishes items for describing data names in the Netcdf file as a standard data item reference source, establishes a data processing strategy around the reference source, introduces several calculation rules for each strategy, controls the processing mode of the calculation rules through the strategy, and finally outputs a strategy processing result. The application standardizes data sources, uses the description of data items as a standard reference source, and makes numerical analysis strategies independent. The strategy of the application supports multiple calculation rules, meets actual business requirements, introduces a flexible strategy-rule management method, and makes numerical analysis strategies linked with a business system in a passive mode, thereby reducing viscosity.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of security industry system monitoring services, and particularly relates to an emergency analysis and disposal method based on a Netcdf data file. BACKGROUND

[0002] In recent years, with the development of Internet of Things technology and the popularization of 5G technology, more and more sensors are applied in various projects. The types of sensors accessed in business systems and the stored sensor data are also increasing. Sensors are used to collect various environmental data such as temperature, humidity, atmospheric pressure, rainfall, PM2.5, and pressure. They provide detection basis and reference for business systems.

[0003] Business systems generally obtain environmental data through sensors or third-party institutions. Sensors are different types of physical devices that are directly connected to the system through serial ports or other means. Third-party institutions are usually data providers or sensor integration device providers, such as weather data providers or gas monitoring device manufacturers.

[0004] The various sensor data accessed by business systems and the monitoring data provided by third-party institutions generally follow their own data formats. Business systems first need to parse their respective protocols before they can obtain monitoring data, which is then used by the business system. Business systems use this type of environmental monitoring data as a monitoring object and set monitoring indicators based on actual business needs. When there is a deviation from the indicator value, the business system triggers an alarm.

[0005] This results in a business system having to adapt to many different protocols, and different protocols have different descriptions of the meaning of data, which directly causes complexity and instability in the use of data by business systems. When a device needs to be replaced or a data provider needs to be changed, the business system may need to adapt to a new data protocol.

[0006] The existing data analysis technology route, the data source often comes from different devices or systems, or some data files. The format of these data is usually defined by different manufacturers according to their own devices, tools, or systems. The protocol has a different format, and this format is often not an industry standard, and its universality and standardization are difficult to meet the requirements, with the characteristics of uncertainty and instability. Under the premise of non-standard data format definition, subsequent data parsing, data storage, and data analysis are also difficult to have certain standardization. Therefore, it is difficult to establish a general data analysis and processing method according to an uncertain data format. A standard data format is very important in data parsing, data exchange, data storage, and data analysis. This is also a disadvantage of the previous scheme.

[0007] The existing numerical calculation analysis is often for a parameter item to set a threshold, when exceeding or below the threshold, the subsequent business action is executed. The limitation of such analysis calculation is that the meaning of some parameter items may not be a specific numerical value, such as 'typhoon', which means 'yes' or 'no', and cannot be judged according to the size of the numerical value. Moreover, in the actual business environment, it is often necessary to calculate and analyze several different numerical values, and finally get the result. Therefore, such operation of directly setting analysis and calculation method for parameters is relatively rigid and not flexible. Because a certain analysis and calculation method may also be applied to other parameter items. SUMMARY

[0008] The present application aims to overcome the above-mentioned deficiencies, and provides an emergency analysis and disposal method based on Netcdf data file, which reduces the viscosity by linking the numerical analysis strategy with the business system in a passive way.

[0009] In order to achieve the above-mentioned purpose, the present application comprises the following steps:

[0010] Establishing a data item name storage data table;

[0011] Establishing a data storage data table;

[0012] Establishing a data strategy analysis and calculation table;

[0013] Establishing a data strategy calculation flow table;

[0014] Processing the dimensions, variables, attributes and data items in the Netcdf file, obtaining the data item name and data description in the Netcdf file, and storing them in the corresponding data item name storage data table and data storage data table;

[0015] Establishing a strategy data for the variables in the Netcdf file, and generating a data of strategy processing unified model JSON structure;

[0016] According to the variable name, obtaining the corresponding strategy processing flow and calculation rule in the data strategy analysis and calculation table from the strategy processing unified model JSON structure;

[0017] According to the strategy processing flow and calculation rule, performing rule calculation on the current variable value and the value corresponding to the attribute in the data strategy calculation flow table, and judging whether the rule is satisfied;

[0018] According to the logic setting of the strategy in the data strategy calculation flow table, judging whether all the calculation rules are satisfied with the setting, and outputting the result.

[0019] The data item name storage data table comprises a primary key GUID, a variable name, a variable description and a variable unit.

[0020] The data storage data table comprises a self-incrementing primary key, a variable name and a variable data value.

[0021] The data strategy analysis calculation table comprises a primary key GUID, a strategy description and a strategy effective mode.

[0022] The strategy effective mode comprises effective after all conditions are met and effective after one condition is met.

[0023] The strategy calculation flow table comprises a self-incrementing primary key, a strategy table primary key, a variable name, a calculation strategy of a value corresponding to an attribute and a numerical value.

[0024] The calculation strategy of the value corresponding to the attribute comprises greater than, less than, greater than or equal to, less than or equal to and range.

[0025] The description of the numerical value is saved according to built-in rules.

[0026] Compared with the prior art, the present application takes a Netcdf file as a standard input data source format, analyzes and processes dimensions, variables, attributes and data items in the Netcdf file, establishes an item described data name in the Netcdf file as a standard data item reference source, establishes a data processing strategy around the reference source, introduces several calculation rules for each strategy, controls the processing mode of the calculation rules through the strategy, and finally outputs a strategy processing result. The present application standardizes the data source, uses the description of the data item as a standard reference source, and makes the numerical analysis strategy independent. The strategy of the present application supports multiple calculation rules, meets actual business requirements, introduces a flexible strategy-rule management method, and the numerical analysis strategy is linked with the business system in a passive way, thereby reducing the viscosity.

[0027] Further, the present application provides a unique GUID code for each strategy, and a business system can bind a certain business link according to the code, and can perform data reverse search according to the GUID of the strategy to obtain which variables do not meet the strategy calculation. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 The present application is a flow chart. DETAILED DESCRIPTION

[0029] The present application will be further described below with reference to the accompanying drawings.

[0030] Referring to Figure 1 , the present application comprises the following steps:

[0031] Referring to Table 1, a data item name storage data table dt_fields is established, and the data item name storage data table includes a primary key GUID, a variable name, a variable description, and a variable unit. The field records the variable name in the NetCDF file. The variable name is the KEY of data acquisition.

[0032] Table 1

[0033] Number Name Description Data Type 1 id Primary Key GUID String 2 field Variable Name String 3 desc Variable Description String 4 unit Variable Unit String

[0034] Referring to Table 2, a data storage data table dt_datas is established, and the data storage data table includes a self-incrementing primary key, a variable name, and a variable data value. The field is the variable name in step one.

[0035] Table 2

[0036] Number Name Description Data Type 1 id Auto-increment Primary Key Long 2 field Variable Name String 3 value Variable Data Value Double

[0037] Referring to Table 3, a data strategy analysis calculation table dt_strategy is established, and the data strategy analysis calculation table includes a primary key GUID, a strategy description, and a strategy effective mode. The strategy effective mode includes a condition all satisfied effective mode and a condition satisfied effective mode.

[0038] Table 3

[0039]

[0040] Referring to Table 4, a data strategy calculation process table dt_strategy_process is established, and the data strategy calculation process table includes a self-incrementing primary key, a strategy table primary key, a variable name, a calculation strategy of a value corresponding to an attribute, and a numerical value. The calculation strategy of the value corresponding to the attribute includes greater than, less than, greater than or equal to, less than or equal to, and range. The description of the numerical value is saved according to the built-in rules.

[0041] Table 4

[0042]

[0043] The NetCDF file is processed using netcdfAll.jar, the findVariable function of which is used to process the dimensions, variables, attributes, and data items in the NetCDF file, to obtain the data item names and data descriptions in the NetCDF file, and to store them in the corresponding data item name storage data table and data storage data table;

[0044] Strategy data for the variables in the NetCDF file are established, and data in a strategy processing unified model JSON structure are generated, and a GUID value is exposed externally;

[0045] According to the variable name, the corresponding policy processing flow and calculation rule in the data policy analysis calculation table are obtained from the policy processing unified model JSON structure;

[0046] According to the policy processing flow and the calculation rule, the current variable value is calculated with the value corresponding to the attribute in the data policy calculation flow table, and it is judged whether the rule is satisfied or not;

[0047] According to the logical setting of the policy in the data policy calculation flow table, it is judged whether all the calculation rules satisfy the setting, and the result is output.

[0048] The policy processing unified model JSON structure is as follows:

[0049]

[0050] The application uses the Netcdf file format as a standard data source.

[0051] The application analyzes and processes the dimensions, variables, attributes and data items in the NetCDF file, and establishes the item describing the data name as a standard data item reference source.

[0052] The application establishes a data analysis strategy based on the reference source and introduces parameter calculation rules.

[0053] The parameter calculation rules of the application support numerical value and threshold value comparison, and support: greater than, less than, greater than or equal to, less than or equal to, range.

[0054] The parameter calculation rules of the application support logical operation of numerical value, and support: yes, no.

[0055] The data analysis strategy of the application can include multiple parameter calculation rules, and the multiple rules support two strategies of'simultaneously satisfying' or'satisfying one of them'.

[0056] The data analysis strategy of the application is output as a JSON format data, which is convenient for processing in different business links, for example: browser end.

[0057] 7、The data analysis strategy provides a GUID unique identifier to the outside, which is convenient for other businesses to be linked.

[0058] The following takes marine wind power weather data analysis as an example to illustrate the specific implementation steps.

[0059] Note:

[0060] The application does not limit which database to use, and table 1, table 2, table 3 and table 4 only give the definition of data item name and data type, and the user should determine the specific type according to the actual database used.

[0061] The patent does not limit the programming language used, and the embodiment uses the java development language for implementation and explanation.

[0062] The patent does not limit the JSON tool library used, and the use or non-use or the use of which tool is determined by the developer according to the actual situation.

[0063] The embodiment only explains the core steps involved in the patent, and programming problems encountered in actual development need to be handled by the user himself.

[0064] 1. Environment preparation

[0065] Database software Mysql8.0.27 or Mariadb10.5.13 or MSSQLserver2019, and the embodiment uses Mariadb10.5.13.

[0066] OracleJDK8 or OpenJDK8.

[0067] Netcdf parsing library netcdfAll-4.6.16.1.jar.

[0068] Any complete and correctly formatted Netcdf data file should contain data items and data.

[0069] JSON library fastjson1.2.7.jar.

[0070] 2. Create a data table

[0071] According to the following database script, create data tables (data table names are table_1, table_2, table_3 and table_4, respectively corresponding to Table 1, Table 2, Table 3 and Table 4 in the specific embodiment).

[0072] Create table_1 script

[0073] CREATE TABLE IF NOT EXISTS`table_1`(

[0074] `id`varchar(50)NOT NULL,

[0075] `field`varchar(50)DEFAULT NULL,

[0076] `unit` varchar(20) DEFAULT NULL,

[0077] PRIMARY KEY (`id`)

[0078] ) ENGINE = InnoDB DEFAULT CHARSET = utf8mb4; Create table_2 script

[0079] CREATE TABLE IF NOT EXISTS `table_2` (

[0080] `id` bigint(20) NOT NULL AUTO_INCREMENT,

[0081] `file` varchar(255) DEFAULT NULL,

[0082] `value` double DEFAULT NULL,

[0083] PRIMARY KEY (`id`)

[0084] ) ENGINE = InnoDB DEFAULT CHARSET = utf8mb4;

[0085] Create table_3 script

[0086] CREATE TABLE IF NOT EXISTS `table_3` (

[0087] `id` varchar(50) NOT NULL,

[0088] `desc` varchar(100) DEFAULT NULL,

[0089] `logic` text DEFAULT NULL,

[0090] PRIMARY KEY (`id`)

[0091] ) ENGINE = InnoDB DEFAULT CHARSET = utf8mb4;

[0092] Create table_4 script

[0093] CREATE TABLE IF NOT EXISTS `table_4` (

[0094] `id` bigint(20) NOT NULL AUTO_INCREMENT,

[0095] `strategy_id` varchar(50) CHARACTER SET utf8 DEFAULT NULL,

[0096] field` varchar(50) CHARACTER SET utf8 DEFAULT NULL,

[0097] `strategy` varchar(50) CHARACTER SET utf8 DEFAULT NULL,

[0098] `value` varchar(50) DEFAULT NULL,

[0099] PRIMARY KEY (`id`)

[0100] ) ENGINE = InnoDB DEFAULT CHARSET = utf8mb4;

[0101] 3. Parse the Netcdf data file and save the data items and the data corresponding to the data items

[0102] Use the ucar.nc2.NetcdfFile.open("netcdf file disk storage path") method provided by the netcdfAll-4.6.16.1.jar library to open the Netcdf data file.

[0103] Get all variable information through netcdfFile.getVariables(), and store it in List <variable>variables variable. Get the data item name by Variable.getName(); get the dimension information by Variable.getDimensions(); store the obtained data to table_1.

[0104] Get the corresponding data value by Variable.read(); store it to table_2.

[0105] By the above steps, suppose we get the information of data item name "temp" and its data value "25.0".

[0106] Save a data in data table table_1 as follows:

[0107] id field desc unit table_1_01 temp temp temp table_1_02 humd humd humd

[0108] Save a data in data table table_2 as follows:

[0109] id field value table_2_01 temp 25.2 table_2_02 humd 55

[0110] 4、Create a strategy

[0111] By reading the data item field content in table_1, create a drop-down menu in the user interface, fill its content into it, and let the user select. The strategy effect mode can also be selected by the user. Save the data after the user selects, and save a data in data table table_3 as follows:

[0112] id desc logic table_3_01 test AND

[0113] By reading the data item content in table_1 and the strategy content in table_3, help the user to select the corresponding relationship between the data item name and the strategy by the user interface, select the specific strategy calculation mode and threshold, and save a data in data table table_4 as follows:

[0114] id strategy_id field strategy value 1 table_3_01 temp gt 30 2 table_3_01 humd lt 60

[0115] 5、Calculate the data item analysis result according to the strategy

[0116] Query data from table_3, according to the primary key id value of a data as a query table_4 data table query condition where strategy_id = "table_3 primary key value", from table_3 to obtain and strategy matching calculation process data.

[0117] Convert the two types of data into JSON structure. The JSON structure data is as follows:

[0118]

[0119]

[0120] After querying data from the data table table_2, according to the above JSON structure, the corresponding data item strategy processing flow is obtained. The strategy processing flow is saved in the strategy_process node of the above JSON, which may be multiple processes. According to the definition of the strategy node, the data of the corresponding item in table_2 is operated with the value node value. If the strategy condition is met, return true, if not, return false.

[0121] When all the processes in strategy_process are calculated, according to the content of the logic node in the strategy node, the final decision is made on the result. If logic is AND, all processes are true to pass the strategy, otherwise not. If logic is OR, any one of all processes is true to pass, if all are false, it does not pass.

[0122] 6、System linkage

[0123] Through the above steps, the processing result of a certain strategy for a certain data can be obtained. Through the strategy primary key recorded in JSON, the corresponding record in the database can be obtained. In the third party emergency system, there are often emergency plan processes, so the primary key of the strategy table can be recorded in the emergency plan data. In this way, the strategy data can be obtained when the data is obtained. According to the above steps, whether the strategy process processing passes or not can be obtained. The emergency system can determine whether to start the emergency process according to the strategy result.< / variable>

Claims

1. A method for emergency analysis and disposal based on a Netcdf data file, characterized in that, The method comprises the following steps: establishing a data item name storage data table; establishing a data storage data table; establishing a data strategy analysis calculation table; establishing a data strategy calculation flow table; processing the dimensions, variables, attributes and data items in the Netcdf file, obtaining the data item name and data description in the Netcdf file, and storing them in the corresponding data item name storage data table and data storage data table; establishing strategy data for the variables in the Netcdf file, and generating strategy processing unified model JSON structure data; obtaining the corresponding strategy processing flow and calculation rule in the data strategy analysis calculation table from the strategy processing unified model JSON structure according to the variable name; performing rule calculation on the current variable value and the value corresponding to the attribute in the data strategy calculation flow table according to the strategy processing flow and calculation rule, and judging whether the rule is satisfied; judging whether all the calculation rules are satisfied according to the logical setting of the strategy in the data strategy calculation flow table, and outputting the result.

2. The emergency analysis and disposal method based on the Netcdf data file according to claim 1, characterized in that, The data item name storage data table comprises a primary key GUID, a variable name, a variable description and a variable unit.

3. The emergency analysis and disposal method based on the Netcdf data file according to claim 1, characterized in that, The data storage data table comprises a self-incrementing primary key, a variable name and a variable data value.

4. The emergency analysis and disposal method based on the Netcdf data file according to claim 1, characterized in that, The data strategy analysis calculation table comprises a primary key GUID, a strategy description and a strategy effective mode.

5. The emergency analysis and disposal method based on the Netcdf data file according to claim 4, characterized in that, The strategy effective mode comprises all conditions being satisfied and one condition being satisfied.

6. The emergency analysis and disposal method based on the Netcdf data file according to claim 1, characterized in that, The strategy calculation flow table comprises a self-incrementing primary key, a strategy table primary key, a variable name, a calculation strategy for the value corresponding to the attribute and a numerical value.

7. The emergency analysis and disposal method based on the Netcdf data file according to claim 6, characterized in that, The calculation strategy for the value corresponding to the attribute comprises greater than, less than, greater than or equal to, less than or equal to and range.

8. The emergency analysis and disposal method based on the Netcdf data file according to claim 6, characterized in that, The description of the numerical value is saved according to the built-in rule.

Images