Data processing system of intelligent fusion terminal data center

Abstract

The invention discloses a data processing system of an intelligent fusion terminal data center, which comprises a data center module, the data center module comprises a first container unit and a second container unit, the first container unit is used for operating a data center program, and the second container unit is used for independently operating a database server; the first container unit adopts Docker virtualization, and the second container unit adopts a TD engine data server; the first container unit and the second container unit operate in the same container network, and the first container unit is connected with the second container unit through the container network. The intelligent terminal data center is good in data performance and high in safety.

Description

Technical Field

[0001] The present invention relates to the field of data processing methods, and in particular to a data processing system for an intelligent fusion terminal data center. Background Art

[0002] At present, with the rapid development of intelligent distribution network, low-voltage area intelligent fusion terminal has realized the functions of area power supply collection, equipment status monitoring, edge computing, master station communication, etc., playing a core role. As the center of data operation inside the intelligent fusion terminal, the data center not only collects data sent by the collection APP, but also provides data to various advanced APPs and master station APPs, and stores all data at the same time. Frequent data interaction and huge data processing bring great tests to the performance of various aspects of terminal software and hardware.

[0003] The current low-voltage area intelligent terminal data center uses container technology. The data center APP runs in a single container and is in the same container as the database; the database uses the Sqkite lightweight relational database. When the data center needs to process a large amount of high-concurrency data, the database performance cannot meet the requirements. At the same time, there are security risks in putting the APP and data in the same container, and the Sqkite database itself has low security. Summary of the invention

[0004] In view of the above problems, the present invention proposes a data processing system for an intelligent fusion terminal data center, which solves the defects of low data performance and low security of the existing low-voltage area intelligent terminal data center.

[0005] The technical solution adopted by the present invention is as follows:

[0006] A data processing system for an intelligent fusion terminal data center includes: a data center module, the data center module includes a first container unit and a second container unit, the first container unit is used to run a data center program, the second container unit is used to run a database server alone, the first container unit uses Docker virtualization, and the second container unit uses a TDengine data server; the first container unit and the second container unit run in the same container network, and the first container unit is connected to the second container unit through the container network. The intelligent terminal data center of the present invention has good data performance and high security.

[0007] Optionally, the TDengine data server is deployed separately using a container image, and maps configuration and data files to local storage.

[0008] Optionally, the TDengine data server has multiple models.

[0009] Optionally, the model is used to create a super table.

[0010] Optionally, the super table is used to group data collection points of a specific type together to form a logically unified table.

[0011] Optionally, the super table includes sub-tables, each model generates a super table, and the data of each device creates a sub-table from the model.

[0012] Optionally, the sub-table is a data collection point for storing data.

[0013] Optionally, the data processing system of the intelligent fusion terminal data center further includes a third container unit, wherein the third container unit includes an exchange and acquisition module, and the exchange and acquisition module is used to collect data and transmit it to the data center module.

[0014] Optionally, the working method of the data center program includes the following steps:

[0015] 1) Connect to the database when the main function is initialized,

[0016] 2) The message queue telemetry transmission protocol agent registers, subscribes to the message queue telemetry transmission protocol topic that listens for receiving requests, and registers the callback function when receiving the message;

[0017] 3) Start the network event processing loop of the message queue telemetry transport protocol client;

[0018] 4) Clean up resources and exit.

[0019] Optionally, when a message is received, libmosquitto will call a pre-registered callback function, in which the application name and specific operation information are identified according to the subject of the received message, and then the relevant operation function is called to interact with the database. After the processing is completed, a response message is sent to a specific subject.

[0020] Optionally, the message queue telemetry transport protocol callback function completely hands over the message to the operation function for processing, and is responsible for freeing the string memory of the response message.

[0021] Beneficial Effects

[0022] 1. The intelligent terminal data center of the present invention has good data performance and high security. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is a system architecture diagram of a data processing system of an intelligent converged terminal data center according to Embodiment 1 of the present invention;

[0024] Figure 2This is a data center architecture diagram of a data processing system of an intelligent converged terminal data center according to Embodiment 1 of the present invention;

[0025] Figure 3 is a schematic diagram of a physical model of a data processing system of an intelligent converged terminal data center according to Embodiment 1 of the present invention;

[0026] Figure 4 This is a basic flow chart of the data center APP and application interaction of the data processing system of the intelligent converged terminal data center of embodiment 1 of the present invention;

[0027] Figure 5 It is a data center APP program flow chart of the data processing system of the intelligent fusion terminal data center of embodiment 1 of the present invention;

[0028] Figure 6 It is a model table content diagram of the ADC of the data processing system of the intelligent converged terminal data center in embodiment 1 of the present invention;

[0029] Figure 7 It is a model table equipment diagram of the ADC of the data processing system of the intelligent converged terminal data center in Example 1 of the present invention. DETAILED DESCRIPTION

[0030] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0031] The terms "first", "second", "third", etc. in the present invention are used for descriptive purposes only and should not be understood as indicating or implying relative importance. In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "connected" and "connected" should be understood in a broad sense, for example, they can be indirectly connected through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0032] The libmosquitto in the present invention is a library written in C language, which implements the main functions of the MQTT protocol, including message publishing, subscription, connection management, etc. As a client library of the Mosquitto message agent, libmosquitto enables developers to easily integrate the MQTT protocol in their own applications and realize communication with the Mosquitto message agent or other MQTT clients.

[0033] The MQTT in the present invention is called Message Queuing Telemetry Transport, which is a message queue telemetry transmission protocol. The MQTT protocol adopts a publish / subscribe model, which includes a publisher, a subscriber, and a message broker.

[0034] The TDengine in the present invention is a high-performance, distributed Internet of Things and industrial big data platform developed by Taosi Data. Its core module is a high-performance, cluster open source, cloud-native, minimalist time series database (TimeSeriesDatabase, TSDB).

[0035] In the present invention, cJSON is a lightweight C language JSON parsing library, which can parse JSON data into C language data structures (such as arrays and objects), and can also convert C language data into strings in JSON format to facilitate data transmission and storage.

[0036] JSON (JavaScript Object Notation) is a lightweight data exchange format based on a subset of ECMAScript (js specification developed by the European Computer Association) and uses a completely language-independent text format to store and represent data.

[0037] Example 1

[0038] The technical solution adopted by the present invention is as follows:

[0039] like Figure 1 , Figure 2 As shown, the present invention discloses a data processing system for an intelligent fusion terminal data center, including: a data center module, the data center module includes a first container unit and a second container unit, the first container unit is used to run a data center program, the second container unit is used to run a database server alone, the first container unit adopts Docker virtualization, and the second container unit adopts a TDengine data server; the first container unit and the second container unit run in the same container network, and the first container unit is connected to the second container unit through the container network. The data center APP exchanges data with other APPs through MQTT, and then exchanges data with the database server through the TDengine client.

[0040] The data processing system of the intelligent fusion terminal data center also includes a third container unit, and the third container unit includes an exchange module, and the exchange module is used to collect data and transmit it to the data center module.

[0041] The data processing system of the intelligent fusion terminal data center of the present invention further includes a fourth container unit, and the data center module interacts with the fourth container unit, and the fourth container unit is provided with 104APP.

[0042] The TDengine data server is deployed separately using a container image, and the configuration and data files are mapped to local storage. The TDengine data server has multiple models. The model is used to establish a super table. The super table is used to aggregate a certain type of data collection points to form a logically unified table. The super table includes sub-tables, each model generates a super table, and the data of each device creates a sub-table from the model. The sub-table is a data collection point for storing data.

[0043] It should be noted that the data center APP configures the container by writing a Dockerfile file, and packages it and the TDengine client driver (taosc) in the same container;

[0044] The TDengine database server container is deployed separately using its officially provided container image, mapping the configuration and data files to local storage.

[0045] It should be noted that the data center APP container and the TDengine database server container run in the same container network and are connected to the TDengine database server through the container network. The present invention runs two containers by writing a YML file and using the docker compose up-d command of the Compose tool.

[0046] like Figure 3 As shown, the data center of the present invention adopts the concept of physical model. Physical model is an abstraction of physical devices or virtual devices. Attributes, events, and services are the components of physical model. Attributes represent the characteristics of things, which are composed of name, attribute content, and time; events represent the situation of things, which are composed of name, event information, and time; services represent the capabilities of things, which are composed of name and related information. In layman's terms, model is an abstraction of physical devices or virtual devices, and devices are specific instances derived from models.

[0047] The data center adopts the concept of physical model. Physical model is an abstraction of physical devices or virtual devices. Attributes, events, and services are the components of physical model. Attributes represent the characteristics of things, which are composed of name, attribute content, and time; events represent the state of things, which are composed of name, event information, and time; services represent the capabilities of things, which are composed of name and related information. In layman's terms, model is an abstraction of physical devices or virtual devices, and devices are specific instances derived from models.

[0048] Use physical models to manage device data information. Each device will have a corresponding storage space in the data center, and the space storage format is defined based on the physical model.

[0049] like Figure 4 and Figure 5 As shown, the working method of the data center program includes the following steps:

[0050] 1) Connect to the database when the main function is initialized,

[0051] 2) The message queue telemetry transmission protocol agent registers, subscribes to the message queue telemetry transmission protocol topic that listens for receiving requests, and registers the callback function when receiving the message;

[0052] 3) Start the network event processing loop of the message queue telemetry transport protocol client;

[0053] 4) Clean up resources and exit.

[0054] When receiving a message, libmosquitto will call the pre-registered callback function, in which it will identify the application name and specific operation information based on the subject of the received message, and then call the relevant operation function to interact with the database. After the processing is completed, it will send a response message to the specific subject.

[0055] The message queue telemetry transport protocol callback function completely hands over the message to the operation function for processing, and is responsible for releasing the string memory of the response message.

[0056] In the present invention, a physical model is used to manage device data information. Each device has a corresponding storage space in the data center, and the space storage format is defined based on the physical model.

[0057] A hypertable is a data structure that can group data collection points of a specific type together to form a logically unified table.

[0058] A specific example of a super table is shown in Table 1 below:

[0059]

[0060] The specific sub-table examples are shown in Table 2 below:

[0061] Table name Corresponds to Each row of data model_<model_name> Each model A data field of the model parameter_<parameter_name> Each parameter A data field for a parameter <model_name>_ <guid> < / guid> Per device A data point uploaded by the device

[0062] in<model_name> is the specific model name,<parameter_name> is the parameter name, <guid>is the GUID of the device.

[0063] Take the cross-acquisition ADC model as an example:

[0064] The content of ADC model table is as follows Figure 6 Said:

[0065] The content of the ADC device table is as follows Figure 7 Said:

[0066] From the above example, it can be seen that the data content, data type, etc. stored in the ADC model are defined in the ADC model table model_ADC, and in a certain ADC device table ADC_5aeeb33b6dcaceff, the data is stored in chronological order according to the structure of the model table.

[0067] A subtable is a logically abstract representation of a data collection point. It is a specific table belonging to a supertable.

[0068] The structure of the super table and sub table is very consistent with the concept of the data center object model. The super table corresponds to the model in the object model; the sub table corresponds to the device in the object model. Each model generates a super table, and the data of each device creates a sub table from the model.

[0069] At the same time, the APP application writes data in a fixed and frequent cycle, which is highly correlated with the real time. The data is basically not modified after being written, which has typical time series data characteristics. TDengine, as a time series data, just meets this requirement. The present invention uses the TDengine database as the storage backend of the data center to complete the data read and write operations.

[0070] The data center APP in this invention is developed in C language, libmosquitto is selected as the MQTT client implementation, and cJSON is used to support JSON serialization / deserialization. Basic components such as variable-length strings, dynamic hash tables, and timestamps are implemented using glib 2.80. Static hash tables are generated using gperf before compilation.

[0071] In the present invention, gperf is commonly used to generate a keyword identifier for the lexical analyzers of multiple commercial compilers (such as GNU C, GNU C++, GNU Pascal, GNU Modula 3, etc.) and research compilers.

[0072] A model will create a super table to store data for the subsequent creation of a sub-table of the model's device; it will also create a model sub-table to store the model's own metadata information.

[0073] Each interface corresponds to an operation function. Most operation functions receive three parameters as input: a pointer to the MQTT message payload const char*msg, the length of the message payload const size_t msg_len, and the application name const char*app. The interface for application writing data has an additional parameter: the device name const char*dev_name. The operation function returns a string pointer to the message body that constructs the MQTT response.

[0074] The above description is only a preferred embodiment of the present invention, and does not limit the patent protection scope of the present invention. Any equivalent structural transformation made by using the contents of the present invention and the drawings, directly or indirectly used in other related technical fields, is also included in the protection scope of the present invention.< / guid>

Claims

1. A data processing system for an intelligent fusion terminal data center, characterized in that: include: A data center module, the data center module includes a first container unit and a second container unit, the first container unit is used to run the data center program, the second container unit is used to run a database server alone, the first container unit adopts Docker virtualization, and the second container unit adopts a TDengine data server; the first container unit and the second container unit run in the same container network, and the first container unit is connected to the second container unit through the container network.

2. The data processing system of the intelligent fusion terminal data center according to claim 1, characterized in that: The TDengine data server is deployed separately using a container image, and maps configuration and data files to local storage.

3. The data processing system of the intelligent fusion terminal data center according to claim 2, characterized in that: The TDengine data server has multiple models.

4. The data processing system of the intelligent fusion terminal data center according to claim 2, characterized in that: The model is used to build a super table.

5. The data processing system of the intelligent converged terminal data center according to claim 4, characterized in that: The super table is used to aggregate data collection points of a specific type to form a logically unified table.

6. The data processing system of the intelligent fusion terminal data center according to claim 5, characterized in that: The super table includes sub-tables. Each model generates a super table, and the data of each device creates a sub-table from the model. The sub-table is a data collection point for storing data.

7. A data processing system for an intelligent converged terminal data center as described in claim 2 or 3 or 4 or 5 or 6 or 7, characterized in that: It also includes a third container unit, which includes an exchange module. The exchange module is used to collect data and transmit it to the data center module.

8. A data processing system for an intelligent converged terminal data center as described in claim 2 or 3 or 4 or 5 or 6 or 7, characterized in that: The working method of the data center program includes the following steps: 1) Connect to the database when the main function is initialized, 2) The message queue telemetry transmission protocol agent registers, subscribes to the message queue telemetry transmission protocol topic that listens for receiving requests, and registers the callback function when receiving the message; 3) Start the network event processing loop of the message queue telemetry transport protocol client; 4) Clean up resources and exit.

9. The data processing system of the intelligent converged terminal data center according to claim 8, characterized in that: When receiving a message, libmosquitto will call the pre-registered callback function, in which it will identify the application name and specific operation information based on the subject of the received message, and then call the relevant operation function to interact with the database. After the processing is completed, it will send a response message to the specific subject.

10. The data processing system of the intelligent converged terminal data center according to claim 9, characterized in that: The message queue telemetry transport protocol callback function completely hands over the message to the operation function for processing, and is responsible for releasing the string memory of the response message.

Images