Ocean terminal data distribution method

Abstract

The invention discloses an ocean terminal data distribution method, which comprises the following steps of: 1, defining four core types of ocean terminal data, namely terminal data, terminal equipment position data, terminal equipment observation monitoring data and terminal equipment alarm data, and determining storage positions and characteristics of various types of data; 2, deploying a data processing module, starting a timed task by the data processing module according to a preset time interval, and taking the unique identifier of the equipment as an index; the system has the beneficial effects that through the data processing module, multi-type data originally dispersed in a database, a file and a table are uniformly integrated and packaged by taking a unique identifier of equipment as a core, so that the problems of dispersed data storage and non-uniform formats are solved, the preprocessing complexity of data query and distribution is greatly reduced, and the data query and distribution efficiency is improved. Therefore, the business system can obtain the full-dimension information of the equipment through the single data packet, and the data use convenience is improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of marine terminal data processing and distribution, and particularly relates to a marine terminal data distribution method. BACKGROUND

[0002] With the vigorous development of the marine industry, the number of marine terminal devices such as buoys, fishing net bases and marine observation stations continues to grow, and the data generated by them is increasingly diverse, covering device terminal data, device location data, device collected observation data and device alarm data. These data are the core basis for decision-making, analysis and operation of marine-related business systems, and are crucial to the normal operation of business systems.

[0003] Currently, the data generated by marine terminal devices needs to be aggregated to a big data platform first, and then distributed to each business system by the platform. However, the existing data distribution method has significant defects and cannot meet the actual application requirements:

[0004] Single distribution means, poor adaptability: the existing method uses a unified distribution means to process different types of data, which cannot match the characteristics of various types of data. For example, location data and alarm data with high real-time requirements often have transmission delays due to the use of inappropriate distribution methods, which seriously affects the response efficiency of business systems to emergency situations.

[0005] Lack of business relevance, low data acquisition efficiency: different business systems have different requirements for data update frequency and receiving format, but the existing distribution method does not consider these individualized needs, resulting in business systems being unable to obtain the required data in a timely and accurate manner, which restricts the work efficiency of business systems.

[0006] Data storage is scattered, and the processing complexity is high: in the big data platform, various types of data of terminal devices, such as terminal information, location data and alarm data, are stored in different databases or files without being unified, and whether the platform actively distributes or the business system actively queries, both need to go through a complicated business processing flow, which has very low adaptability, further reducing the data distribution and use efficiency. SUMMARY

[0007] The purpose of the present application is to provide a marine terminal data distribution method to solve the problems of low distribution efficiency, poor adaptability and untimely data acquisition caused by single distribution means, scattered data storage and lack of business relevance in the existing marine terminal data distribution process, and to realize real-time, efficient and secure distribution of data.

[0008] To achieve the above purpose, the present application provides the following technical solution: a marine terminal data distribution method, comprising the following steps:

[0009] Step one: define four core types of marine terminal data, namely terminal data, terminal device location data, terminal device observation monitoring data and terminal device alarm data, and clarify the storage location and characteristics of each type of data;

[0010] Step two: deploy a data processing module, which starts a timing task according to a preset time interval, uses a device unique identifier as an index to obtain four types of data of the corresponding device from different storage locations, and packages the data into a data packet distinguished by the device unique identifier.

[0011] Step three: based on the characteristic differences of terminal data, terminal device location data, terminal device observation monitoring data and terminal device alarm data, the data in the data packet is distributed to each business system in a differentiated distribution manner.

[0012] As a preferred technical solution of the present application, in step one, the terminal data is the name, parameter configuration and running state information of the device, stored in the database; the terminal device location data is the real-time location information of the device, stored in the database or a special table; the terminal device observation monitoring data is the weather, environment and water quality data collected by the device, stored in a file; and the terminal device alarm data is the fault warning information of the device, stored in the database or a special table.

[0013] As a preferred technical solution of the present application, in step two, the working process of the data processing module includes:

[0014] Start the timing task, and the time interval of the timing task can be adjusted through system configuration;

[0015] Use the device unique identifier as an index to read marine terminal data and terminal device location data and alarm data from the database, and read terminal device observation monitoring data from the file;

[0016] Format the four types of data obtained to uniform and logically associate the data, and package the data into a data packet containing all types of data of the corresponding device.

[0017] As a preferred technical solution of the present application, in step two, the data packet is packaged in JSON format, containing the device unique identifier, terminal data, terminal device location data, terminal device observation monitoring data and terminal device alarm data fields.

[0018] As a preferred technical solution of the present application, in step three, the terminal data is pulled by an HTTP interface, specifically: the business system sends an HTTP request carrying the device unique identifier and the data range to the big data platform, the big data platform extracts the corresponding data from the data packet after receiving the request, and returns the data to the business system through an HTTP response.

[0019] As a preferred technical solution of the application, in the third step, the WebSocket push mode is used for the terminal device position data, specifically comprising: the business system initiates a WebSocket connection application to the big data platform through an HTTP request, carrying its own identifier and the required device list; the big data platform establishes a WebSocket connection after verification and maintains the connection stability through a heartbeat mechanism; the big data platform pushes the updated position data to the business system in real time through the WebSocket connection, and supports the business system to feed back control instructions through the connection.

[0020] As a preferred technical solution of the application, the specific implementation of the heartbeat mechanism is that the platform sends a heartbeat packet to the business system every 20-40 seconds, and the business system returns a response packet after receiving it, and if the platform does not receive the response packet for 2-4 times in a row, it will re-establish a WebSocket connection.

[0021] As a preferred technical solution of the application, in the third step, the TCP push mode is used for the terminal device observation and monitoring data, specifically: the big data platform and the business system pre-establish a TCP connection, and the data processing module generates an observation and monitoring data, and the platform immediately pushes the data to the business system through the TCP connection.

[0022] As a preferred technical solution of the application, in the third step, the Pulsar message queue push mode is used for the terminal device alarm data, specifically comprising: the big data platform creates a "terminal device alarm data" PulsarTopic; each business system subscribes to the PulsarTopic; the big data platform publishes the newly generated alarm data to the PulsarTopic, which is automatically pushed to all subscribed business systems by Pulsar.

[0023] As a preferred technical solution of the application, the big data platform is deployed with a MySQL database, an HDFS file system and a Pulsar message queue, which are used to store terminal data, position data, alarm data, terminal device observation and monitoring data, and realize queue distribution of alarm data.

[0024] Compared with the prior art, the application has the following advantages:

[0025] Through the data processing module, multiple types of data originally scattered in databases, files and tables are unified and packaged with the device unique identifier as the core, solving the problem of scattered data storage and non-uniform format, greatly reducing the pre-processing complexity of data query and distribution, enabling the business system to obtain full-dimensional information of the device through a single data packet, and improving the convenience of data use;

[0026] Based on the data characteristics, a differentiated distribution strategy is adopted to realize "on-demand matching": WebSocket and Pulsar message queue guarantee millisecond-level transmission of position data and alarm data, meeting the real-time requirement; the "small data block, high frequency" mode of TCP push ensures the non-delay transmission of massive observation and monitoring data; the HTTP interface pull flexibly adapts to the personalized update demand of the business system to the terminal data, and the overall distribution efficiency is improved compared with the existing single mode.

[0027] Full consideration is given to the receiving capacity and data demand difference of different business systems, which supports both active data pulling of the business system and active data pushing of the platform, and meets the control instruction feedback demand of the business system through bidirectional interaction function, adapts to diversified business scenarios, and provides precise data support for various business systems such as marine observation, fishery management and marine disaster warning. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 Fig. 1 is a diagram of the conventional marine terminal data distribution mode;

[0029] Figure 2 Fig. 4 is a workflow diagram of the data processing module of the present application;

[0030] Figure 3 Fig. 5 is a terminal data distribution flowchart of the present application;

[0031] Figure 4 Fig. 6 is a marine terminal data distribution method flowchart of the present application. DETAILED DESCRIPTION

[0032] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0033] Embodiment 1

[0034] Please refer to Figures 1-4 , which is the first embodiment of the present application, and the embodiment provides a marine terminal data distribution method, including the following steps:

[0035] Step 1: Define four core types of marine terminal data, namely terminal data, terminal device position data, terminal device observation and monitoring data, and terminal device alarm data, and clearly define the storage location and characteristics of each type of data; wherein:

[0036] Terminal data mainly contains basic information of these observation and monitoring devices, including the name of the device, parameter configuration, running state and other basic information, which is usually stored in a database. The characteristics of these data are that the change frequency is not high, and different business systems have different requirements for the update frequency of these data. The data update needs to be carried out according to the actual needs of the business system itself;

[0037] Terminal device location data is real-time location information of the device, which is stored in a database or a special table. The location of the marine terminal device in the sea is uncertain and will change with the flow of seawater and the direction of ocean current. Therefore, the business system needs to know the real-time location information of these terminal devices in real time. Such data is usually stored in a database or a corresponding table in the big data platform;

[0038] Terminal device observation and monitoring data is the meteorological, environmental and water quality data collected by the device, which is stored in a file. The terminal device in the sea will collect various marine observation and monitoring data. The characteristics of these data are large volume, continuous generation and high real-time requirement. The demand of the business system is large, which will also lead to more systems that receive such data. Since the amount of such data is large, the big data platform generally stores such data in a file, which needs to be read when needed.

[0039] Terminal device alarm data is the fault warning information of the device, which is stored in a database or a special table. The terminal device in the sea will generate various alarm data, such as low power alarm and device position offset alarm. These data have the characteristics of strong burst, high priority and need to be responded by multiple business systems simultaneously. After the generation of such data, the data needs to be pushed to the underlying business system in time so as to take corresponding measures in time.

[0040] The above data will be gathered to the big data platform through various ways, and stored in various forms, such as stored in a database, file storage, etc. The big data platform will provide various data interfaces for the underlying business systems to call, and read the corresponding type of data according to the demand. The traditional distribution method of marine terminal data is shown in Figure 1 It is obvious that the traditional scheme has the following disadvantages: the storage methods of various marine terminal data are different, which leads to the non-uniformity of data format, and the data cannot be regularized and processed according to the characteristics of marine terminal data. Moreover, although the data distribution method of the big data platform is diversified, it cannot be combined and adapted with the marine terminal data, and cannot achieve convenient, efficient and fast experience.

[0041] Step two: deploy the data processing module, the data processing module starts a timing task according to the preset time interval, and obtains four types of data of the corresponding device from different storage locations according to the device unique identifier as the index, and packages them as data packets distinguished by the device unique identifier; such as reading the terminal basic information in the database, reading the terminal location data in the corresponding file, reading the device alarm data in the corresponding table, and uniformly processing these data to form multiple data packets, each data packet contains all information of the terminal device, including terminal information, location information, alarm information, and collection data information, and each data packet is distinguished by the unique identifier of the device, so that all device data in a certain time period is uniformly processed, facilitating subsequent data distribution, and the flow chart is shown in Figure 2 .

[0042] Step three: based on the characteristics of terminal data, terminal device location data, terminal device monitoring data and terminal device alarm data, the data in the data packet is distributed to each business system in a differentiated distribution manner; the final purpose of terminal data distribution is to distribute all terminal device data from the big data platform to each business system, not only to distribute all data, but also to ensure convenience, safety and convenience, and the data processing module has processed all data of the marine terminal device, which facilitates various forms of query, so the most core function of this module is to distribute data according to the characteristics of different data and adopt different data distribution methods; the flow chart of terminal data distribution is shown in Figure 3 , and the specific steps are as follows:

[0043] For terminal data, the update frequency of this type of data is not high, and different business systems have different requirements for its update, so each business system needs to update and pull data according to its actual needs, and HTTP interface pulling method can be used, specifically: the business system sends an HTTP request carrying the device unique identifier and data range to the big data platform, the big data platform extracts the corresponding data from the data packet after receiving the request, and returns it to the business system through HTTP response.

[0044] For terminal device location data, due to its small data volume but relatively high real-time requirement, WebSocket push mode can be used, which specifically includes: the business system initiates a WebSocket connection application to the big data platform through an HTTP request, carrying its own identifier and the required device list; the big data platform establishes a WebSocket connection after verification, and maintains the stability of the connection through a heartbeat mechanism; the big data platform pushes the updated location data to the business system in real time through the WebSocket connection, and supports the business system feeding back control instructions through the connection; the specific implementation of the heartbeat mechanism is that the platform sends a heartbeat packet to the business system every 20 seconds, and the business system returns a response packet after receiving it, if the platform does not receive a response packet for 2 consecutive times, it will re-establish a WebSocket connection; WebSocket push can establish a persistent connection between the big data platform and the business system due to the full-duplex communication feature, not only can guarantee the instantaneous transmission of data after its generation, reduce the millisecond-level delay, but also support the business system feeding back control instructions to the platform through the same connection, realize two-way efficient interaction, and perfectly adapt to the real-time and interactive requirements of such data.

[0045] For terminal device observation and monitoring data, due to its large volume and continuous generation, TCP push mode can be used, which specifically includes: the big data platform and the business system pre-establish a TCP connection, and the data processing module generates a piece of observation and monitoring data, the platform immediately pushes the data to the business system through the TCP connection; in the form of TCP push, this way not only can guarantee that a piece of data on the device can be pushed to the business system, the pushing frequency is high, but the volume of each data is small, so that the massive observation and monitoring data can be pushed in batches.

[0046] For terminal device alarm data, such data has strong burstiness and high real-time requirement; Pulsar message queue push mode can be used, which specifically includes: the big data platform creates a "terminal device alarm data" PulsarTopic; each business system subscribes to the PulsarTopic; the big data platform publishes the newly generated alarm data to the PulsarTopic, which is automatically pushed to all subscribed business systems by Pulsar.

[0047] It should be noted that the system of the present application is deployed as follows:

[0048] Hardware deployment: the big data platform is deployed on a server with 4-way CPU, 128GB memory, and 10TB storage to ensure data storage and processing capacity; the business system is deployed on a terminal server according to the requirements, and needs to support HTTP, WebSocket, TCP protocol and Pulsar client.

[0049] Software configuration: the big data platform installs MySQL database (stores terminal data, position data, alarm data), HDFS file system (stores observation monitoring data), Pulsar message queue; the data processing module is developed by using Java language, integrates timing task scheduling framework (Quartz), and sets that a data integration task is started once every 5 minutes; the distribution module respectively realizes HTTP interface (based on SpringBoot framework), WebSocket service (based on Netty framework), TCP service (based on JavaNIO) and Pulsar message publishing function.

[0050] Embodiment 2

[0051] For the second embodiment of the application, the embodiment is based on the previous embodiment, except that: Figures 1-4

[0052] The specific implementation of the heartbeat mechanism is that the platform sends a heartbeat packet to the business system every 30 seconds, the business system returns a response packet after receiving, and if the platform does not receive the response packet for 3 times continuously, the WebSocket connection is re-established.

[0053] Embodiment 3

[0054] For the third embodiment of the application, the embodiment is based on the previous embodiment, except that: Figures 1-4

[0055] The specific implementation of the heartbeat mechanism is that the platform sends a heartbeat packet to the business system every 40 seconds, the business system returns a response packet after receiving, and if the platform does not receive the response packet for 4 times continuously, the WebSocket connection is re-established.

[0056] Although the embodiments of the application have been shown and described in detail, according to the above detailed description, those skilled in the art can understand that various changes, modifications, replacements and variations can be made to these embodiments without departing from the principles and spirits of the application, and the scope of the application is defined by the appended claims and their equivalents.​​

Claims

1. A method for distributing marine terminal data, characterized in that: Includes the following steps: Step 1: Define the four core types of marine terminal data, namely terminal data, terminal equipment location data, terminal equipment observation and monitoring data, and terminal equipment alarm data, and clarify the storage location and characteristics of each type of data; Step 2: Deploy the data processing module. The data processing module starts a timed task according to a preset time interval, uses the device's unique identifier as an index, retrieves four types of data corresponding to the device from different storage locations, and organizes and encapsulates them into data packets with the device's unique identifier as the distinguishing identifier. Step 3: Based on the differences in characteristics of terminal data, terminal device location data, terminal device monitoring data, and terminal device alarm data, a differentiated distribution method is used to distribute the data in the data packet to each business system.

2. The marine terminal data distribution method according to claim 1, characterized in that: In step one, the terminal data includes the device name, parameter configuration, and operating status information, which are stored in a database; the terminal device location data includes the device's real-time location information, which are stored in a database or a dedicated table; and the terminal device monitoring data includes meteorological, environmental, and water quality data collected by the device, which are stored in a file. The alarm data from the terminal device is a fault warning information of the device, which is stored in a database or a dedicated table.

3. The marine terminal data distribution method according to claim 1, characterized in that: In step two, the workflow of the data processing module includes: Start a scheduled task; the time interval of this scheduled task can be adjusted through system configuration. Using the device's unique identifier as an index, marine terminal data, terminal device location data, and alarm data are read from the database, and terminal device monitoring data are read from the file. The four types of data are processed for format unification and logical association, and then encapsulated into a data packet containing all types of data for the corresponding devices.

4. The marine terminal data distribution method according to claim 1, characterized in that: In step three, the terminal data is retrieved using an HTTP interface. Specifically, the business system sends an HTTP request carrying the device's unique identifier and data range to the big data platform. After receiving the request, the big data platform extracts the corresponding data from the data packet and returns it to the business system via an HTTP response.

5. A marine terminal data distribution method according to claim 1, characterized in that: In step three, the location data of the terminal device is pushed using WebSocket. Specifically, the business system initiates a WebSocket connection request to the big data platform via an HTTP request, carrying its own identifier and a list of required devices. After the big data platform verifies the request, it establishes a WebSocket connection and maintains the connection stability through a heartbeat mechanism. The big data platform pushes the updated location data to the business system in real time through the WebSocket connection and supports the business system in feeding back control commands through this connection.

6. A marine terminal data distribution method according to claim 5, characterized in that: The heartbeat mechanism is implemented as follows: the platform sends a heartbeat packet to the business system every 20-40 seconds. After receiving the packet, the business system returns a response packet. If the platform does not receive a response packet for 2-4 consecutive times, it re-establishes the WebSocket connection.

7. A marine terminal data distribution method according to claim 1, characterized in that: In step three, the monitoring data from the terminal device is pushed via TCP. Specifically, the big data platform and the business system establish a TCP connection in advance. Every time the data processing module generates a piece of monitoring data, the platform immediately pushes the data to the business system through the TCP connection.

8. A marine terminal data distribution method according to claim 1, characterized in that: In step three, the alarm data from terminal devices is pushed using the Pulsar message queue method. Specifically, this includes: the big data platform creating a "Terminal Device Alarm Data" PulsarTopic; each business system subscribing to the PulsarTopic; the big data platform publishing the newly generated alarm data to the PulsarTopic, which is then automatically pushed to all subscribed business systems by Pulsar.

Images