Sea optical cable disturbance monitoring protection method and device based on heterogeneous data

Abstract

The invention provides a sea optical cable disturbance monitoring protection method based on heterogeneous data. The sea optical cable disturbance monitoring protection method comprises the following steps: collecting real-time monitoring data by adopting a heterogeneous data source; processing the real-time monitoring data to obtain standard transmission data and sending the standard transmission data to the distributed stream processing platform; when the disturbance event occurs, acquiring disturbance event information; processing the disturbance event information to obtain standard event data and sending the standard event data to the distributed stream processing platform; based on a distributed stream processing platform, obtaining a time period of the disturbance event according to the standard event data; based on the time period, acquiring comprehensive information of the disturbance event from standard transmission data of the distributed stream processing platform; and analyzing the disturbance event according to the comprehensive information. According to the method provided by the invention, the disturbance event is comprehensively analyzed and judged in combination with data collected by multiple data sources, meanwhile, data loss or transmission delay in transmission can be avoided, the judgment result is accurate and high in reliability, and the method is high in modular processing capacity, stable in data transmission and low in cost and high in benefit.

Description

Technical Field

[0001] The present invention relates to the technical field of submarine optical cable monitoring, and in particular to a method and device for submarine optical cable disturbance monitoring and protection based on heterogeneous data. Background Art

[0002] Submarine optical cables are communication transmission cables laid on the seabed and are an important part of the international Internet and other underwater optical networks. Traditional submarine optical cable disturbance monitoring systems are mainly based on fiber optic sensing technology, especially phase-sensitive optical time domain reflectometry (Φ-OTDR) or distributed acoustic sensing (DAS) technology. These technologies use optical fiber as a data acquisition sensor to sense external disturbances by detecting changes in the phase or amplitude of the light transmitted in the optical fiber. When the submarine optical cable is subjected to external forces (such as ship anchoring, submarine geological activities, etc.), the optical fiber in the optical cable will undergo a slight deformation, causing the phase or amplitude of the light to change. The monitoring system can locate and identify disturbances by demodulating these change signals. However, the traditional submarine optical cable disturbance monitoring system has the following problems that are difficult to solve: 1. The accuracy and reliability of data acquisition sensor monitoring are not high. 2. The security of traditional data transmission disturbance data is low. 3. During the disturbance transmission process, the amount of a single disturbance data is tens of M, and it is easy to lose data or delay data during transmission. 4. The signal is single, and there is a problem of inaccurate judgment of the cause of abnormal submarine optical cable disturbance data. Summary of the invention

[0003] In view of the problems existing in the prior art, the present invention provides a submarine cable disturbance monitoring and protection method based on heterogeneous data with strong modular processing capability, stable data transmission, low cost and high efficiency, and accurate judgment. The disturbance event analysis combines data collected by multiple devices, including sensor data, video object recognition, AIS ship data and other information for comprehensive judgment, thereby avoiding inaccurate judgment due to data loss or transmission delay in transmission, and improving the accuracy and reliability of the judgment result.

[0004] To achieve the above object, the technical solution adopted by the present invention is as follows: a method for monitoring and protecting submarine optical cable disturbances based on heterogeneous data, comprising the following steps:

[0005] Using the heterogeneous data source to collect real-time monitoring data;

[0006] Processing the real-time monitoring data to obtain standard transmission data and sending it to a distributed stream processing platform;

[0007] When a disturbance event occurs, obtain disturbance event information;

[0008] Processing the disturbance event information to obtain standard event data and sending the standard event data to the distributed stream processing platform;

[0009] Based on the distributed stream processing platform, obtaining a time period of the disturbance event according to the standard event data;

[0010] Based on the time period, obtaining comprehensive information of disturbance events from the standard transmission data of the distributed stream processing platform;

[0011] The disturbance event is analyzed according to the comprehensive information.

[0012] In some of the embodiments, the device types of the heterogeneous data sources include sensors, video monitoring devices and AIS devices.

[0013] In some of the embodiments, the heterogeneous data sources include serial port devices and network devices.

[0014] In some of the embodiments, the heterogeneous data sources are scheduled by a collection thread pool.

[0015] In some of the embodiments, the real-time monitoring data is processed by a data processing unit to obtain standard transmission data and send it to a distributed stream processing platform, and the disturbance event information is processed to obtain standard event data and send it to the distributed stream processing platform;

[0016] The submarine optical cable disturbance monitoring and protection method based on heterogeneous data also includes timed synchronization of the heterogeneous data source, the data processing unit and the distributed stream processing platform.

[0017] In some of the embodiments, the standard transmission data and the standard event data are directly sent to the distributed stream processing platform, or the standard transmission data and the standard event data are sent to the distributed stream processing platform in the form of offline data.

[0018] In some of the embodiments, when a disturbance event occurs, the heterogeneous data source is used to obtain the disturbance event information.

[0019] In some of the embodiments, event waveform data is pre-stored in the heterogeneous data source used to obtain the disturbance event information, and the standard event data is obtained according to the disturbance event information and the event waveform data.

[0020] In some of the embodiments, the distributed stream processing platform is Kafka.

[0021] A device for implementing the submarine cable disturbance monitoring and protection method based on heterogeneous data, comprising a data acquisition unit, a data processing unit and a distributed stream processing platform;

[0022] The data acquisition unit includes the heterogeneous data source;

[0023] The data acquisition unit:

[0024] Used to collect the real-time monitoring data and send it to the data processing unit;

[0025] When a disturbance event occurs, the disturbance event information is acquired and sent to the data processing unit;

[0026] The data processing unit:

[0027] Used to process the real-time monitoring data to obtain standard transmission data and send it to the distributed stream processing platform;

[0028] Used to process the disturbance event information to obtain standard event data and send it to the distributed stream processing platform;

[0029] The distributed stream processing platform:

[0030] for obtaining a time period of the disturbance event according to the standard event data;

[0031] for obtaining comprehensive information of disturbance events from the standard transmission data of the distributed stream processing platform based on the time period;

[0032] Used to analyze the disturbance event according to the comprehensive information.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] 1. The present invention adds a distributed stream processing platform as a data transmission carrier, improves the data processing speed, and achieves a higher level of decoupling, thereby improving the modular processing capability and avoiding congestion or data loss caused by the simultaneous transmission of large amounts of data;

[0035] 2. The present invention combines the comprehensive judgment of heterogeneous data such as video object recognition and AIS ship data to analyze disturbance events, avoiding inaccurate judgment due to data loss or transmission delay during transmission, and improving the accuracy and reliability of the judgment results.

[0036] 3. Compared with the traditional optical fiber monitoring system, the present invention is more economical in hardware cost and maintenance fee, thereby providing higher cost-effectiveness and improving long-term return on investment (ROI). BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a flow chart of a method for monitoring and protecting submarine optical cable disturbances based on heterogeneous data according to the present invention;

[0038] Figure 2 It is a schematic diagram of the process of real-time monitoring data and disturbance event information from acquisition to sending to the distributed stream processing platform in an embodiment of the present invention;

[0039] Figure 3 A schematic diagram of a process for timing synchronization of heterogeneous data sources, a data processing unit, and a distributed stream processing platform in an embodiment of the present invention;

[0040] Figure 4 Schematic diagram of a process of analyzing disturbance events according to comprehensive information in an embodiment of the present invention;

[0041] Figure 5 It is a schematic diagram of the structure of a device for implementing a submarine optical cable disturbance monitoring and protection method based on heterogeneous data in an embodiment of the present invention. DETAILED DESCRIPTION

[0042] In order to clearly illustrate the technical features of this solution, the following will be combined with the accompanying drawings and examples to describe in detail the implementation methods of this application, so that the application can fully understand how to use technical means to solve technical problems and achieve the corresponding technical effects and implement them accordingly. The embodiments of this application and the various features in the embodiments can be combined with each other without conflict, and the technical solutions formed are all within the scope of protection of this application.

[0043] See also Figure 1 , a submarine cable disturbance monitoring and protection method based on heterogeneous data, comprising the following steps:

[0044] Heterogeneous data sources are used to collect real-time monitoring data; heterogeneous data sources refer to multiple data sources with different data structures, access methods or forms. The device types of heterogeneous data sources include sensors, video monitoring equipment and AIS equipment, which can realize real-time monitoring of the environment around submarine optical cables.

[0045] Sensors include distributed fiber optic vibration sensors, fiber grating sensors, inductive coupling sensors, infrared sensors, etc.;

[0046] Video monitoring equipment includes underwater cameras, distributed fiber-optic online monitoring systems, integrated underwater monitoring systems, towed linear array systems, and fiber-optic hydrophone arrays;

[0047] AIS equipment, or Automatic Identification System, is a new type of digital navigation aid system and equipment that integrates network technology, modern communication technology, computer technology, and electronic information display technology. AIS equipment consists of shore-based facilities and dock terminals, and broadcasts the ship's dynamic information, static information, and safety information to ships and shore stations in nearby waters through very high frequency (VHF) channels. This information includes the ship's status, ship name, Maritime Mobile Service Identity (MMSI), ship size, location, bow direction, etc. It can also receive relevant information that provides navigation aid services for safe navigation of ships;

[0048] Heterogeneous data sources include serial port devices and network devices, that is, real-time monitoring data can be read directly from heterogeneous data sources through physical connection or through network transmission;

[0049] The serial port device is connected to the data processing unit via the serial port, and the network device is connected to the data processing unit via the network port;

[0050] The data processing unit is connected to the serial device connected through the serial port, and obtains the real-time data collected by the serial device according to the acquisition protocol; the data processing unit is connected to the network device connected through the network port, and obtains the real-time data collected by the network device according to the acquisition protocol; the acquisition protocols include TCP, Modbus Tcp, etc. to ensure information security;

[0051] The collected real-time monitoring data is then integrated and formatted to meet the receiving standards of the distributed stream processing platform, that is, the real-time monitoring data is processed to obtain standard transmission data and sent to the distributed stream processing platform. The distributed stream processing platform is used to seamlessly integrate and manage real-time monitoring data from multiple sources, namely heterogeneous data sources, and can process a large amount of received data streams, namely standard transmission data, to support real-time data analysis and further decision-making, which not only ensures efficient data transmission, but also improves the scalability and reliability of the entire system, enabling it to cope with large-scale data sets from different devices and applications. Preferably, the distributed stream processing platform is kafka, and the format of the standard transmission data is json format;

[0052] When a disturbance event occurs, the disturbance event information is obtained; the disturbance event information can be obtained through sensors and / or video monitoring equipment in heterogeneous data sources, or a distributed optical fiber sensing system (such as BOTDA and DAS systems), an optical fiber hydrophone array, and an integrated monitoring platform and other disturbance event acquisition equipment for obtaining disturbance event information can be separately set up. The disturbance event acquisition equipment is connected to the data processing unit through a serial port or a network port. The data processing unit also obtains the disturbance event information collected by the serial port device or the network device according to the acquisition protocol to ensure information security;

[0053] Process the disturbance event information to obtain standard event data and send it to the distributed stream processing platform; use a heterogeneous data source for obtaining disturbance event information or pre-store event waveform data in the disturbance event acquisition device. The event waveform data is generally stored in the device's own FTP file path. Obtain standard event data based on the disturbance event information and event waveform data. After obtaining the disturbance event information, start downloading the file corresponding to the event waveform data from its own FTP. Combine the event waveform data and the disturbance event information to obtain complete event information. Format the complete event information into standard event data to meet the receiving standard of the distributed stream processing platform. The format of the standard event data is json.

[0054] Based on the distributed stream processing platform, the time period of the disturbance event is obtained according to the standard event data. The standard event data at least includes the start time and end time of the disturbance event, that is, the time period of the disturbance event.

[0055] Based on the time period, comprehensive information about disturbance events is obtained from the standard transmission data of the distributed stream processing platform; that is, in the time period corresponding to the start and end time of the disturbance event, the data collected by the sensor, such as ocean currents and anchor strikes, is retrieved or queried, and the result data identified by the video monitoring equipment, including large fish, small fish, divers and other entities, is retrieved or queried. At the same time, the real-time location information of the AIS-installed ships in this time period is retrieved or queried to check whether there are any ships passing by, so as to obtain comprehensive information about the disturbance event;

[0056] The disturbance event is analyzed based on the comprehensive information. By analyzing the comprehensive information and according to the time period when the disturbance event occurs, the data collected by the sensors, the video analysis results, and the ship AIS information within this time period are queried for comprehensive analysis to obtain the final analysis and judgment result of the disturbance event.

[0057] The present disclosure adds a distributed stream processing platform as a data transmission carrier, improves the data processing speed, and realizes a higher level of decoupling, thereby improving the modular processing capability, avoiding congestion or data loss caused by the simultaneous transmission of large amounts of data, and at the same time, combining the comprehensive judgment of heterogeneous data such as video object recognition and AIS vessel data for disturbance event analysis, avoiding inaccurate judgment due to data loss or transmission delay during transmission, and improving the accuracy and reliability of the judgment results.

[0058] The data processing unit processes the real-time monitoring data to obtain standard transmission data and sends it to the distributed stream processing platform, and processes the disturbance event information to obtain standard event data and sends it to the distributed stream processing platform; preferably, the data processing unit is a gateway program;

[0059] The gateway program has a time deviation due to the insufficient precision of the internal crystal oscillator, which gradually appears over time, so it needs to be corrected regularly. The submarine cable disturbance monitoring and protection method based on heterogeneous data also includes the time synchronization of heterogeneous data sources, data processing units and distributed stream processing platforms;

[0060] See also Figure 3 , preferably, the following steps are performed to synchronize heterogeneous data sources, data processing units and distributed stream processing platforms:

[0061] When the real-time monitoring data is collected, the gateway program starts the NTP service.

[0062] Request NTP time from the edge computing server every preset time:

[0063] If the request is successful, the time for synchronizing heterogeneous data sources, data processing units, and distributed stream processing platforms will be refreshed;

[0064] Otherwise, do not refresh.

[0065] Preferably, the preset time is 120s;

[0066] See also Figure 2 ,Heterogeneous data sources are scheduled by the collection thread pool, which improves overall performance, simplifies thread management, and optimizes resource utilization;

[0067] For real-time monitoring data: the collection thread pool starts monitoring the network port or serial port corresponding to the real-time monitoring data and captures data from it. The collection thread pool starts the send interface for all real-time monitoring data collection, sends query messages to the corresponding network port or serial port, monitors the serial port or network port of all corresponding devices, and starts the unpacking and encapsulation process once data arrives;

[0068] For disturbance event information: after connecting to the corresponding heterogeneous data source or disturbance event acquisition device, the event service of the acquisition thread pool will periodically send a send query command to the corresponding device to query whether the corresponding device currently has a disturbance event. Once a disturbance event is generated, the unpacking will be started and the FTP of the corresponding device will be connected to log in and download the event waveform data until the closed process is completed.

[0069] See also Figure 2 , 4 When the network is good, the standard transmission data and the standard event data are directly sent to the distributed stream processing platform, or when the network is poor, the standard transmission data and the standard event data are sent to the distributed stream processing platform in the form of offline data. Since the present disclosure analyzes the comprehensive information and queries the data collected by the sensors, the video analysis results, and the ship AIS information within the time period when the disturbance event occurs according to the time period, and obtains the final analysis and judgment result of the disturbance event, the standard transmission data and the standard event data can be sent to the distributed stream processing platform directly or in the form of offline data.

[0070] In some of the embodiments, a monitoring topic is constructed on a distributed stream processing platform; the monitoring topic includes sensor data, video recognition data, and AIS vessel data;

[0071] Based on heterogeneous data sources, data is transmitted according to the monitoring subject reception standards.

[0072] Building an appropriate number of monitoring topics can achieve classification and labeling of real-time monitoring data, help isolate data, achieve load balancing and parallel processing, and improve scalability and fault tolerance, making monitoring and maintenance easier;

[0073] Monitoring topics, as classification labels for messages, help to classify standard transmission data and standard event data by logic or function, and achieve effective organization and management of data. Different monitoring topics can store data of different types or from different sources to achieve data isolation, help avoid data confusion, and improve data readability and maintainability. Building an appropriate number of monitoring topics can avoid overloading a single monitoring topic, improve overall throughput and response speed, and multiple consumers can read data from different topics in parallel, thereby improving data processing efficiency.

[0074] See also Figure 5 , an embodiment of the present invention further provides a device for implementing a submarine optical cable disturbance monitoring and protection method based on heterogeneous data, including a data acquisition unit, a data processing unit and a distributed stream processing platform;

[0075] The data acquisition unit includes heterogeneous data sources; the device types of the heterogeneous data sources include sensors, video monitoring devices and AIS devices; the heterogeneous data sources include serial port devices and network devices;

[0076] Data acquisition unit:

[0077] Used to collect real-time monitoring data and send it to the data processing unit; used to obtain disturbance event information and send it to the data processing unit when a disturbance event occurs;

[0078] Real-time monitoring data can be read directly from heterogeneous data sources through physical connection or through network transmission, that is, connecting to the corresponding heterogeneous data source through device address and port number (serial device plus sub-device address), and obtaining data from heterogeneous data sources according to the acquisition protocol;

[0079] Data processing unit:

[0080] Used to process real-time monitoring data to obtain standard transmission data and send it to the distributed stream processing platform;

[0081] Used to process disturbance event information to obtain standard event data and send it to the distributed stream processing platform;

[0082] The data processing unit also includes a collection thread pool, a serial port service, a network service, an event service and a data formatter. The serial port service and the network service are connected to heterogeneous data sources through corresponding ports externally, and the serial port service and the network service are connected to the collection thread pool and the event service internally;

[0083] During the operation, the collection thread pool starts monitoring the serial port service and network service respectively and collects real-time monitoring data or disturbance event information from the corresponding serial port device or network device, i.e., heterogeneous data sources;

[0084] When the collected data is real-time monitoring data, the real-time monitoring data is transmitted to the data formatter for integration and formatting to meet the receiving standard of the distributed stream processing platform, that is, the real-time monitoring data is processed by the data formatter to obtain standard transmission data, and then sent to the distributed stream processing platform;

[0085] When the collected data is disturbance event information, the transaction service is started, and the pre-stored event waveform data is downloaded from the FTP file path of the corresponding heterogeneous data source device itself. The event waveform data and the disturbance event information are combined to obtain complete event information. The complete event information is transmitted to the data formatter for integration and formatting to obtain standard event data to meet the receiving standards of the distributed stream processing platform, and then sent to the distributed stream processing platform;

[0086] Distributed stream processing platform:

[0087] The time period used to obtain disturbance events based on standard event data;

[0088] Used to obtain comprehensive information about disturbance events from standard transmission data of the distributed stream processing platform based on time periods; query the data collected by sensors, video monitoring equipment, and AIS equipment within this time period: determine whether there are identifiable objects in the time period of the disturbance event based on the data collected by the video monitoring equipment, if so, perform video recognition and obtain video recognition results, or if not, the data collected by the video monitoring equipment can be ignored; determine whether there are ships passing through the time period of the disturbance event based on the data collected by the AIS equipment, if there are ships passing through, obtain the basic information and real-time positioning information (ship AIS information) of the passing ship, if not, it can be ignored;

[0089] It is used to analyze disturbance events based on comprehensive information; it obtains comprehensive information and conducts comprehensive analysis through data collected by sensors during the time period of the disturbance event, video recognition results, basic information of the ship and real-time positioning information, and draws the final analysis and judgment result of the disturbance event.

[0090] The data processing unit also includes log service. In addition to transmitting real-time monitoring data and complete event information to the data formatter, the serial port service and the network service also send signals to the log service. In addition to sending standard transmission data and standard event data to the distributed stream processing platform, the data formatter also sends signals to the log service to achieve information storage and help users discover potential security threats, performance bottlenecks and other problems, providing strong support for optimizing network performance and improving security.

[0091] The data processing unit also includes configuration, which includes configuration settings and hot updates. Users can perform parameter configuration and other operations on the data processing unit through configuration settings. The distributed stream processing platform can dynamically update or replace code, resources or configurations in the application through hot updates.

[0092] The distributed stream processing platform also includes an exception handling service, which processes and identifies data with anomalies or information anomalies. Furthermore, the exception handling service also sends signals to the log service to further help users discover potential security threats, performance bottlenecks and other issues.

[0093] In some of the embodiments, the data processing unit also includes a clock service. When the real-time monitoring data collection is started, the clock service starts the NTP service. The clock service requests the NTP time from the edge computing server once every preset time. If the request is successful, the time of the synchronized heterogeneous data source, data processing unit and distributed stream processing platform will be refreshed; otherwise, it will not be refreshed.

[0094] Finally, it should be noted that the above content is only used to illustrate the technical solution of the present invention, rather than to limit the scope of protection of the present invention. Simple modifications or equivalent substitutions of the technical solution of the present invention by ordinary technicians in this field do not deviate from the essence and scope of the technical solution of the present invention.

Claims

1. A method for monitoring and protecting submarine optical cable disturbances based on heterogeneous data, characterized in that: The following steps are involved: Use heterogeneous data sources to collect real-time monitoring data; Processing the real-time monitoring data to obtain standard transmission data and sending it to a distributed stream processing platform; When a disturbance event occurs, obtain disturbance event information; Processing the disturbance event information to obtain standard event data and sending the standard event data to the distributed stream processing platform; Based on the distributed stream processing platform, obtaining a time period of the disturbance event according to the standard event data; Based on the time period, obtaining comprehensive information of disturbance events from the standard transmission data of the distributed stream processing platform; The disturbance event is analyzed according to the comprehensive information.

2. The method for monitoring and protecting submarine optical cable disturbances based on heterogeneous data according to claim 1 is characterized in that: The device types of the heterogeneous data sources include sensors, video monitoring devices and AIS devices.

3. The method for monitoring and protecting submarine optical cable disturbances based on heterogeneous data according to claim 2 is characterized in that: The heterogeneous data sources include serial port devices and network devices.

4. The method for monitoring and protecting submarine optical cable disturbances based on heterogeneous data according to claim 3 is characterized in that: The heterogeneous data sources are scheduled by a collection thread pool.

5. The method for monitoring and protecting submarine optical cable disturbances based on heterogeneous data according to claim 1 is characterized in that: Processing the real-time monitoring data by a data processing unit to obtain standard transmission data and sending the data to a distributed stream processing platform, and processing the disturbance event information to obtain standard event data and sending the data to the distributed stream processing platform; The submarine optical cable disturbance monitoring and protection method based on heterogeneous data also includes timed synchronization of the heterogeneous data source, the data processing unit and the distributed stream processing platform.

6. The method for monitoring and protecting submarine optical cable disturbances based on heterogeneous data according to claim 1 is characterized in that: The standard transmission data and the standard event data are directly sent to the distributed stream processing platform, or the standard transmission data and the standard event data are sent to the distributed stream processing platform in the form of offline data.

7. The method for monitoring and protecting submarine optical cable disturbances based on heterogeneous data according to claim 1 is characterized in that: When a disturbance event occurs, the heterogeneous data source is used to obtain the disturbance event information.

8. The method for monitoring and protecting submarine optical cable disturbances based on heterogeneous data according to claim 7 is characterized in that: Event waveform data is pre-stored in the heterogeneous data source used to obtain the disturbance event information, and the standard event data is obtained according to the disturbance event information and the event waveform data.

9. The method for monitoring and protecting submarine optical cable disturbances based on heterogeneous data according to claim 1, characterized in that: The distributed stream processing platform is Kafka.

10. A device for implementing the submarine cable disturbance monitoring and protection method based on heterogeneous data according to any one of claims 1 to 9, characterized in that: It includes a data collection unit, a data processing unit and a distributed stream processing platform; The data acquisition unit includes the heterogeneous data source; The data acquisition unit: Used to collect the real-time monitoring data and send it to the data processing unit; When a disturbance event occurs, the disturbance event information is acquired and sent to the data processing unit; The data processing unit: Used to process the real-time monitoring data to obtain standard transmission data and send it to the distributed stream processing platform; Used to process the disturbance event information to obtain standard event data and send it to the distributed stream processing platform; The distributed stream processing platform: for obtaining a time period of the disturbance event according to the standard event data; for obtaining comprehensive information of disturbance events from the standard transmission data of the distributed stream processing platform based on the time period; Used to analyze the disturbance event according to the comprehensive information.

Images