Ship remote monitoring method and system, product and medium

Abstract

The invention discloses a ship remote monitoring method and system, a product and a medium, and relates to the field of ship remote monitoring, and the method comprises the steps: receiving real-time sensor data of a target ship, comparing the real-time sensor data with historical sensor data, and obtaining a target fault sensor and a data change time period; acquiring video monitoring data of the target fault sensor area in the data change time period, and comparing the video monitoring data with historical monitoring data to obtain a ship change area; analyzing a time sequence relationship between the sensor change trend of the target fault sensor and the monitoring change trend of the video monitoring data; in a preset fault mode library, judging whether a fault mode matched with the time sequence relationship, the sensor change trend and the monitoring change trend exists or not; and if yes, an early warning instruction containing the fault mode is sent to the target ship client. By implementing the method, the ship fault monitoring accuracy can be improved.

Description

Technical Field

[0001] The present application relates to the field of remote monitoring of ships, and in particular to a method, system, product and medium for remote monitoring of ships. Background Art

[0002] With the rapid development of the shipping industry, the safe operation of ships has become the focus of the industry. In order to ensure navigation safety and improve ship management efficiency, ship remote monitoring technology has emerged.

[0003] In related technologies, sensor data can be analyzed to collect parameters such as temperature, pressure, vibration, etc. of various parts of the ship and compare them with the preset normal value range. When a parameter exceeds the normal range, an alarm is issued to remind relevant personnel to pay attention to the fault.

[0004] However, relying solely on sensor data for condition monitoring has certain limitations. Sensors may produce errors due to environmental factors or their own faults, affecting the reliability of the data and resulting in low accuracy in ship fault monitoring. Summary of the invention

[0005] The present application provides a ship remote monitoring method, system, product and medium for improving the accuracy of ship fault monitoring.

[0006] In the first aspect, the present application provides a ship remote monitoring method, which is applied to a ship remote monitoring system, the method comprising: receiving real-time sensor data of a target ship, comparing the real-time sensor data with historical sensor data, and obtaining a target fault sensor and a data change time period; obtaining video surveillance data of an area of ​​the target fault sensor within the data change time period, and comparing the video surveillance data with historical surveillance data to obtain a ship change area; analyzing a temporal relationship between a sensor change trend of the target fault sensor and a monitoring change trend of the video surveillance data; determining in a preset fault mode library whether there is a fault mode that matches the temporal relationship, the sensor change trend, and the monitoring change trend; if so, sending an early warning instruction containing the fault mode to a target ship client.

[0007] By adopting the above technical solution, the target ship's real-time and historical sensor data are received and compared to obtain the target fault sensor and time period, and then the corresponding video surveillance data is obtained for comparison to obtain the ship's change area. Then, the trend time series relationship between the two is analyzed and the matching fault mode is determined in the preset library. Finally, according to the result, an early warning instruction or related information is sent. The sensor data comparison can quickly locate the sensor that may be faulty, and the video surveillance data further determines the ship's change area. The time series relationship analysis and fault mode matching improve the accuracy of fault judgment and reduce the error rate of ship fault monitoring.

[0008] In combination with some embodiments of the first aspect, in some embodiments, the step of comparing the video surveillance data with the historical surveillance data to obtain the ship change area specifically includes: comparing the video surveillance data with the historical surveillance data to obtain a difference image; performing image segmentation on the difference image to identify potential change areas; screening the potential change areas according to a preset feature threshold to obtain the ship change area.

[0009] By adopting the above technical solution, the video surveillance data is compared with the historical data to obtain a difference image, the potential change area is segmented and identified, and the ship change area is screened. The image difference is obtained by comparison, the area that may change is segmented and identified, and then the irrelevant area is screened and excluded. This can highlight the change information. Image segmentation can refine the area, and screening can accurately determine the ship change area, which is helpful to timely discover local abnormalities of the ship and improve the accuracy and efficiency of ship fault monitoring.

[0010] In combination with some embodiments of the first aspect, in some embodiments, after the step of screening the potential change area according to a preset feature threshold to obtain the ship change area, the method also includes: matching the ship change area with the ship structure information to determine the position data, area data and shape characteristics of the ship change area in the ship; prioritizing the ship change area based on the position data, the area data and the shape characteristics to obtain a change area importance index; giving a graded alarm to the ship change area according to the change area importance index, and adjusting the video surveillance frequency of the ship change area.

[0011] By adopting the above technical solution, the changed area of ​​the ship is matched with the structural information to determine the location, area and shape characteristics, and then the importance index is obtained based on this sorting. Finally, the alarm is graded according to the index and the monitoring frequency is adjusted. Matching the structural information can clarify key information such as the location of the area in the ship. The index obtained by sorting can quantify the importance of the area. Grading the alarm according to the index allows operators to quickly distinguish the degree of urgency, and adjusting the monitoring frequency can reasonably allocate monitoring resources.

[0012] In combination with some embodiments of the first aspect, in some embodiments, the step of analyzing the temporal relationship between the sensor change trend of the target fault sensor and the monitoring change trend of the video surveillance data specifically includes: inputting the sensor change trend and the monitoring change trend into a temporal relationship model to obtain a quantitative result between the sensor change trend and the monitoring change trend, the quantitative result including sequence, time delay and causal relationship strength; and determining the temporal relationship between the sensor change trend and the monitoring change trend based on the quantitative result.

[0013] By adopting the above technical solution, the sensor and monitoring change trends are input into the timing relationship model to obtain quantitative results to determine the timing relationship. Sensor and monitoring change trends as input reflect the state changes of different aspects of the ship. The timing relationship model outputs quantitative results by analyzing and processing these trends. The information such as sequence, time delay and causal relationship strength in the quantitative results accurately presents the relationship between the two, which helps to accurately judge the development process of ship failures and improve the accuracy and scientificity of ship failure analysis. In combination with some embodiments of the first aspect, in some embodiments, before the step of inputting the sensor change trend and the monitoring change trend into the time series relationship model to obtain a quantitative result between the sensor change trend and the monitoring change trend, wherein the quantitative result includes a sequence of precedence, a time delay, and a causal relationship strength, the method further includes: performing a time series analysis on the sensor change trend to obtain key feature points, wherein the key feature points include data mutation points, trend change points, and periodic change points; performing a time series analysis on the monitoring change trend to obtain key event points including area appearance, area expansion, area reduction, and area disappearance; calculating the time interval and occurrence order between the key feature point and the key event point, and extracting a sensor-video event pair, wherein the time interval of the sensor-video event pair is less than a preset time threshold and conforms to a preset occurrence order pattern; performing a cluster analysis on the sensor-video event to obtain a time series pattern event group, and calculating the time series similarity between different time series pattern event groups; and constructing a time series relationship model between the sensor change trend and the monitoring change trend based on the time series similarity and the time series pattern event group.

[0014] By adopting the above technical solutions, time series analysis is performed on the sensor and monitoring change trends, key feature points and event points are extracted, and the relationship is calculated and clustered to build a time series relationship model. Time series analysis can mine key information in the trend, and the extracted points and calculated relationships provide a basis for cluster analysis. The model constructed by cluster analysis integrates this information, making the relationship between sensor and monitoring change trends clearer. This process strengthens the ability to analyze changes in ship status, more accurately grasps the ship's operating status, effectively reduces the possibility of misjudgment of ship faults, and improves the quality of ship monitoring.

[0015] In combination with some embodiments of the first aspect, in some embodiments, after the step of determining whether there is a fault mode matching the timing relationship, the sensor change trend and the monitoring change trend in the preset fault mode library, the method further includes: if not, sending the data change time period and the ship change area to the target ship client.

[0016] By adopting the above technical solution, after the preset fault mode library determines that there is no matching mode, the data change time period and the ship change area are sent to the client. This operation is based on the previous determination of the data and area. When the fault mode cannot be determined, the relevant information is sent to the client so that the ship personnel can obtain the key information of the ship status change in time, which is convenient for further viewing and analysis, enhances the ship operator's understanding of the ship status, improves the ability to deal with abnormal situations during the operation of the ship, and ensures the normal operation and safety of the ship.

[0017] In combination with some embodiments of the first aspect, in some embodiments, after the step of sending an early warning instruction containing the fault mode to the target ship client if it exists, the method also includes: saving the current fault mode matching situation, and saving the current fault mode matching situation to the preset fault mode library.

[0018] By adopting the above technical solution, the current fault mode matching situation is saved and stored in the preset fault mode library. After determining whether there is a matching fault mode, saving the relevant situation can record the detailed information of each fault judgment of the ship. This information includes whether it matches, timing relationship, sensor and monitoring change trend, etc. After being stored in the preset fault mode library, the data of the fault mode library can be enriched.

[0019] In a second aspect, an embodiment of the present application provides a ship remote monitoring system, which includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the ship remote monitoring system to execute the method described in the first aspect and any possible implementation method of the first aspect.

[0020] In a third aspect, an embodiment of the present application provides a computer program product comprising instructions, which, when the above-mentioned computer program product is run on a ship remote monitoring system, enables the above-mentioned ship remote monitoring system to execute the method described in the first aspect and any possible implementation method of the first aspect.

[0021] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, comprising instructions. When the instructions are executed on a ship remote monitoring system, the ship remote monitoring system executes the method described in the first aspect and any possible implementation method in the first aspect.

[0022] It can be understood that the ship remote monitoring system provided in the second aspect, the computer program product provided in the third aspect, and the computer storage medium provided in the fourth aspect are all used to execute the method provided in the embodiment of the present application. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method, which will not be repeated here.

[0023] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: 1. This application obtains the target fault sensor and time period by receiving the real-time and historical sensor data of the target ship for comparison, and then obtains the corresponding video surveillance data for comparison to obtain the ship change area, and then analyzes the trend timing relationship between the two and determines the matching fault mode in the preset library. Finally, according to the results, an early warning instruction or related information is sent. The sensor data comparison can quickly locate the possible faulty sensors, and the video surveillance data further determines the ship change area. The timing relationship analysis and fault mode matching improve the accuracy of fault judgment and reduce the error rate of ship fault monitoring.

[0024] 2. This application determines the location, area and shape characteristics by matching the changed area of ​​the ship with the structural information, and then obtains the importance index based on this sorting. Finally, the alarm is graded according to the index and the monitoring frequency is adjusted. Matching the structural information can clarify key information such as the location of the area in the ship. The index obtained by sorting can quantify the importance of the area. Grading the alarm according to the index allows operators to quickly distinguish the degree of urgency, and adjusting the monitoring frequency can reasonably allocate monitoring resources.

[0025] 3. This application conducts time series analysis on the sensor and monitoring change trends respectively, extracts key feature points and event points, calculates the relationship and performs cluster analysis to build a time series relationship model. Time series analysis can dig out key information in the trend, and the extracted points and calculated relationships provide a basis for cluster analysis. The model constructed by cluster analysis integrates this information to make the relationship between sensors and monitoring change trends clearer. This process strengthens the ability to analyze changes in ship status, more accurately grasps the ship's operating status, effectively reduces the possibility of misjudgment of ship failures, and improves the quality of ship monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a flow chart of a method for remote monitoring of a ship in an embodiment of the present application; Figure 2 This is another flow chart of the method for remote monitoring of ships in an embodiment of the present application; Figure 3 It is a schematic diagram of the structure of a physical device of a ship remote monitoring system in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to be used as limitations to the present application. As used in the specification of the present application, the singular expressions "one", "a kind of", "above", "the" and "this" are intended to also include plural expressions, unless there is a clear indication to the contrary in the context. It should also be understood that the term "and / or" used in the present application refers to any or all possible combinations comprising one or more of the listed items.

[0028] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.

[0029] For ease of understanding, the following is a description of the process of the method provided by this implementation. Figure 1 , which is a flow chart of the ship remote monitoring method in an embodiment of the present application.

[0030] S101, receiving real-time sensor data of a target ship, comparing the real-time sensor data with historical sensor data, and obtaining a target faulty sensor and a data change time period.

[0031] Among them, "target ship" refers to a specific ship that needs to be remotely monitored. "Real-time sensor data" refers to the data obtained from various sensors on the target ship (such as temperature sensors, pressure sensors, liquid level sensors, etc.) at the current moment, which can reflect the operating status of the ship in real time. "Historical sensor data" refers to the data obtained from the same sensor over a period of time in the past, which is used for comparative analysis with real-time data.

[0032] This step is performed continuously during the operation of the ship remote monitoring system. Specifically, the ship remote monitoring system will continuously receive real-time data from various sensors of the target ship. Then, these real-time data are compared with the stored historical sensor data. By comparing the difference in data, it can be determined which sensors' data have changed. When the data change exceeds a certain threshold, the target faulty sensor and the time period corresponding to the data change can be determined.

[0033] S102: Acquire video monitoring data of the target fault sensor area within the data change time period, compare the video monitoring data with historical monitoring data, and obtain a ship change area.

[0034] Among them, "data change time period" refers to the time period in which the sensor data determined in step S101 changes. "Target fault sensor area" refers to the area of ​​the ship where the target fault sensor is located. This area may be related to the installation location of the sensor and its monitoring range. "Video surveillance data" refers to the image or video data obtained from the video surveillance equipment installed on the target ship. These data can intuitively reflect the actual situation of a specific area of ​​the ship. "Historical monitoring data" refers to the image or video data obtained from the same video surveillance equipment in the past period of time, which is used for comparative analysis with real-time video surveillance data.

[0035] Specifically, once the data change time period and the target faulty sensor area are determined, the video surveillance data of the target faulty sensor area within the time period is obtained from the ship's video surveillance system. These video surveillance data are then compared with the historical surveillance data. By comparing the differences in images or videos, it is possible to determine which areas on the ship have changed. For example, if the video surveillance data shows that a new object has appeared in a certain area or that the color, shape, etc. of a certain area has changed, and these changes do not appear in the historical surveillance data, then it can be determined that this area is a ship change area.

[0036] In some embodiments, this can be achieved in the following ways: first, a time stamp is set in the video surveillance system installed on the ship so that the video surveillance data within the specified time period can be accurately obtained, and the obtained video surveillance data is compared with the historical surveillance data using a suitable image or video processing algorithm. For example, an image difference algorithm can be used to calculate the difference between two images, and the comparison results can be analyzed and screened. For example, if the comparison results show that the difference in a certain area is very small, it may be due to normal fluctuations caused by factors such as lighting, then these differences can be ignored, and only the areas with larger differences can be focused on, so as to determine the area where the ship has changed.

[0037] S103: Analyze the temporal relationship between the sensor change trend of the target fault sensor and the monitoring change trend of the video monitoring data.

[0038] "Target fault sensor" refers to the sensor whose data has changed abnormally as determined in the previous step. "Sensor change trend" refers to the direction of the data of the target fault sensor changing over time, such as continuous rise, fall, fluctuation, etc. "Video surveillance data" refers to the video image data of the area related to the target fault sensor. "Monitoring change trend" refers to the trend of the regional conditions reflected by the video surveillance data changing over time, such as the changing rules of the appearance, disappearance, movement, etc. of objects in the area. "Time series relationship" refers to the relationship between the sensor change trend and the monitoring change trend in time, the delay, and whether there is a causal relationship.

[0039] Specifically, we must first clarify the specific manifestations of sensor change trends and monitoring change trends. Then analyze their various relationships on the timeline. For example, if the sensor data starts to rise at a certain moment, and the video surveillance data shows that abnormal smoke appeared in the area at a later time, then we need to analyze this sequence and whether there is a causal relationship between the appearance of smoke and the rise in sensor data.

[0040] In some embodiments, this can be achieved by: quantifying the sensor change trend. The speed and stability of the change can be described by calculating the slope, variance and other statistics of the data, and the monitoring change trend can be extracted. For example, the change characteristics of the area in the video, such as the size, shape, color and other changes of the object, are identified by image recognition technology. Finally, the quantified sensor change trend and the extracted monitoring change trend characteristics are compared and analyzed. By establishing a timeline, marking the key change points, and then analyzing the time intervals and logical relationships between these points, the timing relationship can be determined.

[0041] S104. In a preset fault mode library, determine whether there is a fault mode that matches the timing relationship, the sensor change trend, and the monitoring change trend.

[0042] The "preset fault mode library" refers to a pre-established database containing various possible fault modes and their related characteristic information. These fault modes are summarized based on previous ship fault experience and data analysis. Each fault mode has its own specific sensor change trend, monitoring change trend and the time series relationship between them.

[0043] Specifically, the obtained time series relationship and the change trend of sensors and monitoring are used as features to search and match in the preset fault mode library. If one or more fault modes that fully match these features can be found, it can be determined that the ship may have a corresponding fault. For example, if the sensor data continues to rise, the video surveillance shows that the temperature in a certain area rises and smoke appears, and there is a fault mode in the preset fault mode library that is the temperature and smoke changes in the relevant area caused by engine overheating, and its sensor and monitoring change trends and time series relationship match the current situation, then it can be determined that the ship may have an engine overheating fault.

[0044] In some embodiments, this can be achieved in the following ways: encoding the sensor change trend, monitoring change trend and timing relationship, converting them into a form that can be recognized and processed by a computer, establishing an indexing mechanism in a preset fault mode library so that a matching fault mode can be quickly found, and by traversing all modes in the preset fault mode library, comparing the encoded features with the features of each mode. If the matching degree reaches a certain threshold (for example, 80%), it is considered that a matching fault mode exists.

[0045] S105: If yes, send a warning instruction including the fault mode to the target ship client.

[0046] “Target vessel client” refers to the software or equipment installed on the target vessel for receiving and processing information sent by the remote monitoring system.

[0047] Specifically, when it is determined that there is a matching fault mode, the system will generate an early warning instruction containing detailed information of the fault mode. This early warning instruction will be sent to the target ship client. For example, if the matching fault mode is the overheating of the ship engine, the early warning instruction may contain relevant data on the overheating of the engine, possible dangerous situations, and recommended measures. After receiving this early warning instruction, the target ship client can take corresponding measures according to the information in the instruction, such as checking the engine cooling system, reducing the engine load, etc.

[0048] In some embodiments, this can be achieved in the following ways: First, the content of the early warning instruction is generated according to the matched fault mode. This requires extracting relevant fault information from the fault mode library, such as the fault name, possible cause, relevant sensor data and video surveillance data features, etc., and then arranging this information into a standardized instruction format and sending it to the target ship client.

[0049] The following is a more detailed description of the process of the method provided by this implementation. Figure 2 , is another flow chart of the ship remote monitoring method in an embodiment of the present application.

[0050] S201, receiving real-time sensor data of a target ship, comparing the real-time sensor data with historical sensor data, and obtaining a target faulty sensor and a data change time period.

[0051] It can be understood that this step is similar to step S101 and will not be described in detail here.

[0052] S202: Acquire video monitoring data of the target fault sensor area within the data change time period, compare the video monitoring data with historical monitoring data, and obtain a ship change area.

[0053] This step specifically includes: The video monitoring data of the ship remote monitoring system is compared with the historical monitoring data to obtain a difference image.

[0054] Image segmentation is performed on the difference images of the ship remote monitoring system to identify potential change areas.

[0055] The potential change area of ​​the ship remote monitoring system is screened according to the preset feature threshold to obtain the ship change area.

[0056] Specifically, the video surveillance data of the specific area of ​​the target ship at the current moment and the corresponding historical surveillance data are obtained. Then, a suitable image comparison algorithm, such as an image difference algorithm, is used. For each pixel, the difference between the video surveillance data and the historical surveillance data is calculated. If the difference exceeds a certain threshold, the pixel will be marked in the difference image. For example, if the color of a certain area becomes darker in the current video surveillance data, the difference calculated by the comparison algorithm will be reflected in the difference image, so as to obtain a difference image that highlights the change and obtain the difference image. Then, an image segmentation algorithm, such as a threshold-based segmentation algorithm or a region growth-based segmentation algorithm, is used. Taking the threshold-based segmentation algorithm as an example, a suitable threshold is set, and the area with a pixel value greater than the threshold in the difference image is divided into one area, and the area less than the threshold is divided into another area. In this way, the difference image is divided into multiple areas. Then, each segmented area is analyzed to determine whether it may be a potential change area. For example, if the color, texture and other features of a region are significantly different from those of the surrounding areas, or if there is no similar situation in the historical monitoring data, then it is identified as a potential change area, and the potential change area and the preset feature threshold are obtained. Then, for each potential change area, it is screened according to the preset feature threshold. For example, if the area threshold is preset to be 10 square meters, when the area of ​​a potential change area is less than 10 square meters, it will be excluded. If the color change threshold is preset to a certain color difference, when the color change of a potential change area does not reach the threshold, it will also be excluded. By screening each potential change area according to the preset feature threshold, the ship change areas that meet the requirements are finally obtained. These areas are the areas that have been strictly screened and determined to have actual changes.

[0057] S203: Match the ship change area with the ship structure information to determine the position data, area data and shape characteristics of the ship change area of ​​the ship remote monitoring system in the ship.

[0058] Ship structure information refers to detailed information about the overall structure of the ship, including the location, shape, size of each part of the ship and the connection relationship between them, etc. For example, the location and structural characteristics of each area of ​​the ship's deck, cabin, engine room, etc.

[0059] Specifically, the ship structure information and the determined ship change area are obtained, and the ship change area is matched with the ship structure information. For a certain shape feature in the ship change area, a similar structure part can be found in the ship structure information, and the matching relationship can be determined by comparing the shape, size and other features. If the match is successful, the position data of the ship change area in the ship can be determined, such as which deck or cabin of the ship it is located in. At the same time, its area data, that is, the size of the area, can be determined by calculating the number of pixels in the area or using other geometric measurement methods. In addition, its shape characteristics can also be determined, such as whether it is round, square or irregular.

[0060] S204: Based on the position data, the area data and the shape feature, the change area of ​​the ship is prioritized to obtain an importance index of the change area.

[0061] Position data refers to the specific location information of the ship's change area in the ship, such as a certain area on a certain deck, etc. Area data refers to the size of the space occupied by the ship's change area, which is obtained through a specific calculation method. Shape characteristics describe the appearance characteristics of the ship's change area, such as round, square, etc. The change area importance index is a value obtained by comprehensive evaluation of position data, area data and shape characteristics, and is used to measure the importance of the ship's change area.

[0062] The importance index of the change area is calculated based on the determined position data, area data and shape features. For position data, if the change area of ​​the ship is located in a key area, such as the engine room, cockpit and other important areas, then its importance will be relatively high. Different weight values ​​can be set for different key parts, for example, the weight value of the engine room is 0.5, the weight value of the cockpit is 0.4, etc. For area data, if the area of ​​the ship change area is large, it may mean a greater potential risk, and it will also be given a higher importance. A functional relationship can be set according to the size of the area to determine its contribution to the importance index. For shape features, if the shape is irregular, it may increase the difficulty of processing and potential risks, and will also affect the importance index. For example, the importance coefficient of a regular shape area can be set to 0.8, and the importance coefficient of an irregular shape area can be set to 0.6. Then these importance coefficients obtained according to the position, area and shape are weighted and summed to obtain the importance index of the change area.

[0063] S205: Perform graded alarms on the ship's changed area according to the importance index of the changed area, and adjust the video monitoring frequency of the ship's changed area.

[0064] The graded alarm is to issue different levels of alarms according to the different importance indexes of the changed areas, so that the operators can quickly understand the urgency of the changed areas of the ship. The video monitoring frequency refers to the time interval for video monitoring of the changed areas of the ship.

[0065] The graded alarm and video surveillance frequency are adjusted based on the calculated change area importance index. If the change area importance index is high, such as greater than 0.8, a high-level alarm, such as a red alarm, is issued, and the video surveillance frequency can be increased to once every 10 minutes or more frequently. If the change area importance index is between 0.5-0.8, a medium-level alarm, such as a yellow alarm, is issued, and the video surveillance frequency can be adjusted to once every 30 minutes. If the change area importance index is less than 0.5, a low-level alarm, such as a green alarm, is issued, and the video surveillance frequency can be adjusted to once every 60 minutes.

[0066] S206 , performing a time series analysis on the sensor change trend to obtain key feature points, which include data mutation points, trend change points, and periodic change points.

[0067] The sensor change trend refers to the direction of the data of the target fault sensor changing over time, such as continuous rise, fall, fluctuation, etc. The key feature points are points of special significance obtained after time series analysis of the sensor change trend, including data mutation points (points where the data suddenly changes drastically), trend change points (points where the data trend changes) and periodic change points (points where the data shows periodic changes).

[0068] Obtain the change trend data of the target fault sensor and perform time series analysis on it, such as the moving average method, the difference method, etc. Taking the moving average method as an example, by calculating the average value of the data in a certain time window, the data is smoothed and the change trend of the data is observed. For the data mutation point, a threshold can be set. When the change amplitude of the data exceeds the threshold, it is determined as the data mutation point. For example, the temperature sensor data suddenly rises from 25℃ to 50℃ at a certain moment, exceeding the set threshold of 10℃, then this point is the data mutation point. For the trend change point, when the slope of the data changes significantly, it is determined as the trend change point. For example, the pressure sensor data originally continued to rise, and at a certain moment the slope became zero and began to remain stable. This point is the trend change point. For the periodic change point, it is determined by observing whether the data shows periodic fluctuations. For example, the liquid level sensor data will have a peak and a trough every once in a while, which is a periodic change point.

[0069] S207, performing a time series analysis on the monitored change trend, and obtaining key event points including area appearance, area expansion, area reduction and area disappearance.

[0070] Obtain the changing trend of the regional conditions reflected by the video surveillance data. Then, perform time series analysis on it. The moving average method and other methods can also be used. For the regional appearance point, if at a certain moment, an object appears in the originally blank area of ​​the video surveillance screen, then the point corresponding to this moment is the regional appearance point. For example, in a cargo hold surveillance screen of a ship, a box of cargo appears in an area that originally had no cargo at a certain moment, and this moment is the regional appearance point. For the regional expansion point, when it is observed that the area occupied by the object in the region gradually increases, the corresponding point is the regional expansion point. For example, on the deck of the ship, there was originally a small pile of cargo. As time goes by, the area occupied by the cargo becomes larger and larger. In this process, the corresponding point for each increase in area is the regional expansion point. Similarly, for the regional reduction point and the regional vanishing point, when the area occupied by the object in the region decreases or the object disappears, the corresponding point is the regional reduction point or the regional vanishing point.

[0071] S208, calculating the time interval and occurrence order between the key feature point and the key event point, and extracting sensor-video event pairs, wherein the time interval of the sensor-video event pairs is less than a preset time threshold and conforms to a preset occurrence order pattern.

[0072] The time interval refers to the distance between the key feature point and the key event point on the time axis, and the appearance order refers to the order in which the key feature point and the key event point appear on the time axis. The sensor-video event pair is a pair of events consisting of a key feature point and a key event point. The time interval between them is required to be less than the preset time threshold and conform to the preset appearance order pattern.

[0073] Get the key feature points in S206 and the key event points in S207. Then, calculate the time interval and appearance order between them. For the time interval, it can be calculated by the coordinate difference of the two points on the time axis. For example, the coordinate of the data mutation point on the time axis is t1, and the coordinate of the regional appearance point on the time axis is t2, then the time interval between them is |t1-t2|. For the appearance order, it can be determined by comparing the positions of the two points on the time axis. Next, extract the sensor-video event pair. Set a preset time threshold, such as 5 minutes, and a preset appearance order mode, such as the data mutation point appears first, and then the regional appearance point appears within the specified time. When the time interval between the key feature point and the key event point is less than the preset time threshold and meets the preset appearance order mode, they are combined into a sensor-video event pair. For example, the temperature sensor data mutates at time t1, and at time t2 (t2-t1<5 minutes), an object that may be related to the temperature change appears in the video surveillance screen, then (temperature sensor data mutation point, object appearance point in the video surveillance screen) constitutes a sensor-video event pair.

[0074] S209 , performing cluster analysis on the sensor-video event to obtain a temporal pattern event group, and calculating the temporal similarity between different temporal pattern event groups.

[0075] A temporal pattern event group is a set of sensor-video events with similar temporal characteristics obtained through cluster analysis. These events have commonalities in time sequence and correlation patterns. For example, a group of temporal pattern event groups all show a pattern in which the sensor data first changes suddenly, and then some changes occur in the video surveillance area in a short period of time. For example, the events in a temporal pattern event group all have the characteristics of a sudden increase in temperature sensor data, followed by smoke or abnormal objects appearing in the cargo hold area within 3-5 minutes. Temporal similarity is an indicator to measure the degree of similarity between different temporal pattern event groups in terms of time sequence and pattern. It reflects the similarity of different groups of events in terms of time sequence, interval between events, and overall pattern. The higher the value, the higher the similarity. Suppose there are two temporal pattern event groups A and B. The events in group A are all changes in the area 3 minutes after the sensor data changes suddenly, and the events in group B are all changes in the area 4 minutes after the sensor data changes suddenly, and the change patterns are similar, then the temporal similarity between them is high.

[0076] Collect all sensor-video events. For each event, identify its key feature points of sensor data and key event points of video surveillance, as well as the time interval and order of occurrence between them. Then, use clustering algorithm to perform cluster analysis. Taking K-Means clustering algorithm as an example, first randomly select K initial cluster centers (the value of K can be determined based on experience or data characteristics), calculate the distance of each sensor-video event to these cluster centers (the distance metric can be constructed based on features such as time interval and order of occurrence), and assign each event to the class represented by the nearest cluster center. Then, recalculate the cluster center of each class, and repeat the above process until the cluster center no longer changes or reaches the preset number of iterations. Through such a clustering process, the temporal pattern event group is obtained, and the temporal similarity between different temporal pattern event groups is calculated. Using the dynamic time warping (DTW) algorithm, for two temporal pattern event groups A and B, a time warping path is constructed so that the events in group A and the events in group B are aligned as much as possible in time, and the cumulative distance on this path is calculated as the dissimilarity metric, and the temporal similarity is obtained by subtracting this dissimilarity metric from 1. For example, if there are 3 events in group A and 4 events in group B, and the dissimilarity measure calculated by the DTW algorithm is 0.3, then the temporal similarity is 0.7.

[0077] S210: Based on the time series similarity and the time series pattern event group, construct a time series relationship model between the sensor change trend and the monitoring change trend.

[0078] The temporal relationship model is a model used to describe the relationship between the sensor change trend and the monitoring change trend in terms of time sequence, delay, and causal relationship. The input of the temporal relationship model includes information related to the temporal pattern event group (such as key feature points of sensor data and key event points of video monitoring) and temporal similarity information. The output is the sequence, time delay, and causal strength. The construction process is: first collect preprocessed data, obtain the temporal pattern event group through cluster analysis, calculate the temporal similarity, select appropriate features to define independent variables (such as mutation time, change time, similarity, etc.) and dependent variables (relationship indicators), select multivariate linear regression model training, minimize the error by adjusting parameters, and finally use the validation set to evaluate and verify. If the requirements are met, the model is successfully constructed.

[0079] Obtain all the time series pattern event groups and the time series similarities between them. Then, select a suitable modeling method to construct a time series relationship model. For example, a multivariate linear regression model can be used. The sensor data mutation time, the video surveillance area change time and the related time series similarity in the time series pattern event group are used as independent variables, and the relationship between the sensor change trend and the monitoring change trend (such as time delay, causal relationship strength, etc.) is used as the dependent variable. By collecting a large amount of time series pattern event group data, analyzing and processing these data, determining the relationship coefficient between the independent variable and the dependent variable, a time series relationship model is constructed. For example, if it is found that the video surveillance area changes on average 3 minutes after the sensor data in a certain time series pattern event group mutates, and the time series similarity is high, then this time delay and the corresponding causal relationship strength will be reflected in the model.

[0080] S211, input the sensor change trend and the monitoring change trend into a time series relationship model to obtain a quantitative result between the sensor change trend and the monitoring change trend, wherein the quantitative result includes a sequence, a time delay, and a causal relationship strength.

[0081] The quantitative results are obtained after inputting the sensor change trend and the monitoring change trend into the temporal relationship model, including indicators such as sequence, time delay and causal strength, which reflect the specific relationship between the sensor change trend and the monitoring change trend. The sequence indicates which event occurs first; the time delay indicates the time interval between two events; the causal strength indicates the strength of the causal relationship between the two events. If the sensor change trend is that the data continues to rise, and the monitoring change trend is that objects appear in the area and gradually increase, after inputting them into the temporal relationship model, the quantitative results that may be obtained are that the sequence is that the sensor data changes first and the objects appear in the area later; the time delay is that the objects appear in the area 5 minutes after the sensor data changes; the causal strength is medium, indicating that there is a certain causal relationship between the sensor data change and the appearance of objects in the area, but it is not an absolute causal relationship.

[0082] Obtain data on sensor change trends and monitoring change trends. These data can be obtained from previous analysis of sensors and video surveillance. Then, input these data into the temporal relationship model. The model will perform calculations based on the input data and previously constructed model parameters. For example, if the sensor change trend is that the data continues to rise, and the monitoring change trend is that objects appear in the area and gradually increase, the model will calculate the order of sensor data change first and objects appear in the area later based on the temporal pattern event group and temporal similarity information obtained from the previous analysis; the time delay is that objects appear in the area 5 minutes after the sensor data changes; and the causal relationship strength is medium. The specific calculation process may involve algorithms and formulas within the model, which are determined based on the methods used to build the model (such as multivariate linear regression model) and data characteristics.

[0083] S212: Determine the temporal relationship between the sensor change trend and the monitoring change trend based on the quantification result.

[0084] The temporal relationship refers to the temporal sequence, delay, and whether there is a causal relationship between the sensor change trend and the monitoring change trend.

[0085] First, from S206, a time series analysis is performed on the sensor change trend to obtain key feature points, S207, a time series analysis is performed on the monitoring change trend to obtain key event points, S208, the relationship between the key feature points and the key event points is calculated and the sensor-video event pairs are extracted, S209, a cluster analysis is performed on the sensor-video events to obtain the temporal pattern event group and the temporal similarity is calculated, S210, a temporal relationship model is constructed based on the temporal similarity and the temporal pattern event group, and S211, the sensor change trend and the monitoring change trend are input into the model to obtain a quantitative result.

[0086] Then, the timing relationship is determined based on the quantification result obtained in S211. If the sequence in the quantification result indicates that the sensor change trend comes first, for example, the sensor data increases first, then in the timing relationship, it is determined that the sensor change trend precedes the monitoring change trend. For time delay, if the quantification result shows that an object appears in the area 5 minutes after the sensor data changes, then this time delay is clearly defined in the timing relationship. In terms of the strength of the causal relationship, if it is of medium strength, combined with the previous analysis, it can be seen that there is a certain causal relationship between the sensor change trend and the monitoring change trend, but it is not an absolute causal relationship. For example, an increase in sensor data may be a factor that causes smoke to appear in the area, but there may be other factors that affect it. By closely linking the previous steps in this way, the timing relationship between the sensor change trend and the monitoring change trend can be accurately determined based on the quantification results.

[0087] S213. In a preset fault mode library, determine whether there is a fault mode that matches the timing relationship, the sensor change trend, and the monitoring change trend.

[0088] It can be understood that this step is similar to step S104 and will not be described in detail here.

[0089] S214: If yes, a warning instruction including the fault mode is sent to the target ship client.

[0090] It can be understood that this step is similar to step S105 and will not be described in detail here.

[0091] S215: If not, send the data change time period and the ship change area to the target ship client.

[0092] When it is determined in the preset fault mode library that there is no fault mode that matches the timing relationship, sensor change trend, and monitoring change trend, it is necessary to send relevant information to the target ship client. First, obtain the information of the determined data change time period and the ship change area. Then, send this information through the communication link between the ship remote monitoring system and the target ship client. Network communication protocols such as TCP / IP protocol can be used to achieve data transmission. For example, if the ship remote monitoring system detects that the data of a pressure sensor changes from 2 to 3 pm, and the corresponding video monitoring area also changes, but the fault mode cannot be determined, the data change time period from 2 to 3 pm and the area information of the change in the video monitoring will be sent to the target ship client for further viewing and analysis by the ship personnel.

[0093] S216: Save the current fault mode matching situation, and save the current fault mode matching situation into the preset fault mode library.

[0094] After completing the fault mode judgment of the target ship, whether there is a matching fault mode or not, the current fault mode matching situation needs to be saved. First, sort out the relevant information of the current fault mode matching situation, including whether there is a match, what the matching fault mode is if there is a match, and the corresponding timing relationship, sensor change trend and monitoring change trend, etc. If there is no match, record the relevant unmatched information. Then, save this information to the preset fault mode library. The storage of information can be achieved through a database management system. For example, if the fault mode that is judged to be matched this time is engine overheating, the relevant timing relationship is that the sensor data first rises and then smoke appears in the video monitoring area, the sensor change trend is that the data continues to rise, and the monitoring change trend is that smoke appears and increases in the area, then this information is saved as the current fault mode matching situation to the preset fault mode library. If there is no match, record the unmatched relevant information, such as the data change time period, the ship change area, and the failure mode cannot be determined, and also save it to the preset fault mode library for subsequent reference and further improvement of the fault mode library.

[0095] The following describes the ship remote monitoring system in the embodiment of the present invention from the perspective of hardware processing. Figure 3 , which is a schematic diagram of a physical device structure of a ship remote monitoring system in an embodiment of the present application.

[0096] It should be noted that Figure 3 The structure of the ship remote monitoring system shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0097] like Figure 3 As shown, the ship remote monitoring system includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 302 or the program loaded from the storage part 308 to the random access memory (RAM) 303, such as executing the method described in the above embodiment. Various programs and data required for system operation are also stored in RAM 303. CPU 301, ROM 302 and RAM 303 are connected to each other through a bus 304. Input / output (I / O) interface 305 is also connected to bus 304.

[0098] The following components are connected to the I / O interface 305: an input section 306 including an audio input device, a button switch, etc.; an output section 307 including a liquid crystal display (LCD) and an audio output device, an indicator light, etc.; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to the I / O interface 305 as needed. A removable medium 311, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 310 as needed so that a computer program read therefrom is installed into the storage section 308 as needed.

[0099] In particular, according to an embodiment of the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 309, and / or installed from a removable medium 311. When the computer program is executed by the central processing unit (CPU) 301, various functions defined in the present invention are performed.

[0100] It should be noted that specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device.

[0101] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. Each box in the flowchart or block diagram may represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in an order different from that marked in the accompanying drawings.

[0102] Specifically, the ship remote monitoring system of this embodiment includes a processor and a memory. The memory stores a computer program. When the computer program is executed by the processor, the ship remote monitoring method provided in the above embodiment is implemented.

[0103] As another aspect, the present invention further provides a computer-readable storage medium, which may be included in the ship remote monitoring system described in the above embodiment; or may exist independently without being assembled into the ship remote monitoring system. The above storage medium carries one or more computer programs, and when the above one or more computer programs are executed by a processor of the ship remote monitoring system, the ship remote monitoring system implements the ship remote monitoring method provided in the above embodiment.

[0104] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0105] As used in the above embodiments, the term "when..." may be interpreted to mean "if..." or "after..." or "in response to determining..." or "in response to detecting...", depending on the context. Similarly, the phrases "upon determining..." or "if (the stated condition or event) is detected" may be interpreted to mean "if determining..." or "in response to determining..." or "upon detecting (the stated condition or event)" or "in response to detecting (the stated condition or event)", depending on the context.

[0106] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

Claims

1. A ship remote monitoring method, characterized in that: Applied to a ship remote monitoring system, the method comprises: Receiving real-time sensor data of a target ship, comparing the real-time sensor data with historical sensor data, and obtaining a target fault sensor and a data change time period; Acquire video monitoring data of the target fault sensor area within the data change time period, compare the video monitoring data with historical monitoring data, and obtain a ship change area; Analyzing the temporal relationship between the sensor change trend of the target fault sensor and the monitoring change trend of the video monitoring data; In a preset fault mode library, determining whether there is a fault mode that matches the timing relationship, the sensor change trend, and the monitoring change trend; If so, an early warning instruction including the failure mode is sent to the target ship client.

2. The method according to claim 1, characterized in that The step of comparing the video monitoring data with the historical monitoring data to obtain the ship change area specifically includes: Comparing the video surveillance data with historical surveillance data to obtain a difference image; Performing image segmentation on the difference image to identify potential change areas; The potential change area is screened according to a preset feature threshold to obtain a ship change area.

3. The method according to claim 2, characterized in that After the step of screening the potential change area according to the preset feature threshold to obtain the ship change area, the method further includes: Matching the ship change area with the ship structure information to determine the position data, area data and shape characteristics of the ship change area in the ship; Based on the position data, the area data and the shape feature, the ship change area is prioritized to obtain a change area importance index; The ship change area is graded and alarmed according to the change area importance index, and the video monitoring frequency of the ship change area is adjusted.

4. The method according to claim 1, characterized in that: The step of analyzing the temporal relationship between the sensor change trend of the target fault sensor and the monitoring change trend of the video monitoring data specifically includes: Inputting the sensor change trend and the monitoring change trend into a time series relationship model to obtain a quantitative result between the sensor change trend and the monitoring change trend, wherein the quantitative result includes a sequence, a time delay, and a causal relationship strength; The temporal relationship between the sensor change trend and the monitoring change trend is determined based on the quantification result.

5. The method according to claim 4, characterized in that Before the step of inputting the sensor change trend and the monitoring change trend into a time series relationship model to obtain a quantitative result between the sensor change trend and the monitoring change trend, wherein the quantitative result includes a sequence, a time delay, and a causal relationship strength, the method further includes: Performing time series analysis on the sensor change trend to obtain key feature points, wherein the key feature points include data mutation points, trend change points and periodic change points; Performing time series analysis on the monitoring change trend to obtain key event points including area appearance, area expansion, area reduction and area disappearance; Calculating the time interval and the occurrence order between the key feature point and the key event point, and extracting sensor-video event pairs, wherein the time interval of the sensor-video event pairs is less than a preset time threshold and conforms to a preset occurrence order pattern; Performing cluster analysis on the sensor-video events to obtain time sequence pattern event groups, and calculating the time sequence similarities between different time sequence pattern event groups; Based on the time series similarity and the time series pattern event group, a time series relationship model between the sensor change trend and the monitoring change trend is constructed.

6. The method according to claim 1, characterized in that After the step of determining whether there is a fault mode matching the timing relationship, the sensor change trend, and the monitoring change trend in the preset fault mode library, the method further includes: If not present, the data change time period and the ship change area are sent to the target ship client.

7. The method according to claim 1, characterized in that After the step of sending the warning instruction containing the fault mode to the target ship client if it exists, the method further includes: The current fault mode matching situation is saved, and the current fault mode matching situation is saved in the preset fault mode library.

8. A ship remote monitoring system, characterized in that: The ship remote monitoring system includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the ship remote monitoring system to execute the method as described in any one of claims 1-7.

9. A computer-readable storage medium comprising instructions, characterized in that: When the instruction is executed on the ship remote monitoring system, the ship remote monitoring system is caused to execute the method according to any one of claims 1 to 7.

10. A computer program product, characterized in that When the computer program product runs on a ship remote monitoring system, the ship remote monitoring system is enabled to execute the method according to any one of claims 1 to 7.

Images