A method and device for identifying an anomaly in ship navigation data

Abstract

The application discloses a kind of ship navigation data exception discrimination method and device, the method includes: ship navigation text data set is obtained by acquisition;The ship navigation text data set includes the text data generated by monitoring each subsystem of ship, ship navigation instruction data, ship navigation rule text data, ship each subsystem's manual data and equipment acceptance data;The ship navigation text data set is preprocessed, and preprocessing data set is obtained;The preprocessing data set is processed by exception discrimination, and exception discrimination result information is obtained;The exception discrimination result information is used to indicate whether the navigation state of ship appears exception.

Description

Technical Field

[0001] This invention relates to the fields of industrial data processing, text data processing, and big data mining, specifically to a method and apparatus for anomaly detection in ship navigation data. Background Technology

[0002] In the process of safe navigation, the safe operation of ships heavily relies on the accurate monitoring and anomaly detection of various navigation data. With the increasing level of ship automation, the amount of textual data generated by various ship subsystems, such as monitoring data, navigation instructions, navigation rules, and equipment manuals, is exploding. This data is diverse in format, scattered in source, and contains a large number of technical terms, posing significant challenges to traditional data processing and anomaly detection methods. Current technologies for ship anomaly detection largely rely on threshold judgments or simple rule matching for structured data (such as sensor values), making it difficult to process unstructured textual data. Ship navigation-related textual data (such as equipment operation logs, operating instructions, and rule manuals) suffers from problems such as high noise levels, inconsistent formats, and mixed language. Existing technologies lack targeted preprocessing methods, resulting in low data quality and affecting the accuracy of subsequent analysis. Constraintive texts such as ship navigation rules and equipment usage specifications are often processed separately from real-time operational status data, making it difficult to establish a correlation between the two and to promptly detect anomalies where "actual operational status violates preset rules."

[0003] The maritime field contains a large number of specialized entities (such as equipment names, operating instructions, parameter indicators, etc.). Existing entity recognition and relationship extraction methods are not optimized for maritime scenarios, resulting in mismatch or omission of entity relationships, which affects the reliability of anomaly detection.

[0004] Existing technologies often rely on simple numerical comparisons to identify anomalies, neglecting the implicit semantic relationships and complex connections within textual data, making it difficult to capture potential, non-explicit anomaly patterns. These issues result in low accuracy in ship anomaly detection, high false alarm and missed alarm rates, and an inability to meet the high reliability requirements for ship navigation safety. Summary of the Invention

[0005] This invention primarily addresses the problem of how to effectively monitor the safety status of ship navigation based on unstructured ship operation status and rule text data. This invention discloses a method and apparatus for anomaly detection in ship navigation data.

[0006] In a first aspect, this invention discloses a method for anomaly detection in ship navigation data, comprising:

[0007] S1, Collect the ship navigation text dataset; the ship navigation text dataset includes text data generated by monitoring various subsystems of the ship, ship navigation instruction data, ship navigation rule text data, user manual data of various subsystems of the ship, and equipment acceptance data;

[0008] S2, preprocess the ship navigation text dataset to obtain a preprocessed dataset;

[0009] S3, perform anomaly detection processing on the preprocessed dataset to obtain anomaly detection result information; the anomaly detection result information is used to characterize whether the ship's navigation status is abnormal.

[0010] The preprocessing of the ship navigation text dataset to obtain a preprocessed dataset includes:

[0011] S21, perform text cleaning processing on the ship navigation text dataset to obtain the first dataset;

[0012] S22, Perform data normalization on the first dataset to obtain the second dataset;

[0013] S23, perform text representation processing on the second dataset to obtain a preprocessed dataset.

[0014] The anomaly detection processing of the preprocessed dataset to obtain anomaly detection result information includes:

[0015] S31, Perform constraint rule extraction processing on the preprocessed dataset to obtain a rule relationship matrix;

[0016] S32, Perform runtime status extraction processing on the preprocessed dataset to obtain the entity relationship matrix;

[0017] S33, perform difference discrimination on the rule relationship matrix and entity relationship matrix to obtain anomaly discrimination result information.

[0018] The step of extracting constraint rules from the preprocessed dataset to obtain a rule relationship matrix includes:

[0019] S311, Entity recognition processing is performed on the rule-related data in the preprocessed dataset to obtain the first entity set information; the rule-related data includes the corresponding data of ship navigation rule text data, user manual data of each subsystem of the ship, and equipment acceptance data in the preprocessed dataset;

[0020] S312, based on the first entity set information, perform entity relationship extraction processing on the rule-related data in the preprocessed dataset to obtain first relationship data information;

[0021] S313, Based on the first relational data information, a rule relational matrix is ​​constructed.

[0022] The step of extracting the runtime status of the preprocessed dataset to obtain the entity relationship matrix includes:

[0023] S321, Entity recognition processing is performed on the state-related data in the preprocessed dataset to obtain the second entity set information; the state-related data includes the corresponding data of text data generated by monitoring various subsystems of the ship and ship navigation command data in the preprocessed dataset;

[0024] S322, Based on the second entity set information, perform entity relationship extraction processing on the state-related data in the preprocessed dataset to obtain the second relationship data information;

[0025] S323, Based on the second relational data information, an entity relation matrix is ​​constructed.

[0026] The step of performing difference discrimination on the rule relation matrix and entity relation matrix to obtain anomaly discrimination result information includes:

[0027] S331, Subtract the rule relation matrix and the entity relation matrix to obtain the difference matrix;

[0028] S332, Calculate the median and mode values ​​of each row vector of the difference matrix;

[0029] S333, perform the first difference fusion calculation on the median and mode values ​​of all row vectors to obtain the first difference value;

[0030] S334, Based on the rule relationship matrix and entity relationship matrix, a difference feature fusion model is constructed;

[0031] S335, Solve the difference feature fusion model to obtain the second difference value;

[0032] S336, The first difference value and the second difference value are weighted and summed to obtain the outlier;

[0033] S337, determine whether the abnormal value is greater than a preset abnormal threshold to obtain a first determination result; if the first determination result is yes, determine that the abnormal determination result information is abnormal; if the first determination result is no, determine that the abnormal determination result information is not abnormal.

[0034] The expression for the differential feature fusion model is:

[0035] f = argmax f|rank(X1-f×X2)-λ0|,

[0036] 0 < f < K,

[0037] Where K is the preset boundary value, f is the second difference value to be solved, λ0 is the average value of all eigenvalues ​​of the difference matrix, rank(X1-f×X2) means to calculate the rank of the matrix (X1-f×X2), and X1 and X2 represent the regular relation matrix and the entity relation matrix, respectively.

[0038] A second aspect of the present invention discloses an anomaly detection device for ship navigation data, the device comprising:

[0039] Memory containing executable program code;

[0040] A processor coupled to the memory;

[0041] The processor calls the executable program code stored in the memory to execute the anomaly detection method for ship navigation data.

[0042] In a third aspect of this invention, a computer-storable medium is disclosed, wherein the computer-storable medium stores computer instructions, and when the computer instructions are invoked by a computer, they are used to execute the method for anomaly detection of ship navigation data.

[0043] In a fourth aspect of this invention, an information data processing terminal is disclosed, which is used to implement the anomaly detection method for ship navigation data.

[0044] The beneficial effects of this invention are as follows:

[0045] This invention addresses the diversity of ship text data (monitoring data, command data, rule manuals, etc.) by designing a preprocessing workflow that includes text cleaning, normalization, and representation. Through technologies such as Unicode normalization, spelling correction, and language detection, the quality of unstructured text data is significantly improved, laying a reliable foundation for subsequent analysis.

[0046] This invention innovatively divides ship text data into rule-related data (navigation rules, manuals, etc.) and status-related data (monitoring logs, instructions, etc.), constructs rule relationship matrices and entity relationship matrices respectively, and achieves accurate comparison between "preset rules" and "actual status" through matrix difference analysis, effectively identifying inconsistencies between the two.

[0047] Entity recognition is performed by combining a pre-defined maritime vessel entity name database with a CRF model, and the recognition effect is optimized for maritime terminology. In relation extraction, multiple similarity indicators (Hamin distance, Jaccard coefficient, etc.) are integrated with text difference calculation to comprehensively capture the semantic relationships between entities, improving the accuracy of relation extraction. Through multi-level difference discrimination methods such as difference matrix analysis, first difference fusion calculation, and difference feature fusion model, which comprehensively consider matrix element differences, row vector distribution characteristics, and matrix rank characteristics, explicit and potential anomaly patterns can be identified. Weighted fusion and threshold judgment mechanisms further enhance the robustness of anomaly detection.

[0048] All the technical means of this invention are designed for the characteristics of ship text data, such as multilingual standardization, adaptation to marine entity databases, and fusion of complex relationships, which significantly improves the practicality and effectiveness in ship navigation scenarios and provides strong protection for safe ship navigation. Attached Figure Description

[0049] Figure 1 This is a flowchart illustrating the implementation of the method of the present invention. Detailed Implementation

[0050] To better understand the content of this invention, an embodiment is provided here.

[0051] Figure 1 This is a flowchart illustrating the implementation of the method of the present invention.

[0052] In a first aspect, this invention discloses a method for anomaly detection in ship navigation data, comprising:

[0053] S1, Collect the ship navigation text dataset; the ship navigation text dataset includes text data generated by monitoring various subsystems of the ship, ship navigation instruction data, ship navigation rule text data, user manual data of various subsystems of the ship, and equipment acceptance data;

[0054] S2, preprocess the ship navigation text dataset to obtain a preprocessed dataset;

[0055] S3, perform anomaly detection processing on the preprocessed dataset to obtain anomaly detection result information; the anomaly detection result information is used to characterize whether the ship's navigation status is abnormal.

[0056] The preprocessing of the ship navigation text dataset to obtain a preprocessed dataset includes:

[0057] S21, perform text cleaning processing on the ship navigation text dataset to obtain the first dataset;

[0058] S22, Perform data normalization on the first dataset to obtain the second dataset;

[0059] S23, perform text representation processing on the second dataset to obtain a preprocessed dataset;

[0060] The text cleaning process can employ Unicode normalization and spell correction methods.

[0061] The data normalization process can be achieved using language detection tools, such as Polyglot.

[0062] The text representation processing can employ either the bag-of-words model or one-hot encoding.

[0063] The anomaly detection processing of the preprocessed dataset to obtain anomaly detection result information includes:

[0064] S31, Perform constraint rule extraction processing on the preprocessed dataset to obtain a rule relationship matrix;

[0065] S32, Perform runtime status extraction processing on the preprocessed dataset to obtain the entity relationship matrix;

[0066] S33, perform difference discrimination on the rule relationship matrix and entity relationship matrix to obtain anomaly discrimination result information.

[0067] The step of extracting constraint rules from the preprocessed dataset to obtain a rule relationship matrix includes:

[0068] S311, Entity recognition processing is performed on the rule-related data in the preprocessed dataset to obtain the first entity set information; the rule-related data includes the corresponding data of ship navigation rule text data, user manual data of each subsystem of the ship, and equipment acceptance data in the preprocessed dataset;

[0069] S312, based on the first entity set information, perform entity relationship extraction processing on the rule-related data in the preprocessed dataset to obtain first relationship data information;

[0070] S313, Based on the first relational data information, a rule relational matrix is ​​constructed.

[0071] The step of extracting the runtime status of the preprocessed dataset to obtain the entity relationship matrix includes:

[0072] S321, Entity recognition processing is performed on the state-related data in the preprocessed dataset to obtain the second entity set information; the state-related data includes the corresponding data of text data generated by monitoring various subsystems of the ship and ship navigation command data in the preprocessed dataset;

[0073] S322, Based on the second entity set information, perform entity relationship extraction processing on the state-related data in the preprocessed dataset to obtain the second relationship data information;

[0074] S323, Based on the second relational data information, an entity relation matrix is ​​constructed.

[0075] The entity recognition process is achieved by using a pre-defined maritime vessel entity name database and a named entity recognition algorithm to process the text data. The named entity recognition algorithm can be implemented using a CRF model.

[0076] The entity relationship extraction process includes:

[0077] For each combination of two entity information in the first or second entity set information, the corresponding text vector information in the relevant data in the preprocessed dataset is extracted;

[0078] Text similarity calculation is performed on the corresponding text vector information of the combination of the two entity information to obtain text similarity set information; the text similarity set information includes the Hamming distance, Jaccard coefficient, and VDM (Value Difference Metric) distance of the corresponding text vector information of the combination of the two entity information.

[0079] For the corresponding text vector information of the combination of the two entity information, the text difference is calculated to obtain the text vector difference value;

[0080] The text similarity set information and the text vector difference value are fused and calculated to obtain the relationship data corresponding to the combination of the two entity information;

[0081] By combining the relationship data corresponding to all two entity information, the first or second relationship data information can be constructed.

[0082] The expression for calculating the text difference is:

[0083]

[0084] Among them, r1 i and r2 iLet r10 and r20 be the mean of all elements of the corresponding text vector information of the first entity information and the corresponding text vector information of the second entity information in the combination of two entity information, respectively. Let r10 and r20 be the mean of all elements of the corresponding text vector information of the first entity information and the corresponding text vector information of the second entity information in the combination of two entity information, respectively. Let N be the length of the text vector information and c be the text vector difference value.

[0085] The expression for calculating text differences captures local details by comparing vector elements point by point (numerator) and reflects the global distribution characteristics of the vectors through the overall mean (denominator), avoiding biases caused by single-scale analysis. For example, in ship equipment operation logs, it can identify both local anomalies such as "temperature values ​​exceeding limits" and global anomalies such as "overall parameter distribution shifts." By using exponential functions and square root operations to perform nonlinear transformations on the difference values, the impact of isolated noise points on the overall difference assessment is reduced. For occasional spelling errors or formatting errors in ship text data, this expression can effectively filter out interference and focus on substantive differences. Addressing the sparsity and specialization of ship-related text vectors (such as equipment parameters and operation command vectors), mean normalization (denominator) eliminates the incomparability of different types of vectors (such as parameters with different dimensions like temperature and pressure), making the difference values ​​more meaningful in practice.

[0086] The expression for the fusion calculation is:

[0087]

[0088] Where Π represents the calculation of the third kind of incomplete elliptic integral. and These are Hamming distance, Jaccard coefficient, and VDM (Value Difference Metric) distance, respectively, and g is the relational data corresponding to the combination of the two entity information.

[0089] This expression employs a third type of incomplete elliptic integral to fuse multi-dimensional features. The nonlinear properties of elliptic integrals can simulate complex nonlinear relationships in entity relations, and compared to simple weighted summation, it better reflects the implicit relationships between entities in the shipbuilding domain (such as the nonlinear mapping relationship between "main engine speed" and "ship speed"). Hamming distance (reflecting differences in vector binary features), Jaccard coefficient (reflecting set overlap), and VDM distance (reflecting differences in discrete features) are incorporated into a unified framework. Through integral operations, complementary information is organically fused, avoiding the one-sidedness of a single indicator. For example, when judging the consistency between "navigation instructions" and "execution results," text structure, semantic overlap, and the matching degree of discrete parameters can be considered simultaneously.

[0090] The smoothing properties of integral operations reduce the impact of fluctuations in a single indicator on the results, making relationship assessment more robust. It maintains stable relationship measurement results even for common variations in ship text (such as "full speed ahead" and "driving at full speed").

[0091] The relation matrix is ​​constructed by using the relation data corresponding to the i-th entity information and the j-th entity information as the elements of the i-th row and j-th column of the relation matrix.

[0092] The dimensions of the rule relation matrix and the entity relation matrix are the same.

[0093] The step of performing difference discrimination on the rule relation matrix and entity relation matrix to obtain anomaly discrimination result information includes:

[0094] S331, Subtract the rule relation matrix and the entity relation matrix to obtain the difference matrix;

[0095] S332, Calculate the median and mode values ​​of each row vector of the difference matrix;

[0096] S333, perform the first difference fusion calculation on the median and mode values ​​of all row vectors to obtain the first difference value;

[0097] S334, Based on the rule relationship matrix and entity relationship matrix, a difference feature fusion model is constructed;

[0098] S335, Solve the difference feature fusion model to obtain the second difference value;

[0099] S336, The first difference value and the second difference value are weighted and summed to obtain the outlier;

[0100] S337, determine whether the abnormal value is greater than a preset abnormal threshold to obtain a first determination result; if the first determination result is yes, determine that the abnormal determination result information is abnormal; if the first determination result is no, determine that the abnormal determination result information is not abnormal.

[0101] The expression for the differential feature fusion model is:

[0102] f = argmax f |rank(X1-f×X2)-λ0|,

[0103] 0 < f < K,

[0104] Where K is the preset boundary value, f is the second difference value to be solved, λ0 is the average value of all eigenvalues ​​of the difference matrix, rank(X1-f×X2) means to calculate the rank of the matrix (X1-f×X2), and X1 and X2 represent the regular relation matrix and the entity relation matrix, respectively.

[0105] The matrix rank in the described difference feature fusion model reflects the number of linearly independent rows (columns) of the matrix and is highly sensitive to structural changes in the relationship network (such as the addition / disappearance of entity associations). Compared to element-level comparison, it can more efficiently identify "systemic conflicts" between ship operating status and rules (such as a certain type of operation completely violating the rule framework). By solving for the parameter that minimizes the deviation between the matrix rank and the mean of eigenvalues ​​through argmax, it ensures the optimal characterization of overall differences and avoids misjudgments caused by local optima. For example, when analyzing "ship-wide equipment cooperation relationships" and "rule-stipulated cooperation processes," it can assess whether there are systematic deviations from a global perspective. By compressing high-dimensional relationship data into a single difference value through matrix operations, computational complexity is reduced while retaining key features. For ship relationship networks containing hundreds of entities, it can still efficiently complete difference assessment.

[0106] The expression for the first difference fusion calculation is:

[0107]

[0108] Where f1 is the first difference value, M is the row dimension of the difference matrix, and u i and a i These are the median and mode values ​​of the i-th row vector of the difference matrix, respectively.

[0109] The arcsine function in the first difference fusion calculation amplifies small differences while the exponential function strengthens large differences, complementing each other. This makes the model sensitive to both significant anomalies (such as excessive deviation between the median and mode) and subtle anomalies (such as slight distribution shifts). For example, when analyzing row vectors in a ship's subsystem log, it can identify obvious anomalies like "parameter mutations" and capture potential anomalies like "slow trend deviations." Normalization using the ratio of (median - mode) to the median eliminates the magnitude differences between different row vectors (corresponding to different devices or operations), making the difference values ​​horizontally comparable. For example, the difference assessment between the "navigation system" and the "power system" can be compared on a unified scale. The arcsine function maps differences to the [0,1] interval, while the exponential function highlights extreme differences. The superposition of the two facilitates quantitative analysis and aligns with human intuitive understanding of "anomaly severity," which is beneficial for the rationality of subsequent threshold judgments.

[0110] The weight values ​​for weighted fusion of the first difference value and the second difference value can be 0.3 and 0.7, respectively.

[0111] The abnormal threshold value is M / 3.

[0112] The Monte Carlo algorithm can be used to solve the differential feature fusion model.

[0113] All calculation expressions in this invention have undergone dimension normalization of the variables involved before calculation.

[0114] A second aspect of the present invention discloses an anomaly detection device for ship navigation data, the device comprising:

[0115] Memory containing executable program code;

[0116] A processor coupled to the memory;

[0117] The processor calls the executable program code stored in the memory to execute the anomaly detection method for ship navigation data.

[0118] In a third aspect of this invention, a computer-storable medium is disclosed, wherein the computer-storable medium stores computer instructions, and when the computer instructions are invoked by a computer, they are used to execute the method for anomaly detection of ship navigation data.

[0119] In a fourth aspect of this invention, an information data processing terminal is disclosed, which is used to implement the anomaly detection method for ship navigation data.

[0120] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A method of anomaly discrimination of ship navigation data, characterized by, The method comprises the following steps: S1, collecting a ship navigation text data set; the ship navigation text data set comprises text data generated by monitoring each subsystem of the ship, ship navigation instruction data, ship navigation rule text data, manual data of each subsystem of the ship, and equipment acceptance data; S2, preprocessing the ship navigation text data set to obtain a preprocessed data set, comprising: S21, performing text cleaning processing on the ship navigation text data set to obtain a first data set; S22, performing data normalization processing on the first data set to obtain a second data set; S23, performing text representation processing on the second data set to obtain a preprocessed data set; S3, performing abnormality discrimination processing on the preprocessed data set to obtain abnormality discrimination result information; the abnormality discrimination result information is used to represent whether the ship navigation state is abnormal, comprising: S31, performing constraint rule extraction processing on the preprocessed data set to obtain a rule relationship matrix, comprising: S311, performing entity recognition processing on rule-related data in the preprocessed data set to obtain first entity set information; the rule-related data comprises corresponding data of ship navigation rule text data, manual data of each subsystem of the ship, and equipment acceptance data in the preprocessed data set; S312, based on the first entity set information, performing entity relationship extraction processing on the rule-related data in the preprocessed data set to obtain first relationship data information; S313, based on the first relationship data information, constructing a rule relationship matrix; S32, performing running state extraction processing on the preprocessed data set to obtain an entity relationship matrix, comprising: S321, performing entity recognition processing on state-related data in the preprocessed data set to obtain second entity set information; the state-related data comprises corresponding data of text data generated by monitoring each subsystem of the ship and ship navigation instruction data in the preprocessed data set; S322, based on the second entity set information, performing entity relationship extraction processing on the state-related data in the preprocessed data set to obtain second relationship data information; S323, based on the second relationship data information, constructing an entity relationship matrix; S33, performing difference discrimination on the rule relationship matrix and the entity relationship matrix to obtain abnormality discrimination result information.

2. The method of claim 1, wherein The difference discrimination on the rule relationship matrix and the entity relationship matrix to obtain abnormality discrimination result information comprises: S331, subtracting the rule relationship matrix from the entity relationship matrix to obtain a difference matrix; S332, statistically obtaining the median value and the mode value of each row vector of the difference matrix; S333, performing first difference fusion calculation on the median value and the mode value of all row vectors to obtain a first difference value; S334, based on the rule relationship matrix and the entity relationship matrix, constructing a difference feature fusion model; S335, solving the difference feature fusion model to obtain a second difference value; S336, performing weighted summation on the first difference value and the second difference value to obtain an abnormal value; S337, determining whether the abnormal value is greater than a preset abnormal threshold to obtain a first determination result; if the first determination result is yes, determining that the abnormal determination result information is abnormal; if the first determination result is no, determining that the abnormal determination result information is not abnormal.

3. The method of claim 2, wherein the abnormality determination of the ship navigation data is performed by using a neural network. An expression of the difference feature fusion model is: f = argmax f |rank(X1 - f X2) - λ0|, 0 < f < K, wherein, K is a preset boundary value, f is a second difference value to be solved, λ0 is an average value of all eigenvalues of a difference matrix, rank(X1-f×X2) represents a rank of a matrix (X1-f×X2), X1 and X2 represent a regular relationship matrix and an entity relationship matrix respectively.

4. An abnormality discriminating device for ship navigation data, characterized by comprising: a ship navigation data abnormality discriminating device according to any one of claims 1 to 3. The device comprises: a memory storing executable program codes; a processor coupled with the memory; the processor invokes the executable program codes stored in the memory to execute the abnormality determination method of ship navigation data according to any one of claims 1 to 3.

5. A computer storable medium, characterized by The computer storage medium stores computer instructions, which are invoked by a computer to execute the abnormality determination method of ship navigation data according to any one of claims 1 to 3.

6. An information data processing terminal, characterized by The information data processing terminal is used to implement the abnormality determination method of ship navigation data according to any one of claims 1 to 3.

Images