Marine surveying and mapping data abnormal value detection and quality control method based on machine learning

Abstract

The invention relates to the field of marine data detection, and discloses a marine surveying and mapping data abnormal value detection and quality control method based on machine learning, which comprises the following steps: carrying out feature extraction on collected marine surveying and mapping data to construct and train a model capable of realizing feature anomaly detection, and abnormal value detection and confidence evaluation are performed on all features of the marine surveying and mapping data through the model, and a feature quality control method is generated according to detection and evaluation results. According to the invention, a bottom-layer technical support can be provided for ocean resource development, navigation safety and scientific research through modes of multi-dimensional feature deep mining, complex nonlinear relationship construction and automatic processing flow construction; the purposes of improving the detection efficiency, improving the accuracy of complex anomaly recognition, improving the adaptive capacity, reducing the false alarm rate under multi-source data fusion and the like are achieved.

Description

Technical Field

[0001] This invention relates to the field of marine data detection, and in particular to a method for outlier detection and quality control of marine mapping data based on machine learning. Background Technology

[0002] Marine mapping data is a collection of marine spatial information acquired through acoustic, optical, and magnetic methods. Its core objective is to construct digital models of the seabed topography, geology, and hydrological environment. Data types include water depth data, acoustic image data, water body physical data, positioning attitude data, seabed classification data, and magnetic and gravity data. Different types of data have different uses and contents. For example, water depth data contains the depth values, beam angles of arrival, and propagation times at various points on the seabed, and its purpose is to generate seabed topographic maps.

[0003] In the ocean, safe navigation is paramount for ships. Therefore, analyzing marine mapping data fed back to vessels to determine the safety of navigation in the current ocean area is crucial. Ships may face risks due to equipment malfunctions, environmental interference, or operational errors, making marine mapping data essential for ensuring safe navigation. Compared to traditional methods for analyzing marine mapping data and detecting outliers, machine learning offers revolutionary advantages. These include improved detection efficiency, higher accuracy in identifying complex anomalies, enhanced adaptability, and reduced false alarm rates from multi-source data fusion. Anomaly detection and quality control of marine mapping data are core components in ensuring the accuracy and reliability of marine spatial information. Machine learning methods, through multi-dimensional feature depth mining, construction of complex nonlinear relationships, and automated processing workflows, provide underlying technical support for marine resource development, navigation safety, and scientific research. Therefore, this paper proposes a machine learning-based method for anomaly detection and quality control of marine mapping data. Summary of the Invention

[0004] This invention overcomes the shortcomings of existing technologies and provides a method for outlier detection and quality control of marine mapping data based on machine learning.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows: The first aspect of this invention provides a method for outlier detection and quality control of marine mapping data based on machine learning, comprising the following steps: Collect marine mapping data and perform data preprocessing on the marine mapping data to obtain target marine mapping data, and extract the features of the target marine mapping data; By combining the characteristics of the target marine mapping data, a machine learning model for outlier detection is constructed and defined as the target anomaly detection model; An anomaly detection model is used to detect outliers and assess confidence levels for all features of target marine mapping data, and a feature quality control method is generated based on the detection and assessment results.

[0006] Furthermore, in a preferred embodiment of the present invention, the process of collecting marine mapping data, preprocessing the marine mapping data to obtain target marine mapping data, and extracting features from the target marine mapping data specifically includes: The sea area to be surveyed is marked as the target sea area. Multi-source surveying equipment is deployed in the target sea area, and the multi-source surveying equipment is controlled to collect marine surveying data in real time in the target sea area to obtain multi-source real-time marine surveying data. At the same time, the multi-source real-time marine surveying data is correlated with latitude and longitude and with timestamps. Data cleaning processing is performed on multi-source real-time marine mapping data. The data cleaning processing includes interpolation filling processing for local missing ranges and range removal training processing for large missing ranges of multi-source real-time marine mapping data. At the same time, the data cleaning multi-source real-time marine mapping data is filtered by the moving median filtering algorithm to obtain the target marine mapping data. Data processing software is introduced to perform marine physical correction and feature extraction on the target marine mapping data. The extracted features of the target marine mapping data include original data features, derived data features, and marine environmental features.

[0007] Furthermore, in a preferred embodiment of the present invention, the step of constructing a machine learning model for outlier detection by combining the characteristics of the target marine mapping data is defined as a target anomaly detection model, specifically as follows: Within data processing software, the Isolation Forest algorithm, the LOF local anomaly detection algorithm, and the LSTM algorithm are introduced, collectively referred to as machine learning algorithms. Feature sampling is performed on the target ocean mapping data features, which are then labeled as target ocean mapping data feature samples. Based on the machine learning algorithm, an initial machine learning model is constructed, and the target ocean mapping data feature samples are imported into the initial machine learning model. Within the initial machine learning model, the isolated forest algorithm is used to perform a global scan of the target ocean mapping data feature samples and analyze outliers in the target ocean mapping data feature samples. Outliers are target ocean mapping data feature samples that do not remain within the data feature standard range. A big data network is introduced, and a historical citation ranking of marine mapping data types is retrieved based on the big data network. The marine mapping data types with the highest standard citation count in the historical citation ranking of marine mapping data types are identified and marked as types to be selected. The feature samples of the target marine mapping data of the type to be selected are determined, and the feature density analysis of the feature samples of the target marine mapping data of the type to be selected is performed by the LOF local anomaly detection algorithm in the initial machine learning model. If the feature density is not maintained within the preset range, it is determined that the feature samples of the target marine mapping data of the type to be selected have local abnormal features; Based on the LSTM algorithm in the initial machine learning model, and combined with outliers in the target ocean mapping data feature samples and local anomaly features of the target ocean mapping data feature samples of the type to be selected, temporal correlation detection is performed on the target ocean mapping data features. The temporal correlation detection involves detecting the correlation between different outliers and local anomaly features in the target marine mapping data sample, and constructing a data topology map based on the correlation. By combining the data topology diagram, the initial machine learning model is trained to obtain a machine learning model for outlier detection, which is then labeled as the target anomaly detection model.

[0008] Furthermore, in a preferred embodiment of the present invention, the step of combining the data topology graph to train the initial machine learning model to obtain a machine learning model for outlier detection, which is then labeled as the target outlier detection model, specifically includes: Perform matrix transformation on the data topology graph and output the matrix data of the data topology graph; Based on the matrix data of the data topology graph, a dataset is constructed, which includes a training set and a validation set. The training set is then used to train the initial machine learning model. The model training process involves using Bayesian hyperparameter tuning to optimize the initial machine learning model using a dataset. Then, a validation set is used to verify the model against data omissions. If the optimized model is found to have data omissions, resampling is initiated, Bayesian optimization is performed again, and the target anomaly detection model is output.

[0009] Furthermore, in a preferred embodiment of the present invention, the step of using a target anomaly detection model to perform outlier detection and confidence assessment on all features of the target marine mapping data, and generating a feature quality control method based on the detection and assessment results, specifically includes: In the target anomaly detection model, all features of the target marine mapping data are input, and the anomaly status of all features of the target marine mapping data is determined through the target anomaly detection model. Among them, the abnormal states of all features of the target marine mapping data include the presence of feature outliers and the presence of local anomalous features; The control target anomaly detection model is connected to a big data network. Based on the big data network, a scoring table is specified for different combinations of outlier numbers and local anomaly numbers, and this table is labeled as the anomaly feature scoring table. Combining the anomaly feature scoring table, based on the anomaly status of all features of the target ocean mapping data, the feature score value of the target ocean mapping data is output, and based on the score value, the confidence level of the target ocean mapping data is output. A preset standard confidence level range is defined. If the confidence level of the target oceanographic mapping data remains within the standard confidence level range, it is determined that there are no outliers in the target oceanographic mapping data. If the confidence level of the target oceanographic mapping data does not remain within the standard confidence level range, then it is determined that there are outliers in the target oceanographic mapping data; When outliers exist in the target marine mapping data, feature quality control processing is performed on the target marine mapping data by combining the outlier states of all features of the target marine mapping data.

[0010] Furthermore, in a preferred embodiment of the present invention, the step of combining the abnormal states of all features of the target marine mapping data to perform feature quality control processing on the target marine mapping data specifically includes: Analyze the anomalies of all features of the target marine mapping data, calculate the deviation between the anomaly and normal states, and calibrate them as the deviation values ​​to be analyzed. Analyze the characteristics of target marine mapping data that exhibit abnormal conditions to determine whether the characteristics of the target marine mapping data are controllable. If not, the corresponding target marine mapping data features will be saved to the data processing software and shared with the management terminal of the vessel in the target sea area through the data processing software. If so, based on the abnormal state of all features of the target marine mapping data, locate the feature outliers and the locations where there are local abnormal features, mark them as abnormal locations, calculate the proportion of abnormal locations in all features, and output the spatial heat map of the target marine mapping data in combination with the deviation value to be analyzed. The spatial heatmap of the target ocean mapping data uses color gradients to display the abnormal distribution and abnormal deviations of different features of the target ocean mapping data. In a big data network, based on the spatial heat map of the target ocean mapping data, a correction scheme for the abnormal features of the target ocean mapping data is retrieved and output, thereby realizing the feature quality control processing of the target ocean mapping data and generating a quality control report.

[0011] A second aspect of this invention also provides a machine learning-based outlier detection and quality control system for marine mapping data. This system integrates a high-performance computing architecture and a bioinformatics storage module, including a non-volatile memory consisting of a DDR4 RDIMM memory module with ECC verification and an NVMe solid-state storage array using 3D NAND flash memory, as well as a multi-core processor based on the Zen4 microarchitecture. The memory contains a program for outlier detection and quality control, which includes an outlier detection and quality control engine. When this program is executed in parallel via a superscalar pipeline execution unit within the processor, the following steps are implemented: Collect marine mapping data and perform data preprocessing on the marine mapping data to obtain target marine mapping data, and extract the features of the target marine mapping data; By combining the characteristics of the target marine mapping data, a machine learning model for outlier detection is constructed and defined as the target anomaly detection model; An anomaly detection model is used to detect outliers and assess confidence levels for all features of target marine mapping data, and a feature quality control method is generated based on the detection and assessment results.

[0012] This invention addresses the technical deficiencies in the background technology and offers the following beneficial effects: It extracts features from collected marine mapping data to construct and train a model capable of detecting feature anomalies. This model then performs outlier detection and confidence assessment on all features of the marine mapping data. Simultaneously, it generates a feature quality control method based on the detection and assessment results. This invention provides underlying technical support for marine resource development, navigation safety, and scientific research through multi-dimensional feature depth mining, complex nonlinear relationship construction, and automated processing flow construction. It achieves goals such as improved detection efficiency, increased accuracy in complex anomaly identification, enhanced adaptability, and reduced false alarm rates under multi-source data fusion. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained from these drawings without creative effort.

[0014] Figure 1 A flowchart of a machine learning-based method for outlier detection and quality control in marine mapping data is shown. Figure 2 A flowchart illustrating the method for constructing a target anomaly detection model is shown. Figure 3This paper presents a program view of a machine learning-based outlier detection and quality control system for marine mapping data. Detailed Implementation

[0015] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0016] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0017] Figure 1 A flowchart illustrating a machine learning-based method for outlier detection and quality control in marine mapping data is provided, including the following steps: S102: Collect marine mapping data, perform data preprocessing on the marine mapping data to obtain target marine mapping data, and extract the features of the target marine mapping data. S104: Combining the characteristics of the target marine mapping data, a machine learning model for outlier detection is constructed and defined as the target anomaly detection model; S106: Using a target anomaly detection model, outlier detection and confidence assessment are performed on all features of the target marine mapping data, and a feature quality control method is generated based on the detection and assessment results.

[0018] Furthermore, in a preferred embodiment of the present invention, the process of collecting marine mapping data, preprocessing the marine mapping data to obtain target marine mapping data, and extracting features from the target marine mapping data specifically includes: The sea area to be surveyed is marked as the target sea area. Multi-source surveying equipment is deployed in the target sea area, and the multi-source surveying equipment is controlled to collect marine surveying data in real time in the target sea area to obtain multi-source real-time marine surveying data. At the same time, the multi-source real-time marine surveying data is correlated with latitude and longitude and with timestamps. Data cleaning processing is performed on multi-source real-time marine mapping data. The data cleaning processing includes interpolation filling processing for local missing ranges and range removal training processing for large missing ranges of multi-source real-time marine mapping data. At the same time, the data cleaning multi-source real-time marine mapping data is filtered by the moving median filtering algorithm to obtain the target marine mapping data. Data processing software is introduced to perform marine physical correction and feature extraction on the target marine mapping data. The extracted features of the target marine mapping data include original data features, derived data features, and marine environmental features.

[0019] It should be noted that marine mapping data exists within the ocean area and includes, but is not limited to, multibeam echo sounder data, side-scan sonar images, single-beam data, CTD data, sound velocity profiles, positioning data, attitude data, etc., encompassing seawater data and data from ships navigating the ocean. The data types are extremely diverse. Marine mapping data is collected by various acquisition devices, including but not limited to multibeam echo sounders and sonar. Correlating the latitude and longitude and timestamps of multi-source real-time marine mapping data aims to determine the acquisition location and time, and simultaneously identify the mutual influence between acquisition location and time, ensuring the rigor of the data analysis process. Since data acquisition may result in missing data, preprocessing is necessary to ensure the data is free of missing data and noise; this is achieved through filtering. The data possesses data characteristics, including raw data characteristics, derived data characteristics, and marine environmental characteristics. Raw data characteristics include, but are not limited to, ocean depth values, echo intensity, backscatter intensity, positioning accuracy factor, roll and pitch angles, and data acquisition time. Derived data characteristics include, but are not limited to, local depth gradient / rate of change, depth differences between adjacent survey lines / points, and statistical characteristics of echo intensity. Marine environmental characteristics include, but are limited to, seabed type characteristics and slope.

[0020] Furthermore, in a preferred embodiment of the present invention, the step of using a target anomaly detection model to perform outlier detection and confidence assessment on all features of the target marine mapping data, and generating a feature quality control method based on the detection and assessment results, specifically includes: In the target anomaly detection model, all features of the target marine mapping data are input, and the anomaly status of all features of the target marine mapping data is determined through the target anomaly detection model. Among them, the abnormal states of all features of the target marine mapping data include the presence of feature outliers and the presence of local anomalous features; The control target anomaly detection model is connected to a big data network. Based on the big data network, a scoring table is specified for different combinations of outlier numbers and local anomaly numbers, and this table is labeled as the anomaly feature scoring table. Combining the anomaly feature scoring table, based on the anomaly status of all features of the target ocean mapping data, the feature score value of the target ocean mapping data is output, and based on the score value, the confidence level of the target ocean mapping data is output. A preset standard confidence level range is defined. If the confidence level of the target oceanographic mapping data remains within the standard confidence level range, it is determined that there are no outliers in the target oceanographic mapping data. If the confidence level of the target oceanographic mapping data does not remain within the standard confidence level range, then it is determined that there are outliers in the target oceanographic mapping data; When outliers exist in the target marine mapping data, feature quality control processing is performed on the target marine mapping data by combining the outlier states of all features of the target marine mapping data.

[0021] It should be noted that marine mapping data includes various types of data features. All features are imported into the model to determine the presence of outliers and local anomalies, i.e., to identify outliers. If any are found, the anomalies need to be scored to calculate the confidence level and assess the security of the current marine mapping data. Different numbers of outliers combined with different local anomalies result in different feature scores, and therefore different confidence levels.

[0022] Furthermore, in a preferred embodiment of the present invention, the step of combining the abnormal states of all features of the target marine mapping data to perform feature quality control processing on the target marine mapping data specifically includes: Analyze the anomalies of all features of the target marine mapping data, calculate the deviation between the anomaly and normal states, and calibrate them as the deviation values ​​to be analyzed. Analyze the characteristics of target marine mapping data that exhibit abnormal conditions to determine whether the characteristics of the target marine mapping data are controllable. If not, the corresponding target marine mapping data features will be saved to the data processing software and shared with the management terminal of the vessel in the target sea area through the data processing software. If so, based on the abnormal state of all features of the target marine mapping data, locate the feature outliers and the locations where there are local abnormal features, mark them as abnormal locations, calculate the proportion of abnormal locations in all features, and output the spatial heat map of the target marine mapping data in combination with the deviation value to be analyzed. The spatial heatmap of the target ocean mapping data uses color gradients to display the abnormal distribution and abnormal deviations of different features of the target ocean mapping data. In a big data network, based on the spatial heat map of the target ocean mapping data, a correction scheme for the abnormal features of the target ocean mapping data is retrieved and output, thereby realizing the feature quality control processing of the target ocean mapping data and generating a quality control report.

[0023] It should be noted that, firstly, the deviation value between the abnormal state and the normal state is determined. This deviation value is considered as the value needed for data correction. If the data is controllable, for example, if the data is adjustable on the ship, including but not limited to the ship's current speed and displacement, then the mapping data is adjusted. Outliers and locations with local anomalies are identified and marked as anomalous locations. Combined with the deviation value to be analyzed and its proportion, a spatial heat map is generated. This spatial heat map is used for data correction. First, the data distribution is determined, i.e., the distribution of different types of collected data. Based on the distribution and the corresponding deviation value from the standard value, a correction scheme is retrieved from the big data network and output. This ensures the completion of quality control processing of the marine mapping data and generates a quality control report, achieving the purpose of visualization processing.

[0024] Figure 2 The flowchart illustrating the method for constructing a target anomaly detection model is shown, including the following steps: S202: Combining the characteristics of the target's marine mapping data, a machine learning model for outlier detection is constructed and defined as the target anomaly detection model; S204: Combine the data topology diagram to train the initial machine learning model, and obtain a machine learning model for outlier detection, which is then labeled as the target outlier detection model.

[0025] Furthermore, in a preferred embodiment of the present invention, the step of constructing a machine learning model for outlier detection by combining the characteristics of the target marine mapping data is defined as a target anomaly detection model, specifically as follows: Within data processing software, the Isolation Forest algorithm, the LOF local anomaly detection algorithm, and the LSTM algorithm are introduced, collectively referred to as machine learning algorithms. Feature sampling is performed on the target ocean mapping data features, which are then labeled as target ocean mapping data feature samples. Based on the machine learning algorithm, an initial machine learning model is constructed, and the target ocean mapping data feature samples are imported into the initial machine learning model. Within the initial machine learning model, the isolated forest algorithm is used to perform a global scan of the target ocean mapping data feature samples and analyze outliers in the target ocean mapping data feature samples. Outliers are target ocean mapping data feature samples that do not remain within the data feature standard range. A big data network is introduced, and a historical citation ranking of marine mapping data types is retrieved based on the big data network. The marine mapping data types with the highest standard citation count in the historical citation ranking of marine mapping data types are identified and marked as types to be selected. The feature samples of the target marine mapping data of the type to be selected are determined, and the feature density analysis of the feature samples of the target marine mapping data of the type to be selected is performed by the LOF local anomaly detection algorithm in the initial machine learning model. If the feature density is not maintained within the preset range, it is determined that the feature samples of the target marine mapping data of the type to be selected have local abnormal features; Based on the LSTM algorithm in the initial machine learning model, and combined with outliers in the target ocean mapping data feature samples and local anomaly features of the target ocean mapping data feature samples of the type to be selected, temporal correlation detection is performed on the target ocean mapping data features. The temporal correlation detection involves detecting the correlation between different outliers and local anomaly features in the target marine mapping data sample, and constructing a data topology map based on the correlation. By combining the data topology diagram, the initial machine learning model is trained to obtain a machine learning model for outlier detection, which is then labeled as the target anomaly detection model.

[0026] It is important to note that a machine learning model needs to be constructed to effectively identify anomalous features in marine mapping data and determine outliers. The machine learning model includes three training algorithms: global anomaly detection, local anomaly detection, and temporal correlation detection. The algorithms used are the Isolation Forest algorithm, the LOF local anomaly detection algorithm, and the LSTM algorithm, respectively. Global anomaly detection identifies outliers that significantly deviate from the main data set; local anomaly detection identifies data points with abnormal local density; and temporal correlation detection detects the correlation between different outliers and local anomalous features in the target marine mapping data sample, predicting whether abrupt changes in these features will occur. The three algorithms are combined to output a data topology map, which is used to train the model. Outliers and local anomalous features are the anomalous feature values ​​of the data, i.e., the output of the trained model. The model is then used for predictive analysis of all data obtained through sample analysis and training.

[0027] Furthermore, in a preferred embodiment of the present invention, the step of combining the data topology graph to train the initial machine learning model to obtain a machine learning model for outlier detection, which is then labeled as the target outlier detection model, specifically includes: Perform matrix transformation on the data topology graph and output the matrix data of the data topology graph; Based on the matrix data of the data topology graph, a dataset is constructed, which includes a training set and a validation set. The training set is then used to train the initial machine learning model. The model training process involves using Bayesian hyperparameter tuning to optimize the initial machine learning model using a dataset. Then, a validation set is used to verify the model against data omissions. If the optimized model is found to have data omissions, resampling is initiated, Bayesian optimization is performed again, and the target anomaly detection model is output.

[0028] It should be noted that after matrix transformation, the data topology graph can be used to access data within the graph to construct a dataset for model training. The training method is Bayesian hyperparameter tuning, which predicts and determines whether anomalous data features will appear based on the correlation between data points. However, data omissions are prone to occur during training, requiring continued sampling until all data is completely checked to obtain the target anomaly detection model.

[0029] like Figure 3 As shown, the second aspect of the present invention also provides a machine learning-based outlier detection and quality control system for marine mapping data. This outlier detection and quality control system integrates a high-performance computing architecture and a bioinformatics storage module, including a non-volatile memory consisting of a DDR4 RDIMM memory module with ECC verification and an NVMe solid-state storage array using 3D NAND flash memory, and a multi-core processor based on the Zen4 microarchitecture. The memory contains a program for outlier detection and quality control, which has an outlier detection and quality control engine. When the program is executed in parallel through a superscalar pipeline execution unit within the processor, the following steps are implemented: Collect marine mapping data and perform data preprocessing on the marine mapping data to obtain target marine mapping data, and extract the features of the target marine mapping data; By combining the characteristics of the target marine mapping data, a machine learning model for outlier detection is constructed and defined as the target anomaly detection model; An anomaly detection model is used to detect outliers and assess confidence levels for all features of target marine mapping data, and a feature quality control method is generated based on the detection and assessment results.

[0030] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for outlier detection and quality control in marine mapping data based on machine learning, characterized in that, Includes the following steps: Collect marine mapping data and perform data preprocessing on the marine mapping data to obtain target marine mapping data, and extract the features of the target marine mapping data; By combining the characteristics of the target marine mapping data, a machine learning model for outlier detection is constructed and defined as the target anomaly detection model; An anomaly detection model is used to detect outliers and assess confidence levels for all features of target marine mapping data, and a feature quality control method is generated based on the detection and assessment results.

2. The method for outlier detection and quality control of marine mapping data based on machine learning as described in claim 1, characterized in that, The process involves collecting marine mapping data, preprocessing the data to obtain target marine mapping data, and extracting features from the target marine mapping data. Specifically: The sea area to be surveyed is marked as the target sea area. Multi-source surveying equipment is deployed in the target sea area, and the multi-source surveying equipment is controlled to collect marine surveying data in real time in the target sea area to obtain multi-source real-time marine surveying data. At the same time, the multi-source real-time marine surveying data is correlated with latitude and longitude and with timestamps. Data cleaning processing is performed on multi-source real-time marine mapping data. The data cleaning processing includes interpolation filling processing for local missing ranges and range removal training processing for large missing ranges of multi-source real-time marine mapping data. At the same time, the data cleaning multi-source real-time marine mapping data is filtered by the moving median filtering algorithm to obtain the target marine mapping data. Data processing software is introduced to perform marine physical correction and feature extraction on the target marine mapping data. The extracted features of the target marine mapping data include original data features, derived data features, and marine environmental features.

3. The method for outlier detection and quality control of marine mapping data based on machine learning as described in claim 1, characterized in that, The aforementioned method combines the characteristics of target marine mapping data to construct a machine learning model for outlier detection, defined as the target anomaly detection model, specifically as follows: Within data processing software, the Isolation Forest algorithm, the LOF local anomaly detection algorithm, and the LSTM algorithm are introduced, collectively referred to as machine learning algorithms. Feature sampling is performed on the target ocean mapping data features, which are then labeled as target ocean mapping data feature samples. Based on the machine learning algorithm, an initial machine learning model is constructed, and the target ocean mapping data feature samples are imported into the initial machine learning model. Within the initial machine learning model, the isolated forest algorithm is used to perform a global scan of the target ocean mapping data feature samples and analyze outliers in the target ocean mapping data feature samples. Outliers are target ocean mapping data feature samples that do not remain within the data feature standard range. A big data network is introduced, and a historical citation ranking of marine mapping data types is retrieved based on the big data network. The marine mapping data types with the highest standard citation count in the historical citation ranking of marine mapping data types are identified and marked as types to be selected. The feature samples of the target marine mapping data of the type to be selected are determined, and the feature density analysis of the feature samples of the target marine mapping data of the type to be selected is performed by the LOF local anomaly detection algorithm in the initial machine learning model. If the feature density is not maintained within the preset range, it is determined that the feature samples of the target marine mapping data of the type to be selected have local abnormal features; Based on the LSTM algorithm in the initial machine learning model, and combined with outliers in the target ocean mapping data feature samples and local anomaly features of the target ocean mapping data feature samples of the type to be selected, temporal correlation detection is performed on the target ocean mapping data features. The temporal correlation detection involves detecting the correlation between different outliers and local anomaly features in the target marine mapping data sample, and constructing a data topology map based on the correlation. By combining the data topology diagram, the initial machine learning model is trained to obtain a machine learning model for outlier detection, which is then labeled as the target anomaly detection model.

4. The method for outlier detection and quality control of marine mapping data based on machine learning as described in claim 3, characterized in that, The process involves combining the data topology graph to train the initial machine learning model, resulting in a machine learning model for outlier detection, which is then labeled as the target outlier detection model. Specifically: Perform matrix transformation on the data topology graph and output the matrix data of the data topology graph; Based on the matrix data of the data topology graph, a dataset is constructed, which includes a training set and a validation set. The training set is then used to train the initial machine learning model. The model training process involves using Bayesian hyperparameter tuning to optimize the initial machine learning model using a dataset. Then, a validation set is used to verify the model against data omissions. If the optimized model is found to have data omissions, resampling is initiated, Bayesian optimization is performed again, and the target anomaly detection model is output.

5. The method for outlier detection and quality control of marine mapping data based on machine learning as described in claim 1, characterized in that, The method involves using a target anomaly detection model to detect outliers and assess confidence levels for all features of the target marine mapping data, and generating a feature quality control method based on the detection and assessment results. Specifically: In the target anomaly detection model, all features of the target marine mapping data are input, and the anomaly status of all features of the target marine mapping data is determined through the target anomaly detection model. Among them, the abnormal states of all features of the target marine mapping data include the presence of feature outliers and the presence of local anomalous features; The control target anomaly detection model is connected to a big data network. Based on the big data network, a scoring table is specified for different combinations of outlier numbers and local anomaly numbers, and this table is labeled as the anomaly feature scoring table. Combining the anomaly feature scoring table, based on the anomaly status of all features of the target ocean mapping data, the feature score value of the target ocean mapping data is output, and based on the score value, the confidence level of the target ocean mapping data is output. A preset standard confidence range is defined. If the confidence level of the target oceanographic mapping data remains within the standard confidence range, it is determined that there are no outliers in the target oceanographic mapping data. If the confidence level of the target oceanographic mapping data does not remain within the standard confidence level range, then it is determined that there are outliers in the target oceanographic mapping data; When outliers exist in the target marine mapping data, feature quality control processing is performed on the target marine mapping data by combining the outlier states of all features of the target marine mapping data.

6. The method for outlier detection and quality control of marine mapping data based on machine learning as described in claim 5, characterized in that, The process of combining the anomalies of all features of the target marine mapping data to perform feature quality control processing on the target marine mapping data specifically involves: Analyze the anomalies of all features of the target marine mapping data, calculate the deviation between the anomaly and normal states, and calibrate them as the deviation values ​​to be analyzed. Analyze the characteristics of target marine mapping data that exhibit abnormal conditions to determine whether the characteristics of the target marine mapping data are controllable. If not, the corresponding target marine mapping data features will be saved to the data processing software and shared with the management terminal of the vessel in the target sea area through the data processing software. If so, based on the abnormal state of all features of the target marine mapping data, locate the feature outliers and the locations where there are local abnormal features, mark them as abnormal locations, calculate the proportion of abnormal locations in all features, and output the spatial heat map of the target marine mapping data in combination with the deviation value to be analyzed. The spatial heatmap of the target ocean mapping data uses color gradients to display the abnormal distribution and abnormal deviations of different features of the target ocean mapping data. In a big data network, based on the spatial heat map of the target ocean mapping data, a correction scheme for the abnormal features of the target ocean mapping data is retrieved and output, thereby realizing the feature quality control processing of the target ocean mapping data and generating a quality control report.

7. A machine learning-based outlier detection and quality control system for marine mapping data, characterized in that, The outlier detection and quality control system integrates a high-performance computing architecture and a bioinformatics storage module, including a non-volatile memory consisting of a DDR4 RDIMM memory module with ECC verification and an NVMe solid-state storage array using 3D NAND flash memory, and a multi-core processor based on the Zen4 microarchitecture. The memory contains an outlier detection and quality control method program with an outlier detection and quality control engine. When the program is executed in parallel through the superscalar pipeline execution unit in the processor, it implements the outlier detection and quality control steps for marine mapping data as described in any one of claims 1-6.

Images