A multimodal data fusion method and tool for ship equipment

Abstract

The present invention belongs to the field of deep learning technology, and particularly relates to a multimodal data fusion method and tool for ship equipment, wherein the method comprises the following steps: obtaining the monitoring requirements of ship equipment, and obtaining various types of raw data according to the monitoring requirements; extracting data features from the raw data to obtain standardized data; fusing the standardized data and passing it into a first fusion level and a second fusion level to obtain corresponding time series features; obtaining a first fusion level prediction result in the first fusion level; obtaining a second fusion level prediction result in the second fusion level; and obtaining a final prediction result in a meta-learner based on a decision tree. The multimodal data fusion method for ship equipment provided by the present invention ensures the diversity and comprehensiveness of data and improves prediction performance and stability through multimodal information fusion, time series feature extraction, multi-level fusion and decision-making, and integrated learning.

Description

Technical Field

[0001] The present invention relates to the field of computers, and in particular to a multi-modal data fusion method and tool for ship equipment. Background Art

[0002] With the advent of the big data era and the development of information collection technology, the amount of data that needs to be processed is increasing rapidly, and the data types are becoming more and more diverse. Multimodal data provides the possibility of more comprehensive and efficient perception of the world around us, but its value density is low, and the semantic gap caused by differences in data meaning and the heterogeneous gap caused by differences in data types in multimodal data have brought new obstacles to data processing and data representation.

[0003] For the ship information management system, it is necessary to monitor the ship's location, operating status and surrounding environment at all times. Due to the excessive amount of data information, a large-scale data fusion method is needed to automatically associate, match and extract features of information data from all aspects through algorithms, and finally complete the information decision of the target object.

[0004] Traditional data fusion methods have problems such as high sensitivity in data preprocessing, the need for manual feature extraction, and insufficient data fusion. The above defects will lead to problems such as reduced data quality, incomplete features, and insufficient data fusion at the level of ship information management system, which will in turn affect the accuracy of the system and the reliability of decision-making. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a multimodal data fusion method and tool for ship equipment, so as to solve the technical problems of the above-mentioned traditional data fusion method at the level of ship information management system.

[0006] In a first aspect, an embodiment of the present invention provides a multimodal data fusion method for ship equipment, comprising the following steps:

[0007] Obtain monitoring requirements for ship equipment, and obtain various types of raw data according to the monitoring requirements; extract data features from the raw data in a preset manner to obtain various types of standardized data; fuse the various types of standardized data, and pass them into the preset first fusion level and second fusion level to obtain corresponding various types of time series features; in the first fusion level, fuse the various types of time series features to obtain fused features, and input the fused features into the fully connected layer and the Softmax layer to obtain a first fusion level prediction result; in the second fusion level, input the various types of time series features into the fully connected layer and the Softmax layer respectively to obtain multiple second fusion level intermediate prediction results, and perform weighted fusion on the multiple second fusion level intermediate prediction results to obtain a second fusion level prediction result; input the first fusion level prediction result and the second fusion level prediction result into a preset decision tree-based meta-learner and / or gradient boosting tree framework to obtain a final prediction result.

[0008] Preferably, extracting data features from the original data in a preset manner to obtain multiple types of standardized data includes the following steps: analyzing the type of the original data by manual screening and / or database matching, adding a source identifier X to the original data, and generating a type source table based on the source identifier X and the data type of the original data; dividing the data with the source identifier X into a training set, a validation set and a test set; and standardizing the training set, the validation set and the test set according to the type source table to obtain multiple types of standardized data.

[0009] Preferably, the classification training process of the preset first fusion level, the second fusion level, and the decision tree-based meta-learner and / or gradient boosting tree framework is implemented by the following steps: performing model training and verification on the first fusion level and the second fusion level according to the training set and the preset verification method to obtain training results and verification results; training the decision tree-based meta-learner and / or gradient boosting tree framework by taking the training results as input and the verification results as output.

[0010] Preferably, the first fusion level, the second fusion level, and the decision tree-based meta-learner and / or the gradient boosting tree framework are verified according to the verification set, and based on the verification results, the first fusion level, the second fusion level, and the decision tree-based meta-learner and / or the gradient boosting tree framework are structurally adjusted, hyperparameters are adjusted, and feature engineering is performed.

[0011] Preferably, the preset validation methods include: Hold-out Cross-Validation, K-fold Cross-Validation, Leave-One-Out Cross-Validation (LOOCV), Stratified Cross-Validation and Bootstrap.

[0012] Preferably, K-fold cross-validation is used for the classification training of the first fusion level, the second fusion level, and the meta-learner based on the decision tree and / or the gradient boosting tree framework, K=5, and the training is performed according to the following steps: S1: Divide the training set into 5 subsets according to the five-fold cross-validation method, and select one of the neural network stacked 1D-CNN or Wav2vec as the base learner; S2: Select one subset in turn as the test set of the base learner , use the other subsets as the training set of the base learner , output the test results of the training set , while for the test set Make predictions and output prediction results ; S3: loop S2 5 times to get the training set test results ( , , , , ) and the predicted results ( , , , , ), the test results are overlapped and merged vertically to obtain , taking the average of the prediction results to get ; S4: Execute S1-S3 on another base learner to obtain the results generated by the training set and ; S5: , Combined with the source identifier X to obtain a new sample training set ,Will , Merge to get a new test data set ,Will and As input and output of the decision tree based meta learner and / or gradient boosting tree framework, respectively, the decision tree based meta learner and / or gradient boosting tree framework is trained.

[0013] Preferably, the preset decision tree-based meta-learner and / or gradient boosting tree framework includes: LightGBM model, XGBoost model, CatBoost model, Neural Networks model, Random Forests algorithm and AdaBoost algorithm.

[0014] Preferably, a validation set is used to verify the first fusion level, the second fusion level, and the decision tree-based meta-learner and / or the gradient boosting tree framework. If the validation result meets the preset standard, the test set is input to obtain the final prediction result; if the validation result does not meet the preset standard, the parameters of the model are adjusted and the above steps are repeated.

[0015] In a second aspect, an embodiment of the present invention provides a multimodal data fusion tool for ship equipment, including:

[0016] Receiving module: used to obtain monitoring requirements and original data of ship equipment;

[0017] Data feature extraction module: used to extract data features from the original data;

[0018] Time series feature extraction module: used to fuse the standardized data of various types to obtain corresponding time series features of various types;

[0019] Fusion module: includes a first fusion layer, a second fusion layer, and a decision tree-based meta-learner and / or a gradient boosting tree framework.

[0020] In a third aspect, an embodiment of the present invention provides a program product, which includes computer program instructions, and the computer program instructions implement the steps of the above method when executed.

[0021] Compared with the prior art, the multimodal data fusion method for ship equipment provided by the present invention has the following beneficial effects:

[0022] The embodiment of the present invention provides the following steps: obtaining monitoring requirements of ship equipment, and obtaining various types of original data according to the monitoring requirements; extracting data features from the original data in a preset manner to obtain various types of standardized data; fusing the various types of standardized data, and passing them into a preset first fusion layer and a second fusion layer to obtain corresponding various types of time series features; in the first fusion layer, fusing the various types of time series features to obtain fused features, and inputting the fused features into a fully connected layer and a Softmax layer to obtain a first fusion layer prediction result; in the second fusion layer, inputting the various types of time series features into the fully connected layer and the Softmax layer respectively to obtain multiple second fusion layer intermediate prediction results, and weightedly fusing the multiple second fusion layer intermediate prediction results to obtain a second fusion layer prediction result; inputting the first fusion layer prediction result and the second fusion layer prediction result into a preset decision tree-based meta-learner and / or gradient boosting tree framework to obtain a final prediction result.

[0023] Compared with traditional data fusion methods, this scheme has at least the following beneficial effects:

[0024] Compared with the traditional fusion method, the above scheme has the following beneficial effects:

[0025] 1. Improve monitoring accuracy: By acquiring multiple types of raw data and extracting data features, the above solution can obtain more comprehensive information about ship equipment. By integrating multiple types of standardized data, the characteristics and advantages of multiple information sources can be comprehensively utilized to improve monitoring accuracy. This makes the monitoring results of ship equipment more accurate and reliable.

[0026] 2. More comprehensive consideration of time series features: The above scheme considers multiple types of time series features at both the first and second fusion levels, and even at different fusion levels, there is a more comprehensive consideration of time series features. Time series features can better reflect the dynamic changes and trends of equipment, which is of great significance for the monitoring and prediction of ship equipment. This helps to improve the accurate understanding and timely prediction of equipment status.

[0027] 3. Multi-level fusion provides flexibility: The first and second fusion levels in the above scheme provide multi-level fusion processing, allowing the fusion of features and decisions at different levels. This flexibility allows the system to comprehensively consider multiple types of features and information in both global and local scopes. This helps to improve the system's comprehensive prediction and decision-making capabilities for ship equipment.

[0028] 4. Introduction of meta-learning: The introduction of decision tree-based meta-learners and / or gradient boosting tree frameworks in the solution can further learn and integrate prediction results at different levels, which helps to improve the accuracy and stability of the overall prediction results. In the field of marine equipment, hierarchical learning for different features and data types can more accurately infer faults and reduce false positives and negatives.

[0029] In summary, compared with the traditional fusion method, the above scheme has the beneficial effects of integrating multiple data, emphasizing time series characteristics, deep learning structure fusion, introducing meta-learning, and improving accuracy and generalization ability in the application field of ship equipment. It can effectively improve the accuracy and reliability of ship equipment monitoring, fault detection and prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solution of the embodiment of the present invention, the drawings required for use in the embodiment of the present invention will be briefly introduced below. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work, and these are all within the protection scope of the present invention.

[0031] Figure 1 Schematic diagram of the process of the multimodal data fusion method for ship equipment provided by the first embodiment of the present invention Figure 1 .

[0032] Figure 2 A schematic diagram of a type source table provided in accordance with the first embodiment of the present invention.

[0033] Figure 3 Schematic diagram of the process of the multimodal data fusion method for ship equipment provided by the first embodiment of the present invention Figure 2 .

[0034] Figure 4 A schematic diagram of the structure of a multimodal data fusion tool for ship equipment provided by the second embodiment of the present invention.

[0035] Figure 5 A schematic diagram of the structure of a program product provided in the fourth embodiment of the present invention.

[0036] Figure 6 A schematic structural diagram of an electronic device provided in accordance with a fifth embodiment of the present invention.

[0037] Description of Figure Numbers:

[0038] 1. Multimodal data fusion method for ship equipment; 2. Multimodal data fusion tool for ship equipment; 3. Program product; 4. Electronic equipment;

[0039] 20. Receiving module; 21. Data feature extraction module; 22. Time series feature extraction module; 23. Fusion module; 40. Processor; 41. Memory; 42. Bus; 42. Communication interface. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and implementation examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0041] See also Figure 1 The first embodiment of the present invention provides a multimodal data fusion method 1 for ship equipment, which has the following beneficial effects:

[0042] The embodiment of the present invention provides the following steps: obtaining the monitoring requirements of ship equipment, and obtaining various types of original data according to the monitoring requirements; extracting data features from the original data in a preset manner to obtain various types of standardized data; fusing the various types of standardized data, and passing them into the preset first fusion level and second fusion level to obtain the corresponding various types of time series features; in the first fusion level, fusing the various types of time series features to obtain fused features, and inputting the fused features into the fully connected layer and the Softmax layer to obtain the first fusion level prediction results; in the second fusion level, inputting the various types of time series features into the fully connected layer and the Softmax layer respectively to obtain multiple second fusion level intermediate prediction results, and weightedly fusing the multiple second fusion level intermediate prediction results to obtain the second fusion level prediction results; using the preset decision tree-based meta-learner and / or gradient boosting tree framework to perform integrated classification learning on the first fusion level prediction results and the second fusion level prediction results to obtain the final prediction results.

[0043] Compared with traditional data fusion methods, this scheme has at least the following beneficial effects:

[0044] Compared with the traditional fusion method, the above scheme has the following beneficial effects:

[0045] 1. Improve monitoring accuracy: By acquiring multiple types of raw data and extracting data features, the above solution can obtain more comprehensive information about ship equipment. By integrating multiple types of standardized data, the characteristics and advantages of multiple information sources can be comprehensively utilized to improve monitoring accuracy. This makes the monitoring results of ship equipment more accurate and reliable.

[0046] 2. More comprehensive consideration of time series features: The above scheme considers multiple types of time series features at both the first and second fusion levels, and even at different fusion levels, there is a more comprehensive consideration of time series features. Time series features can better reflect the dynamic changes and trends of equipment, which is of great significance for the monitoring and prediction of ship equipment. This helps to improve the accurate understanding and timely prediction of equipment status.

[0047] 3. Multi-level fusion provides flexibility: The first and second fusion levels in the above scheme provide multi-level fusion processing, allowing the fusion of features and decisions at different levels. This flexibility allows the system to comprehensively consider multiple types of features and information in both global and local scopes. This helps to improve the system's comprehensive prediction and decision-making capabilities for ship equipment.

[0048] 4. Introduction of meta-learning: The introduction of decision tree-based meta-learners and / or gradient boosting tree frameworks in the solution can further learn and integrate prediction results at different levels, which helps to improve the accuracy and stability of the overall prediction results. In the field of marine equipment, hierarchical learning for different features and data types can more accurately infer faults and reduce false positives and negatives.

[0049] In summary, compared with the traditional fusion method, the above scheme has the beneficial effects of integrating multiple data, emphasizing time series characteristics, deep learning structure fusion, introducing meta-learning, and improving accuracy and generalization ability in the application field of ship equipment. It can effectively improve the accuracy and reliability of ship equipment monitoring, fault detection and prediction.

[0050] In some embodiments, extracting data features from raw data in a preset manner to obtain multiple types of standardized data includes the following steps: analyzing the type of raw data by manual screening and / or database matching, adding a source identifier X to the raw data, and generating a type source table based on the source identifier X and the data type of the raw data; dividing the data with the source identifier X into a training set, a validation set, and a test set; and standardizing the training set, the validation set, and the test set according to the type source table to obtain multiple types of standardized data.

[0051] Specifically:

[0052] 1. Data type analysis: Through manual screening and database matching, we identified the vibration, temperature, and humidity data required for ship equipment monitoring.

[0053] Furthermore, for different detection requirements, such as the 100th position, only relevant data adjacent to the current 100th position is selected.

[0054] 2. Add source identifier: In order to distinguish data from different sources, we add the corresponding source identifier X to each raw data. For example, add the source identifier "vibration" to vibration data, add the source identifier "temperature" to temperature data, and add the source identifier "humidity" to humidity data.

[0055] 3. Type source table: Generate a type source table based on the source identifier X and data type of the original data. This table records the source identifier corresponding to each data type for subsequent data segmentation and standardization. Figure 2 As shown;

[0056] 4. Data segmentation: According to the source identifier in the type source table, the data with source identifier X is divided into training set, validation set and test set. For example, vibration data is extracted from the original data and marked as "vibration". The data marked with "vibration" is divided into training set, validation set and test set.

[0057] Furthermore, the data with source identifier X is divided into a training set, a validation set, and a test set in a ratio of 8:1:1.

[0058] 5. Data standardization: According to the type source table, the data in the training set, validation set, and test set are standardized, and the data files in TXT format are converted into CTF format suitable for learning to obtain various types of standardized data. For each data type's source identifier, we select the corresponding standardization method for data processing. For example, for vibration data, we can perform mean variance processing; for temperature data, we can perform minimum-maximum normalization; for humidity data, we can perform z-score standardization.

[0059] Through the above steps, we have obtained standardized data of various types, including standardized vibration data, temperature data and humidity data. These standardized data will be used in the subsequent feature extraction and fusion steps to further analyze the status and behavior of ship equipment.

[0060] It can be understood that the above solution has at least the following beneficial effects:

[0061] The above scheme has the following beneficial effects:

[0062] 1. Automatic identification of data types: By analyzing the types of raw data through manual screening and / or database matching, different types of data can be automatically identified. This helps to accurately classify and process data, and provides a basis for subsequent feature extraction and fusion.

[0063] 2. Standardized data processing: By adding source identifier X to the original data and generating a type source table based on the source identifier X and the data type, different original data can be flexibly divided and processed, including dividing the data into training sets, validation sets, and test sets. By standardizing the training set, validation set, and test set through the type source table, the feature representation of different types of data can be unified, improving the comparability and interpretability of the data.

[0064] 3. Multiple types of standardized data: The above scheme can generate multiple types of standardized data, that is, data with different feature representations based on different data types. This helps to better utilize information from different types of data and improve the diversity and comprehensiveness of data.

[0065] 4. Targeted training and validation: By dividing the data into training sets, validation sets, and test sets, and standardizing them according to the type source table, targeted model training and validation can be performed according to actual needs and specific tasks. This helps improve the generalization ability and prediction performance of the model and reduce overfitting and generalization errors.

[0066] In summary, the above solution can improve the accuracy, flexibility and diversity of data processing by automatically identifying data types, standardizing data processing, generating various types of standardized data, and targeted training and verification, providing a better data foundation for subsequent feature extraction and fusion.

[0067] In some embodiments, the classification training process of the preset first fusion level, the second fusion level, and the decision tree-based meta-learner and / or the gradient boosting tree framework is implemented by the following steps:

[0068] Perform model training and verification on the first fusion layer and the second fusion layer according to the training set and the preset verification method to obtain training results and verification results;

[0069] A decision tree based meta-learner and / or a gradient boosted tree framework is trained with the training results as input and the validation results as output.

[0070] It can be understood that the above solution has at least the following beneficial effects:

[0071] 1. Improve the accuracy of classification models: By using the first and second fusion layers for model training and validation, the accuracy and generalization ability of the classification model can be improved.

[0072] 2. Enhance the robustness of the model: By training with a meta-learner of decision trees and / or a gradient boosted tree framework, the robustness of the model can be enhanced so that it can adapt to different data distributions and changes.

[0073] 3. Reduce the risk of overfitting: By using the preset verification method to verify the model, the overfitting problem can be discovered and solved in time, and the generalization ability of the model can be improved.

[0074] 4. Improve the interpretability of the model: The decision tree and gradient boosting tree frameworks are algorithms with strong interpretability. By using the training results as input, these algorithms can be further trained to obtain more interpretable models.

[0075] In general, this scheme can improve the accuracy, robustness and interpretability of the classification model by fusing and combining decision tree meta-learners and gradient boosting tree frameworks at multiple levels, thereby better coping with classification problems.

[0076] In some embodiments, the first fusion layer, the second fusion layer, and the decision tree-based meta-learner and / or the gradient boosting tree framework are verified according to the verification set, and the first fusion layer, the second fusion layer, and the decision tree-based meta-learner and / or the gradient boosting tree framework are structurally adjusted, hyperparameters are adjusted, and feature engineering is performed based on the verification results.

[0077] Specifically:

[0078] Suppose we want to solve an image classification problem, the specific implementation is as follows:

[0079] 1. Model training and validation of the first fusion layer and the second fusion layer: First, we train the models of the first fusion layer and the second fusion layer based on the training set data, and validate them using a preset validation method. For example, we can divide the training set into a training set and a validation set, train the model on the training set, and validate the performance on the validation set. Based on the validation results, we can evaluate the accuracy and generalization ability of the model.

[0080] 2. Training of decision tree-based meta-learners and / or gradient boosted tree frameworks: Based on the training results of the first and second fusion levels as input and the validation results as output, we can use decision tree meta-learners and / or gradient boosted tree frameworks as classification models for training. These algorithms will adjust and optimize the model according to the relationship between input and output during the training process.

[0081] 3. Structural adjustment, hyperparameter adjustment, and feature engineering: By analyzing the validation results of the first fusion layer, the second fusion layer, and the decision tree-based meta-learner and / or gradient boosting tree framework, we can further optimize the structural adjustment, hyperparameter adjustment, and feature engineering. For example, we can try to adjust the model's hyperparameters such as the number of layers, the number of nodes, and the learning rate, as well as perform feature engineering operations on the image, such as scale normalization, feature selection, etc. Through continuous adjustment and optimization, we can improve the performance and accuracy of the model.

[0082] In general, this scheme can optimize the structure and parameters of the model and improve the accuracy and generalization ability of the model in image classification problems by verifying and adjusting the first fusion level, the second fusion level, and the decision tree-based meta-learner and / or gradient boosting tree framework, as well as performing hyperparameter adjustment and feature engineering.

[0083] It can be understood that the above solution has at least the following beneficial effects:

[0084] 1. Optimize model structure and parameters: By adjusting the structure and hyperparameters of the first fusion layer, the second fusion layer, and the decision tree-based meta-learner and / or gradient boosting tree framework based on the validation results, the performance of the model can be optimized and the accuracy and generalization ability of the classification model can be improved.

[0085] 2. Improve feature representation capabilities: By adjusting feature engineering, modifying, combining or selecting features based on the verification results of the verification set, the model's ability to express data can be improved, thereby improving the accuracy of the model.

[0086] 3. Reduce the risk of overfitting: According to the verification results, combined with structural adjustment and hyperparameter adjustment, the risk of overfitting of the model can be reduced and the generalization ability of the model can be improved.

[0087] 4. Improve model interpretability: Through feature engineering adjustments, the interpretability of the model can be further improved, making the model output easier to understand and explain.

[0088] In general, this solution can optimize the performance of the model, improve the accuracy, generalization ability and interpretability of the model by adjusting the model structure, hyperparameters and feature engineering based on the verification results, so as to better cope with classification problems.

[0089] Optionally, the preset verification method includes:

[0090] 1. Hold-out Cross-Validation: Divide the data set into two parts, one for training the model and the other for validating the model. Usually the data set is divided into training set and validation set according to a certain ratio (such as 70% and 30%). Hold-out cross-validation is suitable for situations with large data volume and limited computing resources, but since only one division is performed, the evaluation results of the model may fluctuate to a certain extent.

[0091] 2. K-fold Cross-Validation: Divide the data set into K subsets, select K-1 subsets as training sets each time, and the remaining subset as validation set, and perform K training and validation. Finally, take the average of the results of each validation as the performance indicator of the model. K-fold cross-validation can make better use of data and reduce the deviation between the training set and the validation set. The commonly used K value is generally 5 or 10.

[0092] 3. Leave-One-Out Cross-Validation (LOOCV): Each sample in the data set is used as a validation set, and the other samples are used as training sets, and training and validation are performed n times (n is the number of samples in the data set). LOOCV is a special K-fold cross-validation, which is suitable for small data sets and can reduce the deviation between the training set and the validation set. However, it has a high computational cost and high time complexity.

[0093] 4. Stratified Cross-Validation: When dividing the training set and the validation set, sampling is performed according to the proportion of the category distribution to keep the proportion of each category in the training set and the validation set similar. Stratified cross-validation is suitable for situations where the categories in the data set are unbalanced, and can better evaluate the performance of the model on each category.

[0094] 5. Bootstrap: By sampling from the original data set with replacement, a training set and a holdout set are constructed. The bootstrap method can form multiple training sets and validation sets through multiple sampling. Due to the presence of repeated samples during sampling, the training set usually contains about 63.2% of the original samples, and the holdout set contains samples that did not appear in the training set. The bootstrap method can make better use of data, but the computational cost is higher.

[0095] Preferably, K-fold Cross-Validation is used to train the model.

[0096] Specifically, K-fold cross-validation is used for the classification training of the first fusion level, the second fusion level, and the decision tree-based meta-learner and / or the gradient boosting tree framework, with K=5, and the training is performed according to the following steps:

[0097] S1: Divide the training set into 5 subsets using the 5-fold cross-validation method, and select one of the neural networks stacked 1D-CNN or Wav2vec as the base learner;

[0098] S2: Select a subset in turn as the test set of the base learner , use other subsets as training sets for base learners , output the test results of the training set , while for the test set Make predictions and output prediction results ;

[0099] S3: loop S2 5 times to get the training set test results ( , , , , ) and the predicted results ( , , , , ), the test results are overlapped and merged vertically to obtain , taking the average of the prediction results to get ;

[0100] S4: Execute S1-S3 on another base learner to obtain the results generated by the training set and ;

[0101] S5: , Combined with the source identifier X to obtain a new sample training set ,Will , Merge to get a new test data set ,Will and The decision tree based meta-learner and / or the gradient boosted tree framework are trained as input and output of the decision tree based meta-learner and / or the gradient boosted tree framework, respectively.

[0102] It should be understood that taking K=5 is based on the following considerations:

[0103] 1. Resource utilization and computational efficiency: A smaller K value (such as K=5) can make better use of the available data and use as much data as possible for training and validation. At the same time, training and validation with a smaller K value can also reduce the consumption of computing resources, especially when the data set is large.

[0104] 2. Distribution of data samples: In the application field of ship equipment, data samples may have large variability and uncertainty, such as temperature, route, and humidity. In this case, dividing the data set into smaller K parts can better capture the instability and diversity of the data.

[0105] 3. Trust and stability: Generally speaking, the larger the value of K, the more accurate and stable the performance evaluation of the model will be, but it will also increase the computational complexity and time cost. A smaller value of K (such as K=5) can also provide a certain degree of trust and stability in the model performance.

[0106] However, in some cases, choosing other K values ​​may also have its own advantages. For example, for smaller datasets, larger K values ​​may better evaluate the generalization ability of the model. For unbalanced datasets, choosing other K values ​​may be more helpful for more accurate evaluation of the performance of minority classes. Therefore, the value of K can be flexibly adjusted according to the size of the dataset to better evaluate the generalization ability of the model.

[0107] It can be understood that the above solution has at least the following beneficial effects:

[0108] 1. Provide more accurate model evaluation: By dividing the training set into five mutually exclusive subsets, using four of them for training and the remaining one for validation, a more accurate model performance evaluation can be obtained. Each subset has the opportunity to serve as a validation set, which can better evaluate the performance of the model on different data.

[0109] 2. Reduce the deviation between the training set and the validation set: Five-fold cross validation can be used to train and evaluate the model on multiple training sets and validation sets, thereby reducing the deviation between the training set and the validation set. This helps to better estimate the generalization ability of the model and improve the performance of the model on new data.

[0110] 3. Provide a reference for model stability: Through multiple training and validation, and finally averaging the results, a relatively stable model performance evaluation can be obtained. This can reduce the fluctuation of evaluation results caused by the randomness of the data and provide a more reliable reference for model performance.

[0111] 4. Optimize parameter selection and model tuning: Through five-fold cross validation, you can train and validate on different parameter combinations, which helps select the best parameter combination and tune the model. By comparing the performance on different validation sets, you can better understand the performance of the model in different situations and optimize the model performance.

[0112] 5. Higher data utilization: Five-fold cross validation can divide the dataset into five parts, each of which has the opportunity to become a validation set, thereby making better use of the data for model training and evaluation. This is especially beneficial when data is scarce.

[0113] In summary, five-fold cross validation has the advantages of providing accurate model evaluation, reducing the deviation between training and validation sets, providing stability reference, optimizing parameter selection and model tuning, and improving data utilization.

[0114] Optionally, the preset decision tree based meta-learner and / or gradient boosted tree framework includes:

[0115] 1. LightGBM model:

[0116] LightGBM is a gradient boosting framework that uses a histogram-based decision tree learning algorithm. Compared with traditional gradient boosting frameworks, LightGBM has faster training speed and higher efficiency because it uses a histogram-based method when splitting nodes and can handle large-scale data and high-dimensional features. The model performs well when processing large-scale data and high-dimensional features.

[0117] 2. XGBoost Model:

[0118] XGBoost is a gradient boosting algorithm based on decision trees. It is very powerful in processing structured data and feature engineering. XGBoost trains multiple decision tree models through an incremental approach and continuously improves model performance through gradient boosting. It has been proven to achieve significant results in many Kaggle competitions.

[0119] 3. CatBoost Model:

[0120] CatBoost is a gradient boosting framework designed for processing categorical features. It has built-in capabilities for processing categorical features, can automatically handle missing values, and can handle large-scale data during training. CatBoost performs well when processing data with a large number of categorical features.

[0121] 4. Neural Networks model:

[0122] A neural network is a machine learning model inspired by the nervous system. It consists of multiple nodes (neurons) in a multi-layer structure and can perform tasks such as classification and regression by learning complex patterns of input data. Deep neural network models, such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs), have performed well in areas such as image and natural language processing.

[0123] 5. Random Forests algorithm:

[0124] Random forest is an ensemble learning algorithm consisting of multiple decision trees. During the training process, each tree is trained by randomly sampling the original feature samples, thereby achieving the effect of reducing variance and preventing overfitting. Random forest performs well on classification and regression problems and has good interpretability of feature importance.

[0125] 6. AdaBoost Algorithm:

[0126] AdaBoost is an ensemble learning algorithm that trains a series of weak classifiers by gradually changing the distribution of training data and combines them into a strong classifier. In each round of training, AdaBoost adjusts the data distribution so that previously misclassified samples receive more attention in the next round of training. AdaBoost is robust to noisy data and can also avoid overfitting.

[0127] Preferably, the LightGBM model is used as the dataset and training object.

[0128] In the field of marine equipment applications, the application and advantages of the LightGBM model as a gradient boosting tree framework are mainly reflected in the following aspects:

[0129] 1. Fault diagnosis and prediction: Ship equipment usually contains a large number of sensors and monitoring devices to monitor the status of mechanical components and systems. The LightGBM model can discover implicit patterns and trends in these data, thereby enabling rapid diagnosis and prediction of equipment failures. It can help ship operators better understand the operating status of equipment, promptly detect potential failure risks, reduce maintenance costs and avoid equipment downtime.

[0130] 2. Energy saving and efficiency optimization: The LightGBM model can analyze the operating data of ship equipment, identify situations where energy utilization is low and efficiency is low, and make relevant optimization suggestions. By analyzing the data of ship equipment, LightGBM can help ship managers optimize the operation mode of ship equipment, improve fuel utilization, and reduce operating costs.

[0131] 3. Data-driven preventive maintenance The LightGBM model can use the historical data and real-time monitoring data of ship equipment to perform preventive maintenance of equipment. It can predict the maintenance needs of equipment based on the equipment's usage and performance history, thereby helping ship maintenance personnel to develop reasonable maintenance plans and reduce the risk of sudden equipment failures.

[0132] 4. Multidimensional data analysis: The data generated by ship equipment is usually multidimensional and heterogeneous, including sensor data, working parameters, operating status, etc. The LightGBM model performs well in processing large-scale multidimensional data, and can efficiently mine the inherent associations and laws of the data, and conduct multi-angle and comprehensive analysis of the status of ship equipment.

[0133] Advantages: In the field of marine equipment applications, the advantages of the LightGBM model as a gradient boosting tree framework mainly include:

[0134] 1. Efficiency: The LightGBM model has fast training and prediction speeds, is suitable for processing large-scale data and high-dimensional features, and can complete complex data mining and analysis tasks in a shorter time.

[0135] 2. Accuracy: The LightGBM model performs well in processing structured data and high-dimensional features, and can more accurately discover patterns and laws in the data, providing accurate decision support for fault diagnosis, prediction and maintenance of ship equipment.

[0136] 3. Robustness: The LightGBM model has good robustness and a certain tolerance for noisy data and missing data. It can run stably and produce reliable results in a complex ship equipment environment.

[0137] In the field of marine equipment applications, the efficiency, accuracy and robustness of the LightGBM model make it an ideal tool to help ship operators and maintenance personnel better manage and maintain marine equipment and improve the reliability and availability of equipment.

[0138] Specifically, please combine Figure 1 and Figure 3 ,The multi-level fusion process provided in this embodiment mainly consists of three parts: Early fusion, Late fusion and Hybrid fusion;

[0139] Early fusion is divided into three steps: Feature extraction uses convolutional neural network (CNN) and bidirectional long short-term memory model (BiLSTM) to extract time series features from raw data, Feature fusion uses fusion module to fuse features from different sources, and Classification inputs the fused features into the fully connected layer and softmax layer to obtain prediction results; Late fusion is similar to Early fusion and is also divided into three steps. The difference is the fusion module. In Late fusion, Decision fusion replaces Feature fusion in Early fusion, that is, in Early fusion, time series features are fused, while in Late fusion, different types of prediction results are fused, and Stacking ensemble learning and classification are performed; then in Hybrid fusion, Light Gradient Boosting Machine (Lightgbm) is used to obtain the final prediction result.

[0140] Through the above process, the multi-level fusion method provided in this embodiment uses Early fusion and Late fusion to extract features from data of different dimensions respectively, and then uses the Lightgbm model in Hybrid fusion to integrate and optimize these prediction results to obtain a more accurate and reliable final prediction result.

[0141] In some embodiments, the first fusion level, the second fusion level, and the decision tree-based meta-learner and / or the gradient boosting tree framework are verified using the verification set separated in the data feature extraction step. If the verification result meets the preset standard, the test set separated in the data feature extraction step is input to obtain the final prediction result.

[0142] If the verification result does not meet the preset standard, adjust the parameters of the model and repeat the above steps.

[0143] It can be understood that the above solution has at least the following beneficial effects:

[0144] 1. Model validation and parameter adjustment: By validating the model on the validation set separated in the data feature extraction step, the generalization ability of the model can be evaluated and the model parameters can be adjusted in time. This helps to avoid overfitting or underfitting and improve model performance.

[0145] 2. Improve model stability: By validating the model on the validation set, the model can be made more robust, reduce dependence on specific data features, and improve the generalization ability of the model.

[0146] 3. Avoid data leakage: Limiting the use of the test set to cases where the verification results meet the preset standards can avoid data leakage when adjusting model parameters, thereby ensuring the independence and fairness of the test set.

[0147] 4. Effective use of data: By validating the model on the validation set split out in the feature extraction step, you can make more effective use of the data and improve the training efficiency and quality of the model.

[0148] In summary, this scheme can improve the stability and generalization ability of the model by verifying and adjusting model parameters and effectively utilizing the test set, and ultimately obtain more reliable prediction results.

[0149] Specifically, the model can be evaluated by the following embodiments:

[0150] First, load the trained multi-level fusion model with high accuracy, then iteratively input the test data, record the true type corresponding to each annotation in the test data, the obtained type, and the type inferred by this scheme, and finally perform statistics to obtain the accuracy of the fusion model. At the same time, record the number of ship equipment states and the number of inconsistent type inferences, and finally count the inconsistencies.

[0151] Accuracy: , where tp represents the number of inferred ship equipment states that are consistent with the actual states, and fp represents the number of inferred states that are inconsistent with the actual states.

[0152] Inconsistency: , where inconsistency represents the number of inconsistent state inferences in ship equipment that occur multiple times, and multi_occurences represents the number of ship equipment states that occur multiple times.

[0153] The generalization ability of the model is evaluated based on accuracy and inconsistency.

[0154] See also Figure 4 The second embodiment of the present invention further provides a multimodal data fusion tool 2 for ship equipment, which is used to implement the multimodal data fusion method for ship equipment. The tool 2 includes:

[0155] Receiving module 20: used to obtain monitoring requirements and original data of ship equipment;

[0156] Data feature extraction module 21: used to extract data features from original data;

[0157] Time series feature extraction module 22: used to fuse multiple types of standardized data to obtain corresponding multiple types of time series features;

[0158] Fusion module 23: includes a first fusion level, a second fusion level, and a meta-learner based on a decision tree and / or a gradient boosting tree framework.

[0159] It should be noted that, although several modules or units for executing the process are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0160] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0161] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0162] The third embodiment of the present invention further provides a readable storage medium on which computer program instructions are stored. When the computer program instructions are executed, the steps of the above method are implemented.

[0163] In some possible embodiments, various aspects of the present disclosure may also be implemented in the form of a program product, which includes program code. When the program product runs on a terminal device, the program code is used to enable the terminal device to execute the steps described in the above “” section of this specification according to various exemplary embodiments of the present disclosure.

[0164] See also Figure 5 The fourth embodiment of the present invention further provides a program product 3, which includes computer program instructions 30, and when the computer program instructions are executed, the steps of the above method are implemented.

[0165] See also Figure 6 The fifth embodiment of the present invention further provides an electronic device 4. Specifically, the electronic device 4 includes a processor 40 and a memory 41. The memory 41 stores a computer program, and when the computer program is run by the processor, it executes any method of the above implementation methods.

[0166] Furthermore, the processor 40 , the communication interface 43 and the memory 41 are connected via a bus 42 ; the processor 40 is used to execute executable modules stored in the memory 41 , such as computer programs.

[0167] The memory 41 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 43 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0168] The bus 42 may be an ISA bus, a PCI bus, or an EISA bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0169] Among them, the memory 41 is used to store programs, and the processor 40 executes the program after receiving the execution instruction. The method executed by the flow process definition tool disclosed in any embodiment of the above-mentioned embodiment of the present invention can be applied to the processor 40 or implemented by the processor 40.

[0170] The processor 40 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 40. The above processor 40 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiment of the present invention can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 41, and the processor 40 reads the information in the memory 41 and completes the steps of the above method in combination with its hardware.

[0171] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, electronic device, or network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0172] It should be noted that similar numbers and letters represent similar items in the accompanying drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0173] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal data fusion method for ship equipment, characterized in that: The following steps are involved: Obtain monitoring requirements of ship equipment, and obtain various types of raw data according to the monitoring requirements; Extracting data features from the original data in a preset manner to obtain various types of standardized data; The standardized data of various types are fused and transmitted to a preset first fusion level and a second fusion level to obtain corresponding time series features of various types; In the first fusion level, the multiple types of time series features are fused to obtain fused features, and the fused features are input into the fully connected layer and the Softmax layer to obtain the prediction result of the first fusion level; In the second fusion level, the multiple types of temporal features are respectively input into a fully connected layer and a Softmax layer to obtain a plurality of second fusion level intermediate prediction results, and the plurality of second fusion level intermediate prediction results are weightedly fused to obtain a second fusion level prediction result; The first fusion level prediction result and the second fusion level prediction result are input into a preset decision tree-based meta-learner and / or gradient boosting tree framework to obtain a final prediction result.

2. The multimodal data fusion method for ship equipment according to claim 1, characterized in that: Extracting data features from the raw data in a preset manner to obtain various types of standardized data includes the following steps: Analyze the type of the original data by manual screening and / or database matching, add a source identifier X to the original data, and generate a type source table according to the source identifier X and the data type of the original data; Split the data with the source identifier X into a training set, a validation set and a test set; The training set, validation set and test set are standardized according to the type source table to obtain multiple types of standardized data.

3. The multimodal data fusion method for ship equipment according to claim 2, characterized in that: The classification training process of the preset first fusion level, the second fusion level, and the meta-learner based on the decision tree and / or the gradient boosting tree framework is implemented by the following steps: Performing model training and verification on the first fusion level and the second fusion level according to the training set and a preset verification method to obtain training results and verification results; The decision tree based meta-learner and / or gradient boosted tree framework is trained based on taking the training result as input and the validation result as output.

4. The multimodal data fusion method for ship equipment according to claim 2, characterized in that: The first fusion level, the second fusion level, and the decision tree-based meta-learner and / or the gradient boosting tree framework are verified according to the verification set, and structural adjustment, hyperparameter adjustment, and feature engineering are performed on the first fusion level, the second fusion level, and the decision tree-based meta-learner and / or the gradient boosting tree framework based on the verification result.

5. The multimodal data fusion method for ship equipment according to claim 3, characterized in that: The preset validation methods include: Hold-out Cross-Validation, K-fold Cross-Validation, Leave-One-Out Cross-Validation (LOOCV), Stratified Cross-Validation and Bootstrap.

6. The multimodal data fusion method for ship equipment according to claim 5, characterized in that: K-fold cross-validation is used for the classification training of the first fusion level, the second fusion level, and the decision tree-based meta-learner and / or the gradient boosting tree framework, with K=5, and the training is performed according to the following steps: S1: Divide the training set into 5 subsets according to the five-fold cross-validation method, and select one of the neural networks stacked 1D-CNN or Wav2vec as the base learner; S2: Select a subset in turn as the test set of the base learner , use the other subsets as the training set of the base learner , output the test results of the training set , while for the test set Make predictions and output prediction results ; S3: loop S2 5 times to get the training set test results ( , , , , ) and the predicted results ( , , , , ), the test results are overlapped and merged vertically to obtain , taking the average of the prediction results to get ; S4: Execute S1-S3 on another base learner to obtain the results generated by the training set and ; S5: , Combined with the source identifier X to obtain a new sample training set ,Will , Merge to get a new test data set ,Will and As input and output of the decision tree based meta learner and / or gradient boosting tree framework, respectively, the decision tree based meta learner and / or gradient boosting tree framework is trained.

7. The multimodal data fusion method for ship equipment according to claim 1, characterized in that: The preset decision tree-based meta-learners and / or gradient boosting tree frameworks include: LightGBM model, XGBoost model, CatBoost model, Neural Networks model, Random Forests algorithm, and AdaBoost algorithm.

8. The multimodal data fusion method for ship equipment according to claim 2, characterized in that: Using a validation set to validate the first fusion level, the second fusion level, and the decision tree-based meta-learner and / or the gradient boosting tree framework, if the validation result meets the preset standard, then inputting the test set to obtain the final prediction result; If the verification result does not meet the preset standard, adjust the parameters of the model and repeat the above steps.

9. A multimodal data fusion tool for ship equipment, used to implement the multimodal data fusion method for ship equipment according to any one of claims 1 to 8, characterized in that: include: Receiving module: used to obtain monitoring requirements and original data of ship equipment; Data feature extraction module: used to extract data features from the original data; Time series feature extraction module: used to fuse the standardized data of various types to obtain corresponding time series features of various types; Fusion module: includes a first fusion layer, a second fusion layer, and a decision tree-based meta-learner and / or a gradient boosting tree framework.

10. A program product comprising computer program instructions, characterized in that: When the computer program instructions are executed, the steps of the method according to any one of claims 1 to 8 are implemented.

Images