Self-adaptive active learning method under limited ship multi-modal data labels

Abstract

The embodiment of the invention discloses a self-adaptive active learning method under limited ship multi-modal data labels. The method comprises the steps of obtaining label-free multi-modal ship data and features, constructing a self-adaptive loss function, training a self-adaptive self-supervised model, extracting ship data features and achieving feature aggregation. Performing clustering analysis on the unmarked data feature set, and introducing a nearest neighbor clustering distance difference index to sort the unmarked data; designing a dynamic balance factor according to the maximum performance of the marked data in the decision task model, generating a self-adaptive dynamic sub-pool in combination with the sorting result, and selecting a sample from the sub-pool for marking according to the lowest maximum posterior probability of the unmarked ship multi-modal data in the sub-pool on the decision task model; s3, the data marking result and the decision task model are updated, the step S3 is repeated until the marking number meets the preset value, the marking process of the label-free data is optimized, and the problems that the marking cost of the mass ship label-free data is high and the effect is poor are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal data processing, and in particular to an adaptive active learning method under limited multimodal data labels of ships. Background Art

[0002] As an important pillar of marine economic development, smart ships can achieve efficient decision-making and management in complex marine environments with the help of advanced sensor technology and artificial intelligence algorithms.

[0003] At present, smart ships are generally equipped with a variety of sensors, such as cameras, lidars, and sonars, which can obtain rich multimodal data in real time. These data include not only optical images and point cloud information, but also various forms such as acoustic signals. However, with the rapid increase in the amount of data collected, how to effectively label these unlabeled multimodal data has become a major challenge facing smart ship technology. Traditional data labeling methods mainly rely on manual labeling, which has the disadvantages of high cost and low efficiency, and it is difficult to meet the needs of massive data processing. In addition, when processing multimodal data, existing active learning methods often fail to fully consider the diversity and uncertainty of data, resulting in unsatisfactory labeling results, which in turn affects the performance of the decision model. This not only increases the economic burden of ship operations, but may also hinder timely and effective decision-making at critical moments, affecting marine safety and resource management.

[0004] It is particularly important to improve the autonomous decision-making ability and data processing efficiency of smart ships. Therefore, a new adaptive active learning method is urgently needed to optimize the labeling process of unlabeled multimodal data. This method should be able to strategically select representative and uncertain data under a limited labeling budget, thereby significantly improving the decision-making ability and management level of smart ships in complex marine environments. Summary of the invention

[0005] The purpose of the present invention is to solve the problems raised in the above-mentioned background technology. The present invention provides an adaptive active learning method under limited label of ship multimodal data. When the labels of massive ship data are limited, the method can strategically select data with diversity and uncertainty as sub-sample sets based on a limited labeling budget, optimize the labeling process of unlabeled data, and solve the problems of high cost and poor effect of labeling massive ship unlabeled data.

[0006] In order to achieve the above-mentioned purpose, the present invention specifically adopts the following technical solutions:

[0007] An adaptive active learning method under limited label of multimodal ship data, comprising:

[0008] S1. Obtain unlabeled multimodal ship data and features, construct an adaptive loss function based on the unlabeled multimodal ship data collected by the intelligent ship, train an adaptive self-supervised model, extract ship data features, and realize feature aggregation;

[0009] S2. Sort the unlabeled multimodal ship data, perform cluster analysis on the unlabeled data feature set, and introduce the nearest cluster distance difference index to sort the unlabeled data;

[0010] S3. Construct an adaptive active learning method, design a dynamic balancing factor based on the maximum performance of the labeled data in the decision-making task model, generate an adaptive dynamic sub-pool based on the combined sorting results, and select samples from the sub-pool for labeling based on the lowest maximum a posteriori probability of the unlabeled ship multimodal data in the sub-pool on the decision-making task model;

[0011] S4: Iterate learning, update the data labeling results and decision task model, and repeat step S3 until the number of labels meets the preset value.

[0012] Beneficial effects:

[0013] In view of the fact that a large amount of multimodal sensor data collected by ships during navigation has limited data labels, an active learning strategy is adopted to realize the labeling of unlabeled multimodal data, which effectively saves a lot of manual labeling costs, improves efficiency, and promotes the development of intelligent ship management.

[0014] A new metric, the nearest neighbor cluster difference (NCCDD), is designed to measure the distance to the decision surface, effectively overcoming the difficulty in ensuring the diversity of unlabeled multimodal data in the active learning process. This metric can help identify diverse data, enable the collection and annotation process to cover the characteristics of more different samples, enhance the robustness of the decision model, and improve its generalization on unknown data.

[0015] In the adaptive active learning process in step S3, a dynamic balancing factor, an adaptive dynamic sub-pool generation strategy, and a multimodal ship data sampling and annotation strategy are designed. Different from the equal subset allocation strategy for unlabeled data sets in the conventional active learning process, the adaptive dynamic sub-pool generation strategy implemented by the dynamic balancing factor can adapt to different scenario requirements and ensure the balance between diversity and uncertainty in the sampling and annotation process, especially when facing complex dynamic marine environments, it can use limited annotated data to achieve efficient decision-making tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 This is a flowchart of the overall framework of an embodiment of the present invention;

[0018] Figure 2 This is a workflow diagram of step S1 of an embodiment of the present invention;

[0019] Figure 3 This is a workflow diagram of step S2 of an embodiment of the present invention;

[0020] Figure 4 This is a workflow diagram of step S3 of an embodiment of the present invention;

[0021] Figure 5 This is a workflow diagram of step S4 of an embodiment of the present invention;

[0022] Figure 6 This is a schematic diagram of unlabeled ship multimodal data processing and self-supervised training according to an embodiment of the present invention;

[0023] Figure 7 This is a schematic diagram of clustering and sorting of multimodal data of unlabeled ships according to an embodiment of the present invention;

[0024] Figure 8 Schematic diagram of the overall process of adaptive active learning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0025] To make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0026] like Figure 1 As shown, the present invention provides an adaptive active learning method under limited ship multimodal data labels, comprising the following steps:

[0027] S1. Unlabeled multimodal ship data and feature acquisition. Based on the unlabeled multimodal ship data collected by smart ships, an adaptive loss function is constructed and an adaptive self-supervised model is trained to extract ship data features and achieve feature aggregation;

[0028] S2. Sorting of unlabeled multimodal ship data. Cluster analysis is performed on the unlabeled data feature set, and the nearest cluster distance difference (NCDD) indicator is introduced to sort the unlabeled data;

[0029] S3. Construct an adaptive active learning method. Design a dynamic balancing factor based on the maximum performance of the labeled data in the decision-making task model, generate an adaptive dynamic sub-pool based on the combined sorting results, and select samples from the sub-pool for labeling based on the lowest maximum posterior probability of the unlabeled ship multimodal data in the sub-pool on the decision-making task model.

[0030] S4: Iterate learning, update the data labeling results and decision task model, and repeat step S3 until the number of labels meets the preset value.

[0031] Step S1 defines a class rotation prediction adaptive loss function for unlabeled data of different modalities to achieve effective self-supervised learning, thereby obtaining representative features, and uses a connection method to aggregate features of different modalities to obtain unlabeled multimodal fusion features.

[0032] Further, if Figure 2 and Figure 6 As shown, the specific steps of step S1 are as follows:

[0033] S11. Compared with single-modal data, multimodal data can provide more accurate and comprehensive perception information, improve the decision-making efficiency of smart ships in complex oceans, and overcome the perception difficulties when a single device fails. Therefore, it is necessary to use different high-precision shipborne sensing equipment to synchronously collect multimodal data based on various complex scenes. To this end, the present invention uses three different commonly used high-precision shipborne sensing equipment (such as cameras, lidars, single / multi-beam sonars) as acquisition devices, which are mounted on smart ships to synchronously collect multimodal data based on various complex scenes. The data includes optical images, point clouds, and sonars, expressed as symbols Refers to various types of sensors. The sensors here are shipborne sensing equipment.

[0034] S12. The self-supervised learning method can use the unlabeled ship sample data itself to train an effective model classifier and extract features. Four types of data are obtained by rotating 0, 90, 180 and 270 degrees, and a rotation prediction adaptive loss function is constructed to train adaptive self-supervised models for different modalities. The specific learning process is as follows:

[0035]

[0036] Among them, e ss (·) represents different modal self-supervised learning models Output: Represents the image (point cloud) data obtained by rotating the original image (point cloud) by ε degrees; Refers to the definition label of the adaptive self-supervised model, which can be represented by {0,1,2,3}; represents the cross entropy loss.

[0037] S13. Self-supervised learning model Get the feature sets of three different sensors Using feature concatenation technology to achieve simple and effective high-dimensional aggregation And transform it through a fully connected layer to obtain the fusion feature set

[0038] Further, if Figure 3 and Figure 7 As shown, the specific steps of step S2 are as follows:

[0039] S21, set the number of clusters to K = 5, and use FLS++ to fuse the feature set of unlabeled data Perform clustering and get the cluster center as The specific objective function is implemented as follows:

[0040]

[0041] Among them, μ k Indicates the center position of the cluster.

[0042] S22, to measure the fusion feature m of unlabeled data samples i The difficulty of clustering is to select the data that is most conducive to the perception of decision-making tasks. The present invention designs the nearest neighbor cluster distance difference (NNCDD) as an indicator. Calculate m i The distance difference to the nearest two cluster centers is defined as Ω(m i ), the specific implementation process is as follows:

[0043] Calculate m i Euclidean distance to all cluster centers, select the closest distance distance|m i |1;

[0044]

[0045] Calculate m i The Euclidean distance to the centers of all clusters except the closest cluster in (1), and the closest distance is selected as distance|m i |2;

[0046]

[0047] Calculate m i The distance difference to the two nearest clusters.

[0048] Ω(m i )=distance|m i |2-distance|m i |1

[0049] S23. Since the data with the smallest NNCDD value usually represents the most difficult sample to distinguish, it can provide more valuable information. Therefore, the unlabeled original data set is sorted in ascending order. Sort to get the sorted data set

[0050] Step S3 first sorts the unlabeled dataset Divide into several adaptive dynamic sub-pools Assume that the current decision task model for each round of training in the active learning process is According to the data of the first round pool In order to balance the data diversity and uncertainty in the sampling process, the sample of the multimodal data of the marked ships in the second round of sampling is selected in the decision task model. The maximum performance in the design is the dynamic balance factor.

[0051] like Figure 4 and Figure 8 As shown, the specific steps of step S3 are:

[0052] S31. Different from the conventional active learning process that uses equidistant and non-overlapping sub-pools, the present invention uses adaptive dynamic sub-pool generation to allow flexible adjustment of the labeling strategy to adapt to different data requirements when processing complex and changeable marine ship data. and Synchronous integration And design adaptive dynamic sub-pool based on this The generation strategy is implemented as follows:

[0053]

[0054] Where E represents the number of rounds of running the adaptive active learning method (i.e., unlabeled data sampling); represents the label set of the jth round; N refers to the total number of unlabeled ship data from any mode; ζ represents the dynamic balance factor, which aims to balance the sample diversity and uncertainty in the active learning sampling strategy and maximize the labeling effect.

[0055] S32. In the context of adaptive dynamic sub-pools, the sampling and labeling method of the present invention also has data representativeness and flexibility, which not only avoids repeated labeling of similar data, but also improves the generalization learning ability of the model. Select the first T data as the new label pool At the same time, in order to select data with strong uncertainty as labeled samples, in the lth round (l>1), according to the lth round subpool In the current decision-making task model Select T samples with the lowest maximum posterior probability for labeling. The specific implementation process is as follows:

[0056]

[0057] Among them, T represents the amount of data that needs to be labeled in each round; Refers to the minimum maximum a posteriori probability sampling strategy; Represents the current decision task model The output result of the l-1 round of training is: It means that by Compute sub-pool The output result is:

[0058] S33. In order to effectively cope with the complex and changing needs of multimodal data of marine vessels, ensure that the labeled data is representative, and can well implement intelligent decision-making tasks, the dynamic balance factor ζ allows Unequal generation, with overlapping data (ζ>1) or spaced data (ζ<1) between adjacent sub-pools, is different from the previous principle of equal distance and non-intersection, to ensure that the sampling strategy of active learning can have both data diversity and uncertainty. To this end, the calculation process of the dynamic balance factor ζ is as follows:

[0059] Based on the first round of sampling data Train the corresponding ship decision-making task model

[0060] Given a set of possible dynamic balance factor values Based on these balance factors, the sampled annotation data in the second dynamic sub-pool is obtained

[0061] Update the tag pool to Based on the second round of decision-making task model exist The maximum performance of training Select and determine the optimal dynamic balance factor ζ;

[0062] in

[0063] like Figure 5 and Figure 8 As shown, the specific steps of step S4 are:

[0064] S41. Starting from the second round, data of the current sub-pool is sampled and marked according to the dynamic balance factor ζ;

[0065]

[0066] S42, updating the marked pool after the current round of sampling marking in the active learning process;

[0067]

[0068] S43, based on the updated marked pool Training decision-making task models The decision model is updated in real time. It is determined whether the amount of data in the labeled pool has reached the preset value. If not, the learning continues; otherwise, the active learning process ends. The proposed active learning method is based on a large amount of unlabeled multimodal ship data that is effectively collected and annotated to obtain the data set. It is a decision-making task model trained under limited multimodal data labels, which can be directly used for prediction or classification of downstream decision-making tasks.

[0069] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An adaptive active learning method under limited label of ship multimodal data, characterized in that: include: S1. Obtain unlabeled multimodal ship data and features, construct an adaptive loss function based on the unlabeled multimodal ship data collected by the intelligent ship, train an adaptive self-supervised model, extract ship data features, and realize feature aggregation; S2. Sort the unlabeled multimodal ship data, perform cluster analysis on the unlabeled data feature set, and introduce the nearest cluster distance difference index to sort the unlabeled data; S3. Construct an adaptive active learning method, design a dynamic balancing factor based on the maximum performance of the labeled data in the decision-making task model, generate an adaptive dynamic sub-pool based on the combined sorting results, and select samples from the sub-pool for labeling based on the lowest maximum a posteriori probability of the unlabeled ship multimodal data in the sub-pool on the decision-making task model; S4: Iterate learning, update the data labeling results and decision task model, and repeat step S3 until the number of labels meets the preset value.

2. The method of adaptive active learning under limited label of multimodal ship data according to claim 1 is characterized in that: The specific steps of step S1 are as follows: S11, collect data with three different high-precision ship sensor devices, mounted on smart ships, and synchronously collect multimodal data based on various scenarios, expressed as Refers to various types of sensing devices; S12. Raw data of various types of tag-free ship sensor equipment Four types of data are obtained by rotating 0, 90, 180 and 270 degrees, and a rotation prediction adaptive loss function is constructed to train adaptive self-supervised models for different modalities. The specific learning process is as follows: Among them, l ss (·) represents different modal self-supervised learning models Output: Represents the image data obtained by rotating the original image by ε degrees; Refers to the definition label of the adaptive self-supervised model, represented by {0,1,2,3}; represents the cross entropy loss; S13. Self-supervised learning model Get the feature sets of three different sensors High-dimensional aggregation using feature concatenation And transform it through a fully connected layer to obtain the fusion feature set 3. The adaptive active learning method under limited label of ship multimodal data according to claim 1 is characterized in that: The specific steps of step S2 are as follows: S21, set the number of clusters to K = 5, and use the FLS++ algorithm to fuse the feature set of unlabeled data Perform clustering and get the cluster center as The specific objective function is implemented as follows: Among them, μ k Indicates the center position of the cluster; S22. Design the distance difference between the nearest neighbor clusters to measure the fusion feature m of the unlabeled data sample i The difficulty of clustering is used to select the data that is most conducive to decision-making task perception and calculate m i The distance difference to the nearest two cluster centers is defined as Ω(m i ), the specific implementation process is as follows: (1) Calculate m i Euclidean distance to all cluster centers, select the closest distance distance|m i |1 (2) Calculate m i The Euclidean distance to the centers of all clusters except the closest cluster in (1), and the closest distance is selected as distance|m i |2 (3) Calculate m i The distance difference to the two nearest clusters Ω(m i )=distance|m i |2-distance|m i |1; S23. According to the ascending rule, the unlabeled original data set Sort to get the sorted data set 4. The adaptive active learning method under limited label of ship multimodal data according to claim 1 is characterized in that: The specific steps of step S3 are: S31, sort the multimodal ship dataset and Synchronous integration And design adaptive dynamic sub-pool based on this The generation strategy is implemented as follows: Among them, E represents the number of rounds of running the adaptive active learning method, i.e., unlabeled data sampling; represents the label set of the jth round; N refers to the total number of unlabeled ship data from any modality; ζ represents the dynamic balance factor, which aims to balance the sample diversity and uncertainty in the active learning sampling strategy and maximize the labeling effect; S32. For the first round of sampling strategy, the unlabeled set of sorted ship multimodal data is Select the first T data as the new label pool Marking, at the same time, in order to select data with strong uncertainty as the marked sample, in the lth round (l>1), according to the lth round subpool In the current decision-making task model Select T samples from the lowest maximum posterior probability on the label, and the specific implementation process is as follows: Among them, T represents the amount of data that needs to be labeled in each round; Refers to the minimum maximum a posteriori probability sampling strategy; Represents the current decision task model The output result of the l-1 round of training is: It means that by Compute sub-pool The output result of S33, dynamic balance factor ζ allows Unequally generated, there are overlapping data (ζ>1) or interval data (ζ<1) between adjacent sub-pools. The calculation process of the dynamic balance factor ζ is as follows: (1) Based on the first round of sampling data Train the corresponding ship decision-making task model (2) Given a set of dynamic balance factor values Based on these balance factors, the sampled annotation data in the second dynamic sub-pool is obtained (3) Update the tag pool to Based on the second round of decision-making task model exist The maximum performance of training Select and determine the optimal dynamic balance factor ζ; g=g π* , among them 5. The adaptive active learning method under limited label of ship multimodal data according to claim 1 is characterized in that: The specific steps of step S4 are: S41. Starting from the second round, data of the current sub-pool is sampled and marked according to the dynamic balance factor ζ; S42, updating the marked pool after the current round of sampling marking in the active learning process; S43, based on the updated marked pool Training decision-making task models The decision model is updated in real time to determine whether the amount of data in the labeled pool has reached a preset value. If not, learning continues; Otherwise, the active learning process ends, where the labeled set For the proposed active learning method, a dataset is obtained by effectively collecting and annotating a large amount of unlabeled multimodal ship data. A decision-making task model trained under limited multimodal data labels is used for prediction or classification of downstream decision-making tasks.

Images