Sustainable learning water obstacle detection system and method based on knowledge distillation

Abstract

The application relates to a sustainable learning water obstacle detection system based on knowledge distillation, which comprises a teacher network model module, a knowledge distillation module, a student obstacle detection network module and a student unknown class detection module.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of target detection in machine vision, and in particular to a sustainable learning water obstacle detection system and method based on knowledge distillation. BACKGROUND

[0002] In recent years, due to the harsh working environment in the water scene and the high cost of manned equipment, designing reliable unmanned equipment on water has become a research hotspot. Water obstacle detection, as one of the core technologies to realize unmanned driving on water, has been widely and deeply researched. Most of the existing water obstacle detection methods are mainly focused on full supervised learning, that is, using a sufficient number of image data to drive the network to learn mechanically and inefficiently. However, the existing data set resources available for land environment are already very rich and perfect, basically containing various target images in various situations. Compared with the land environment, in the water field, the existing water data set is far from enough, and it is difficult to collect a large enough data set on water due to the complexity of the sea environment, high collection difficulty and equipment cost. Compared with data collection on land, it requires more manpower and financial cost, and technical personnel often cannot get ideal experimental results because they cannot find a suitable data set - which greatly limits the development of this field. At the same time, compared with the land environment, the obstacles that may exist on water are more complex and less likely to be collected, and it is not realistic to collect enough data for each possible object for the system to learn. Under this condition, it is difficult for the existing deep learning neural network to meet the complex and changing needs in the actual scene.

[0003] In the existing target detection method, the output of the neural network model can only be selected from a limited number of known categories set in advance. In this case, when the input data set does not have an unknown category, the output of the model will be forced to assign the unknown category as one of the known categories. For example, a target detection model is trained with a data set of various ships, and after training, the model inputs a picture of a plastic bottle. The model will classify the plastic bottle as a type of ship. This greatly reduces the accuracy of the neural network model, and since the network has a high probability of identifying objects it has never seen in actual application, this error greatly hinders the application of deep learning algorithms in practice. Therefore, there is an urgent need for a system and method that can solve this problem.

[0004] When receiving new visual information, human curiosity often drives people to pay more attention to unknown things, which makes people have more efficient learning ability than machines. Naturally, imitating the characteristics of humans makes machines have the ability to identify unknowns and output classes that they cannot identify and collect data of these unknown classes to relearn, which can effectively solve various problems in the above practical applications and make machines have efficient sustainable learning ability. SUMMARY

[0005] The purpose of the present application is to overcome the defects of the prior art and provide a sustainable learning water obstacle detection system and method based on knowledge distillation.

[0006] The purpose of the present application can be achieved by the following technical solutions:

[0007] A sustainable learning water obstacle detection system based on knowledge distillation, the system comprises:

[0008] The teacher network model module is used to train the teacher network model through the existing picture data set, to realize the detection of known classes and unknown classes of water obstacles and to teach the student network model;

[0009] The knowledge distillation module is used to accurately imitate the target detection result of the teacher network model by the knowledge distillation method;

[0010] The student obstacle detection network module is used to make the student network model imitate the features and target detection results of the teacher network model, and further realize the function of detecting water obstacles;

[0011] The student unknown class detection module is used to cluster the features of the intermediate layer of the student network model, so that the student network model identifies unknown class targets, and the picture data of the unknown class targets are transmitted to the unknown class data collection module;

[0012] The unknown class data collection module is used to obtain picture data of multiple unknown class targets by the data enhancement method, and train the teacher network model to identify new unknown class targets based on the picture data of the unknown class targets.

[0013] The structure of the teacher network model is the network structure of Faster-RCNN.

[0014] The structure of the student network model is the network structure of Faster-RCNN.

[0015] The knowledge distillation module distills knowledge of multiple feature layers of the student network model, and the multiple feature layers of the student network model subjected to knowledge distillation are respectively an intermediate layer for feature extraction, a classification layer of the RPN network, and a regression layer.

[0016] A detection method of the sustainable learning water obstacle detection system, comprising the following steps:

[0017] Step 1: Collect picture data of a water scene and obtain a picture data set, and label obstacles in the picture data of the picture data set;

[0018] Step 2: input the picture data of the data set into a teacher network model, and train the teacher network model;

[0019] Step 3: detect the input picture data through the trained teacher network model to obtain a target detection result, and transmit the target detection result to a knowledge distillation module;

[0020] Step 4: the knowledge distillation module trains a student network model through the teacher network model based on a knowledge distillation method;

[0021] Step 5: detect obstacles in picture data of a water scene by using the trained student network model to obtain a target detection result, and detect unknown class targets in the picture data of the water scene by using a student unknown class detection module, and transmit the picture data to an unknown data collection module;

[0022] Step 6: after the number of picture data of unknown class targets collected by the unknown data collection module reaches a set value, retrain the teacher network model, return to step 2, and realize sustainable learning of the water obstacle detection system.

[0023] In step 4, the process that the knowledge distillation module trains the student network model through the teacher network model based on the knowledge distillation method is specifically as follows:

[0024] The feature map in the teacher network model is converted to be equal in size to the feature map of the student network model, and then the distillation loss is calculated through the knowledge distillation module to force the features extracted by the intermediate layer of the student network model to approximate the features extracted by the intermediate layer of the teacher network model.

[0025] In step 5, the process of detecting unknown class targets in picture data of a water scene by using a student unknown class detection module specifically includes the following steps:

[0026] Step 501: comparing and clustering the features of the region candidate block of the Faster-RCNN, separating the features of the hidden layer by minimizing the loss function, forcing to reduce the intra-class difference and increase the inter-class difference, and pulling apart the distance between the features of different classes;

[0027] Step 502: given the label corresponding to the feature of the feature space, and modeling the probability density function of the features of different classes as the differentiation of different classes, to judge whether the feature belongs to the feature of the unknown class target.

[0028] In step 501, the unknown class detection module superimposes a contrast clustering function on a conventional loss function to obtain a loss function, and the formula of the loss function is:

[0029]

[0030]

[0031] Where, p i is the feature of the i-th target, f c is the c-th input feature, l u (f c ) is the total loss function of all classes, l(f c , p i ) is the sub-loss function corresponding to the i-th target, C is the number of target classes, D is the Euclidean distance function, D(f c , p i ) is the distance between the feature vectors f c and p i , Δ is a set distance parameter for judging whether different features are similar, and c=i indicates that the c-th input feature is the feature of the i-th target, and c≠i indicates that the c-th input feature is not the feature of the i-th target.

[0032] The unknown class target specifically refers to:

[0033] In the existing picture data set for training the teacher network model, there is no class of target that does not exist, that is, there is no class of water obstacle input into the teacher network model and the student network model for learning.

[0034] The target detection result includes the class of the obstacle and the position of the obstacle.

[0035] Compared with the prior art, the present application has the following advantages:

[0036] 1. The sustainable learning water obstacle detection system of the present application can distinguish unknown categories of targets in the current picture and output, which is the category of the target detection system "curiosity". After adding the picture data of the category, the system will learn the picture data to realize sustainable learning.

[0037] 2. The present application introduces a knowledge distillation method to obtain the water obstacle detection ability of the student network model from the existing picture data set through the teacher network model and make the student network model also obtain the water obstacle detection ability through knowledge distillation.

[0038] 3. In practical application, the student network is required to identify unknown class targets that have not been learned and output. After collecting the obstacles of the category through the unknown class data collection module, the teacher network model is sent for retraining. The teacher network model will impart the learned new class targets to the student network model to realize sustainable learning. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 The system structure diagram of the present application.

[0040] Figure 2 The method flow chart of the present application. DETAILED DESCRIPTION

[0041] The present application will be described in detail below in combination with the drawings and specific embodiments.

[0042] EMBODIMENT

[0043] The present application provides a sustainable learning water obstacle detection system based on knowledge distillation, which comprises a teacher network model module, a student obstacle detection network module, a student unknown category detection module, a knowledge distillation module and an unknown class data collection module.

[0044] The teacher network model module: the teacher network model is trained through the existing data set to detect the water obstacle and learn new unknown class targets, obtain the target detection result, and impart the target detection result to the student network model;

[0045] The knowledge distillation module: used to accurately imitate the target detection result of the teacher network model by selecting a specific knowledge distillation method;

[0046] The student obstacle detection network module: imitates the target detection result of the teacher network model through the student network model to obtain the water obstacle target detection ability;

[0047] The student unknown category detection module is used to cluster the intermediate layer features of the student network model, so that it can identify unknown category targets that the student network model has not learned, and transmit the data of the category targets to the unknown category data collection module.

[0048] The unknown category data collection module obtains sufficient data of unknown category targets through the selected data enhancement method to train the teacher network model to identify new categories.

[0049] In addition, the application also provides a water obstacle detection method based on attention to unknown targets, which comprises the following steps:

[0050] Step 1: Collect picture data of a water scene and obtain a picture data set, and calibrate the obstacles in the picture data of the picture data set;

[0051] Step 2: Determine the structure of the teacher network model, input the picture data of the data set into the teacher network model, and train the teacher network model;

[0052] Step 3: Detect the input picture data through the trained teacher network model to obtain a target detection result, and transmit it to the knowledge distillation module;

[0053] Step 4: Determine the structure of the student network model, and determine the feature layer for knowledge distillation by comparing the teacher network model, determine the structure of the knowledge distillation module, and train the student network model through the teacher network model based on the knowledge distillation method;

[0054] Step 5: Detect the obstacles in the picture data of the water scene by using the trained student network model to obtain a target detection result, and detect unknown category targets in the picture data of the water scene by using the student unknown category detection module, and transmit the picture data to the unknown data collection module;

[0055] Step 6: After the number of picture data of unknown category targets collected by the unknown data collection module reaches a set value, return to step 2 to retrain the teacher network model, and realize sustainable learning of the water obstacle detection system.

[0056] The teacher network model and the student network model adopt the network structure of Faster-RCNN, and the feature layer selection needs to be distilled, the intermediate layer for feature extraction, the classification layer of the RPN network and the regression layer, the feature layer size in the teacher network model is converted into the same feature map size as the student network model, and the knowledge distillation module is used to calculate the distillation loss, and the purpose of knowledge distillation is to force the intermediate layer features extracted by the student network model to approximate the intermediate layer features extracted by the teacher network model.

[0057] The unknown category detection module compares and clusters the features of the region candidate blocks of the Faster-RCNN network, so that the features of different categories are forced to be separated, and the formula of the loss function used in the process of comparison and clustering is:

[0058]

[0059]

[0060] wherein p i is the feature of the i-th target, f c is the c-th input feature, l u (f c ) is the total loss function of all categories, l(f c , p i ) is the sub-loss function corresponding to the i-th target, C is the number of target categories, D is the Euclidean distance function, D(f c , p i ) is the distance between the feature vectors f c and p i , Δ is a set distance parameter for judging whether different features are similar, and c=i indicates that the c-th input feature is the feature of the i-th target, and c≠i indicates that the c-th input feature is not the feature of the i-th target.

[0061] The unknown obstacle category is an obstacle category that has not been learned by the teacher network model, i.e., an unknown target, which refers to a target category that is not in the picture data for training the teacher network model, i.e., a category of water obstacles that is not input to the teacher network model and the student network model.

[0062] The loss function of the unknown category detection module superimposes the comparison and clustering function on the conventional loss function, and minimizing the loss function can ensure the separation of the hidden layer features, forcibly reduce the intra-class difference and increase the inter-class difference. After clustering all the features, the distances of the features of different categories are pulled apart, the labels corresponding to the features in the given feature space are given, and the probability density functions of different categories are modeled as the differentiation of different categories to judge whether the sample data belongs to the unknown category.

[0063] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A detection method of a knowledge distillation based sustainable learning water obstacle detection system, characterized in that, The system comprises: a teacher network model module for training a teacher network model through an existing picture data set, detecting known and unknown water obstacles, and imparting to a student network model; a knowledge distillation module for making the student network model accurately imitate the target detection result of the teacher network model through a knowledge distillation method; a student obstacle detection network module for making the student network model imitate the feature and target detection result of the teacher network model, and realizing the function of detecting water obstacles; a student unknown class detection module for clustering the feature of the intermediate layer of the student network model, making the student network model recognize unknown class targets, and transmitting the picture data of the unknown class targets to an unknown class data collection module; an unknown class data collection module for obtaining picture data of multiple unknown class targets through a data enhancement method, and training the teacher network model to recognize new unknown class targets based on the picture data of the unknown class targets; The method comprises the following steps: Step 1: collecting picture data of a water scene and obtaining a picture data set, and labeling the obstacles in the picture data of the picture data set; Step 2: inputting the picture data of the data set into a teacher network model, and training the teacher network model; Step 3: detecting the input picture data through the trained teacher network model to obtain a target detection result, and transmitting it to a knowledge distillation module; Step 4: training the student network model through the teacher network model based on the knowledge distillation method by the knowledge distillation module; Step 5: detecting the obstacles in the picture data of the water scene by using the trained student network model to obtain a target detection result, and detecting unknown class targets in the picture data of the water scene by using a student unknown class detection module, and transmitting the picture data to an unknown data collection module; Step 6: when the number of picture data of unknown class targets collected by the unknown data collection module reaches a set value, re-training the teacher network model, returning to Step 2, and realizing sustainable learning of the water obstacle detection system; In Step 4, the process of training the student network model through the teacher network model based on the knowledge distillation method by the knowledge distillation module is specifically as follows: transforming the feature map in the teacher network model to make it equal in size to the feature map of the student network model, and then calculating the distillation loss through the knowledge distillation module to force the feature extracted by the intermediate layer of the student network model to approximate the feature extracted by the intermediate layer of the teacher network model; The process of detecting unknown class targets in the picture data of the water scene by using the student unknown class detection module specifically comprises the following steps: Step 501: comparing and clustering the features of the region candidate blocks of Faster-RCNN, separating the features of the hidden layers by minimizing the loss function, forcing to reduce the intra-class difference and increase the inter-class difference, and pulling apart the distance between the features of different classes; Step 502: giving the label corresponding to the feature in the feature space, and modeling the probability density function of the features of different classes as a distinction between different classes to determine whether the feature belongs to an unknown class target. In step 501, the unknown category detection module superimposes a contrast clustering function on a conventional loss function to obtain a loss function, and the formula of the loss function is: wherein, is the loss function of the i-th class target, is the feature of the i-th class target, is the i-th input feature, is the loss function of the i-th class target, is the total loss function of all classes, is the loss function of the i-th class target, is the number of target classes, is the Euclidean distance function, is the distance between the feature vectors and is the distance parameter set to determine whether different features are similar, represents that the i-th input feature is the feature of the i-th class target, represents that the i-th input feature is not the feature of the i-th class target, ​​​​​​ 2. The detection method according to claim 1, characterized in that, The structure of the teacher network model is the network structure of Faster-RCNN.

3. The method of claim 1, wherein, The structure of the student network model is the network structure of Faster-RCNN.

4. The method of claim 1, wherein, The knowledge distillation module performs knowledge distillation on multiple feature layers of the student network model, and the multiple feature layers of the student network model subjected to knowledge distillation are respectively an intermediate layer used for feature extraction, a classification layer of an RPN network, and a regression layer.

5. The method of claim 1, wherein, The unknown category target is specifically: A type of target that does not exist in the existing picture data set for training the teacher network model, that is, the type of water obstacle that is not input into the teacher network model and the student network model for learning.

6. The method of claim 1, wherein, The target detection result includes the type of the obstacle and the position of the obstacle.

Images