Image segmentation method based on small sample domain transfer learning and straw image segmentation model

Abstract

A method for image segmentation based on small sample domain transfer learning and a straw image segmentation model, the method comprises: constructing a twin backbone network structure, in a general network structure, sharing the backbone network weights of a source domain and a target domain with different data feature distributions to form a dual-input twin backbone network; constructing a domain fusion network and performing fusion loss calculation, using the output of the backbone network as the input of the domain fusion network, and using the output of the domain fusion network as the fusion loss calculation; training the backbone network to complete feature layer fine-tuning, freezing the feature analysis network, only training the output layer of the backbone network and the domain fusion network, performing multiple iterations until the fusion loss drops to a threshold value W and stopping the training; and network rollback for tidying up training, rolling back the backbone network to the original general network structure, and then using labeled target domain data for tidying up training to obtain a small sample image segmentation model. The present invention also provides a straw image segmentation model established by the above method.

Description

Technical Field

[0001] The invention relates to machine vision technology, in particular to an image segmentation method based on small sample domain transfer learning and a straw image segmentation model. Background Art

[0002] In the era of big data, the rapid development of many technologies relies on a large amount of data support. Deep learning, which has performed outstandingly in the field of computer vision in recent years, is a typical method that consumes huge amounts of time and computing resources to obtain the best computing model. Figure 1 , Figure 1 It is a common network structure diagram of the prior art. The network structure can be generally divided into four parts, including input layer 10, backbone network 20, feature analysis network 30 and output layer 40. The input layer 10 is responsible for processing the input image; the backbone network 20 is responsible for extracting features from the input image, and its bottom network is the basic features, and the high-level network is the abstract features. The backbone network 20 is mostly a general network framework, such as VGG, ResNet, etc.; the feature analysis network 30 is responsible for integrating and analyzing the features, and this part is different for different networks; the output layer 40 is the final output target of the network.

[0003] Deep learning automatically extracts data features through unsupervised or semi-supervised feature learning algorithms and hierarchical feature extraction. It is a representation learning algorithm based on large-scale data in machine learning. However, agricultural operation data is difficult to obtain and manual data labeling consumes a lot of manpower, material resources and financial resources. It is very difficult to build a large-scale, high-quality labeled data set, which is difficult to support network training needs.

[0004] Traditional machine learning / deep learning can only perform well when the training set data and the test set data are from the same feature space and uniformly distributed, which means that the model must be retrained every time the data is changed. In view of the above-mentioned problems that agricultural data is difficult to obtain and data labeling requires a lot of manpower and material resources, there is a contradiction between the large demand for network training data and the difficulty of establishing high-quality, large-scale databases. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide an image segmentation method and a straw image segmentation model based on small sample domain transfer learning in order to reduce the feature distribution distance between source domain data and target domain data.

[0006] In order to achieve the above object, the present invention provides an image segmentation method based on small sample domain transfer learning, which includes the following steps:

[0007] S100, constructing a twin backbone network structure. In a general network structure, the backbone network weights of the source domain and the target domain with different data feature distributions are shared to form a dual-input twin backbone network, wherein the data with a large number of labels is named the source domain, and S is the input data from the source domain; the data with a small number of labels is named the target domain, and T is the input data from the target domain. The two data S and T are simultaneously input into the dual-input twin backbone network;

[0008] S200, constructing a domain fusion network and performing fusion loss calculation, constructing a domain fusion network corresponding to the input data T of the target domain, inputting the output of the dual-input twin backbone network into the feature analysis network of the general network structure and the domain fusion network at the same time, using the output of the dual-input twin backbone network as the input of the domain fusion network, and calculating the output of the domain fusion network as the fusion loss; and

[0009] S300, training the backbone network to complete feature layer fine-tuning, freezing the feature analysis network, only training the dual-input twin backbone network and the output layer of the domain fusion network, performing multiple iterations until the fusion loss drops to a threshold value W, and then stopping the training; and

[0010] S400, the network is rolled back for sorting and training. Through the dual-input twin backbone network training, the source domain and the target domain obtain approximate feature distribution after mapping. The dual-input twin backbone network is rolled back to the original general network structure, and then the labeled target domain data is used for sorting and training to obtain a small sample image segmentation model.

[0011] In the above-mentioned image segmentation method based on small sample domain transfer learning, when determining the fusion loss in step S200, the domain fusion network projects the source domain and the target domain into a regenerated Hilbert space using the same mapping, and uses the multi-kernel maximum average difference distance to evaluate the discriminability between the features of the source domain and the target domain as the fusion loss.

[0012] In the above-mentioned image segmentation method based on small sample domain transfer learning, a fusion loss calculation is performed in step S200, the output feature map of the dual-input twin backbone network is used as the input of the domain fusion network, a convolution calculation is performed using a k×k size convolution kernel and a c-dimensional feature pool is output, the c-dimensional feature pool is converted into a feature map of the same size P×P, and a fully connected layer is connected to calculate the fusion loss.

[0013] In the above-mentioned image segmentation method based on small sample domain transfer learning, the c-dimensional feature pool is converted into a feature map with the same size of P×P by following the c-dimensional feature pool with ROIPooling convolution calculation.

[0014] In the above-mentioned image segmentation method based on small sample domain transfer learning, the fusion loss is calculated using the following formula:

[0015] Let H be the regenerated Hilbert space, x s is the source domain input, x t As the target domain input, the multi-kernel maximum average difference distance between the two is:

[0016]

[0017] In the formula is the embedded kernel distribution representation, It is a multi-core mapping representation, satisfying the following formula:

[0018]

[0019] Where K is m cores {k u The weighted representation of} is shown in the following formula:

[0020]

[0021] Among them, when d k The closer it is to 0, the smaller the difference between the source domain and the target domain is, and the overall loss function of the network is:

[0022]

[0023] Among them, λ is the penalty coefficient of the fusion distance, l1, l2 are the convolution layer indices to which the fusion layer is added, The maximum average difference distance of multiple kernels in layer l, J(θ(x i ), y i ) is the original general network loss function, and n is the number of samples.

[0024] In order to better achieve the above-mentioned purpose, the present invention also provides a straw image segmentation model based on small sample domain transfer learning, wherein the straw image segmentation model is established by the following steps:

[0025] S100, constructing a twin backbone network structure, copying the backbone network of Mask RCNN, and sharing the backbone network weights of the source domain and the target domain with different data feature distributions to form a dual-input twin backbone network, wherein the source domain data is the Microsoft coco dataset, and the target domain data is the local straw cover image;

[0026] S200, constructing a domain fusion network and calculating the fusion loss, constructing a domain fusion network corresponding to the input data of the target domain, inputting the output of the dual-input twin backbone network into the feature analysis network of the Mask RCNN and the domain fusion network at the same time, using the output of the dual-input twin backbone network as the input of the domain fusion network, and using the output of the domain fusion network as the fusion loss; and

[0027] S300, training the backbone network to complete feature layer fine-tuning, freezing the feature analysis network, only training the dual-input twin backbone network and the output layer of the domain fusion network, performing multiple iterations until the fusion loss drops to a threshold value W, and then stopping the training; and

[0028] S400, the network is rolled back for reorganization training. Through the dual-input twin backbone network training, the source domain and the target domain are mapped to obtain approximate feature distribution. The dual-input twin backbone network is rolled back to the Mask RCNN structure, and then the labeled target domain data is used for reorganization training to obtain a straw image segmentation model.

[0029] In the above straw image segmentation model based on small sample domain transfer learning, in step S300, if loss1 to loss4 are all Mask RCNN network loss functions, and loss5 is a fusion loss function, then the overall network loss function L is:

[0030] L=min(loss1+loss2+loss3+loss4+loss5).

[0031] Based on local straw data and labels, multiple iterations are performed until the fusion loss drops to a threshold value W and the overall network loss function L converges stably, completing the backbone network training.

[0032] In the above-mentioned straw image segmentation model based on small sample domain transfer learning, when determining the fusion loss in step S200, the domain fusion network projects the source domain and the target domain into a regenerated Hilbert space using the same mapping, and uses the multi-kernel maximum average difference distance to evaluate the discriminability between the features of the source domain and the target domain as the fusion loss.

[0033] In the above-mentioned straw image segmentation model based on small sample domain transfer learning, the fusion loss is calculated using the following formula:

[0034] Let H be the regenerated Hilbert space, x s is the source domain input, x t As the target domain input, the multi-kernel maximum average difference distance between the two is:

[0035]

[0036] In the formula is the embedded kernel distribution representation, It is a multi-core mapping representation, satisfying the following formula:

[0037]

[0038] Where K is m cores {k u The weighted representation of} is shown as follows:

[0039]

[0040] Among them, when d k The closer it is to 0, the smaller the difference between the source domain and the target domain is, and the overall loss function of the network is:

[0041]

[0042] Among them, λ is the penalty coefficient of the fusion distance, l1, l2 are the convolution layer indices to which the fusion layer is added, The maximum average difference distance of multiple kernels in layer l, J(θ(x i ), y i ) is the original general network loss function, and n is the number of samples.

[0043] In the above-mentioned straw image segmentation model based on small sample domain transfer learning, a fusion loss calculation is performed in step S200, and the output feature map of the dual-input twin backbone network is used as the input of the domain fusion network. A convolution calculation is performed using a k×k convolution kernel and a c-dimensional feature pool is output. The c-dimensional feature pool is converted into a feature map of the same size P×P, and a fully connected layer is connected to calculate the fusion loss.

[0044] The technical effects of the present invention are:

[0045] The present invention is based on image processing technology, machine learning technology and transfer learning technology, and is based on the network architecture that currently performs well in the field of general target segmentation. A domain fusion module is designed and added to achieve dual-input end-to-end training. Image segmentation learning under few samples can be achieved based on the pre-trained model, and a large amount of labeled other data and a small amount of labeled target image data can be used to transfer learning to obtain the best performing network model, thereby guiding the few-sample machine vision technology in agricultural operations.

[0046] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments, but is not intended to limit the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 This is a common network structure diagram of the prior art;

[0048] Figure 2 A domain migration network structure diagram according to an embodiment of the present invention;

[0049] Figure 3 A domain fusion network according to an embodiment of the present invention;

[0050] Figure 4 This is a twin network structure diagram of an embodiment of the present invention;

[0051] Figure 5 This is a modified Mask RCNN network structure diagram of an embodiment of the present invention;

[0052] Figure 6 This is a diagram of the network structure after rollback according to an embodiment of the present invention.

[0053] Among them, the reference numerals

[0054] Prior art

[0055] 10 Input layer

[0056] 20 Backbone Network

[0057] 30 Feature Analysis Network

[0058] 40 Output layer

[0059] The present invention

[0060] 1 Input layer

[0061] 2 Backbone Network

[0062] 3 Feature Analysis Network

[0063] 4 Output Layer

[0064] 5-Domain Fusion Network DETAILED DESCRIPTION

[0065] The structural principle and working principle of the present invention are described in detail below in conjunction with the accompanying drawings:

[0066] See also Figure 2 , Figure 2 This is a diagram of the domain transfer network structure of an embodiment of the present invention. In the image segmentation method based on small sample domain transfer learning of the present invention, data with a large number of labels is named as the source domain, and data with a small number of labels is named as the target domain. The main difference between the two is that the data feature distribution is different. Constructing a dual-input twin network structure as shown in Figure 2As shown. Image data from different domains are input simultaneously, and insensitive feature representations are obtained through iterative learning of convolutional networks. These features are not sufficient to distinguish the data from the two domains, which is helpful for domain migration applications. According to the output of the backbone network 2 with different inputs, the domain fusion network 5 and fusion loss are designed, and a multi-stage and multi-modal network learning mode is performed to obtain a small sample image segmentation model. Specifically, the following steps may be included:

[0067] Step S100: Construct a twin backbone network 2 structure. In a general network structure, the backbone network weights of the source domain and the target domain with different data feature distributions are shared to form a dual-input twin backbone network 2 (see Figure 4 ), that is, in order to migrate the pre-trained model obtained in the source domain to the target domain, and use its source domain features to enhance the training of the target domain, the weights of the backbone network 2 are shared to form a dual-input twin network. Among them, the general network structure includes an input layer 1, a backbone network 2, a feature analysis network 3 and an output layer 4. The data with a large number of labels is named the source domain, and S of the input layer 1 is the input data from the source domain; the data with a small number of labels is named the target domain, and T of the input layer 1 is the input data from the target domain. The two data S and T are simultaneously input into the dual-input twin backbone network 2, and the weights of this part of the network are shared;

[0068] Step S200: construct a domain fusion network 5 and calculate the fusion loss. The domain fusion network 5 is constructed corresponding to the input data T of the target domain. The outputs of the two backbone networks 2 are simultaneously input into the domain fusion network 5 to calculate the fusion loss (see Figure 3 ). That is, the output of the dual-input twin backbone network 2 is simultaneously input into the feature analysis network 3 of the general network structure and the domain fusion network 5, the output of the dual-input twin backbone network 2 is used as the input of the domain fusion network 5, and the output of the domain fusion network 5 is used as the fusion loss calculation; and

[0069] Step S300, training the backbone network 2 to complete fine-tuning of the feature analysis network 3, freezing the feature analysis network 3, and only training the dual-input twin backbone network 2 and the output layer of the domain fusion network 5, performing multiple iterations until the fusion loss drops to a threshold value W, and then stopping the training; and

[0070] Step S400, the network is rolled back for training. After fine-tuning the feature layer through the dual-input twin backbone network 2 training, the source domain and the target domain are mapped to obtain an approximate feature distribution, and then the labeled target domain data is used for training. During training, the dual-input twin backbone network 2 is rolled back to the original general network structure. At this time, the network loss function is still the original loss function (that is, it does not include the fusion loss function). The optimal computational model is finally obtained through network iterative training, and a small sample image segmentation model is obtained.

[0071] Wherein, when determining the fusion loss in step S200, the domain fusion network 5 projects the source domain and the target domain into a regenerated Hilbert space using the same mapping, and uses the multi-kernel maximum average difference distance to evaluate the discrimination between the features of the source domain and the target domain as the fusion loss. In step S200, the fusion loss calculation is performed, the output feature map of the dual-input twin backbone network 2 is used as the input of the domain fusion network 5, and the convolution calculation is performed using a k×k size convolution kernel and the c-dimensional feature pool is output. Considering that the size of the source domain and the target domain data may be different, the size of the output c-dimensional feature is also different. The c-dimensional feature pool is converted into a feature map of the same size P×P, so that the output features of different input images have the same size, and a fully connected layer is connected to perform the fusion loss calculation of the output feature pool. Preferably, the c-dimensional feature pool is converted into a feature map of the same size P×P by connecting the ROIPooling convolution calculation after the c-dimensional feature pool. Fusion loss is designed for the shared convolution calculation of the backbone network 2. This part can examine the network's discrimination ability for the source domain and the target domain separately. The fusion network projects the source domain and the target domain into a regenerated Hilbert space using the same mapping, and uses the MK-MMD (multiple kernel-maximum mean discrepancies) distance to evaluate the discriminative ability between domain features as the fusion loss.

[0072] In this embodiment, the fusion loss is calculated using the following formula:

[0073] Let H be the regenerated Hilbert space, x s is the source domain input, x t is the target domain input, then the MK-MMD distance between the two is:

[0074]

[0075] In the formula is the embedded kernel distribution representation, It is a multi-core mapping representation, satisfying the following formula:

[0076]

[0077] Where K is m cores {k u The weighted representation of} is shown in the following formula:

[0078]

[0079] Among them, from the above formula, we can see that when d k The closer it is to 0, the smaller the difference between the source domain and the target domain is, and the overall loss function of the network is:

[0080]

[0081] Among them, λ is the penalty coefficient of the fusion distance, l1, l2 are the convolution layer indices to which the fusion layer is added, is the maximum average difference distance of the multi-kernel in layer l, J(θ(x i ), y i ) is the original general network loss function, and n is the number of samples. When the iteration is below the threshold W, the training stops.

[0082] Below, taking the establishment of a straw image segmentation model as an example, the specific application of the image segmentation method based on small sample domain transfer learning of the present invention is explained. Straw is a common agricultural renewable biological resource, and the amount of image annotation data is small and difficult to obtain. Mask RCNN is a classic network for general target segmentation. Mask RCNN is used as an example to implement the straw image segmentation method of domain transfer. The source domain dataset uses the Microsoft coco dataset, which contains 80 types of target annotation data and a total of 117,266 images. The target dataset uses local straw cover images, a total of 840 images. The straw image segmentation model based on small sample domain transfer learning is established using the following steps:

[0083] Step S100, construct the twin backbone network 2 structure, copy the Mask RCNN backbone network 2, and share the backbone network 2 weights of the source domain and the target domain with different data feature distribution to form a dual-input twin backbone network 2, where the source domain data is the Microsoft coco dataset and the target domain data is the local straw cover image (see Figure 5 );

[0084] Step S200, constructing a domain fusion network 5 and calculating the fusion loss, constructing a domain fusion network 5 corresponding to the input data of the target domain, and inputting the output of the dual-input twin backbone network 2 into the feature analysis network 3 of the Mask RCNN and the domain fusion network 5 at the same time, using the output of the dual-input twin backbone network 2 as the input of the domain fusion network 5, and using the output of the domain fusion network 5 as the fusion loss; and

[0085] Step S300, training the backbone network 2 to complete feature layer fine-tuning, selecting M source domain samples and N target domain samples, freezing the feature analysis network 3, only training the dual-input twin backbone network 2 and the output layer of the domain fusion network 5, performing multiple iterations until the fusion loss drops to a threshold value W, stopping training to complete feature layer fine-tuning, in practice, the threshold value W can be set to different values ​​according to needs, in this embodiment, the threshold value W=0.005 is preferably used for the straw image; and

[0086] Step S400, network rollback, through the dual-input twin backbone network 2 training, the source domain and the target domain are mapped to obtain an approximate feature distribution, and the dual-input twin backbone network 2 is rolled back to the Mask RCNN structure to obtain a small sample straw image segmentation model (see Figure 6 ).

[0087] Among them, in step S300, if loss1~loss4 are the network loss functions of the output layer 4 of Mask RCNN respectively, and loss5 is the fusion loss function of the fusion network 5, then the overall network loss function L is:

[0088] L=min(loss1+loss2+loss3+loss4+loss5).

[0089] Based on local straw data and labels, multiple iterations are performed until the fusion loss drops to a threshold value W, the overall network loss function L converges stably, and the training of the backbone network 2 is completed.

[0090] When finishing training, Figure 5 The network structure is iterated repeatedly until the loss function converges stably. At this time, the overall network loss function L is the original Mask RCNN network loss function as follows:

[0091] L=loss1+loss2+loss3+loss4.

[0092] The present invention is based on the general target segmentation method and data, and solves the problem of insufficient training data through transfer learning. A pre-trained model can be reused in another task, and existing knowledge can be used to solve problems in different but related fields. This method evaluates the distribution distance between different data domains by designing the domain fusion network 5 module and fusion loss, thereby realizing feature migration between different data domains. A reliable model can be established using a small amount of labeled training samples or source domain data, shortening the feature difference between general target data and agricultural target data, and predicting target domains with different data distributions, thereby realizing image data segmentation based on small samples.

[0093] Of course, the present invention may have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art may make various corresponding changes and modifications based on the present invention, but these corresponding changes and modifications should all fall within the scope of protection of the claims attached to the present invention.

Claims

1. An image segmentation method based on small sample domain transfer learning, characterized in that: The steps include: S100, constructing a twin backbone network structure. In a general network structure, the backbone network weights of the source domain and the target domain with different data feature distributions are shared to form a dual-input twin backbone network, wherein the data with a large number of labels is named the source domain, and S is the input data from the source domain; the data with a small number of labels is named the target domain, and T is the input data from the target domain. The two data S and T are simultaneously input into the dual-input twin backbone network; S200, constructing a domain fusion network and performing fusion loss calculation, constructing a domain fusion network corresponding to the input data T of the target domain, inputting the output of the dual-input twin backbone network into the feature analysis network of the general network structure and the domain fusion network at the same time, using the output of the dual-input twin backbone network as the input of the domain fusion network, and calculating the output of the domain fusion network as the fusion loss; S300, training the backbone network to complete feature layer fine-tuning, freezing the feature analysis network, only training the dual-input twin backbone network and the output layer of the domain fusion network, performing multiple iterations until the fusion loss drops to a threshold value W, and then stopping the training; and S400, the network is rolled back for sorting and training. Through the dual-input twin backbone network training, the source domain and the target domain are mapped to obtain an approximate feature distribution, the dual-input twin backbone network is rolled back to the original general network structure, and then the labeled target domain data is used for sorting and training to obtain a small sample image segmentation model; Wherein, when determining the fusion loss in step S200, the domain fusion network projects the source domain and the target domain into a regenerated Hilbert space using the same mapping, and uses the multi-kernel maximum average difference distance to evaluate the discrimination between the features of the source domain and the target domain as the fusion loss; In step S200, a fusion loss is calculated, the output feature map of the dual-input twin backbone network is used as the input of the domain fusion network, a convolution calculation is performed using a k×k convolution kernel and a c-dimensional feature pool is output, the c-dimensional feature pool is converted into a feature map of the same size P×P, and a fully connected layer is connected to calculate the fusion loss; The c-dimensional feature pool is converted into a feature map with the same size of P×P by performing ROI Pooling convolution calculation after the c-dimensional feature pool.

2. A straw image segmentation method based on small sample domain transfer learning, characterized in that: The steps include: S100, constructing a twin backbone network structure, copying the backbone network of Mask RCNN, and sharing the backbone network weights of the source domain and the target domain with different data feature distributions to form a dual-input twin backbone network, wherein the source domain data is the Microsoft coco dataset, and the target domain data is the local straw cover image; S200, constructing a domain fusion network and calculating the fusion loss, constructing a domain fusion network corresponding to the input data of the target domain, inputting the output of the dual-input twin backbone network into the feature analysis network of the Mask RCNN and the domain fusion network at the same time, using the output of the dual-input twin backbone network as the input of the domain fusion network, and using the output of the domain fusion network as the fusion loss; S300, training the backbone network to complete feature layer fine-tuning, freezing the feature analysis network, only training the dual-input twin backbone network and the output layer of the domain fusion network, performing multiple iterations until the fusion loss drops to a threshold value W, and then stopping the training; and S400, the network is rolled back for reorganization training. Through the dual-input twin backbone network training, the source domain and the target domain are mapped to obtain an approximate feature distribution, the dual-input twin backbone network is rolled back to the Mask RCNN structure, and then the labeled target domain data is used for reorganization training to obtain a straw image segmentation model; In step S300, if loss1~loss4 are all Mask RCNN network loss functions, and loss5 is the fusion loss function, then the overall network loss function L is: L=min (loss1+loss2+loss3+loss4+loss5); Based on local straw data and labels, multiple iterations are performed until the fusion loss drops to a threshold value W and the overall network loss function L converges stably, completing the backbone network training; When determining the fusion loss in step S200, the domain fusion network projects the source domain and the target domain into a regenerated Hilbert space using the same mapping, and uses a multi-kernel maximum average difference distance to evaluate the discrimination between the features of the source domain and the target domain as the fusion loss; In step S200, a fusion loss calculation is performed, and the output feature map of the dual-input twin backbone network is used as the input of the domain fusion network. A convolution calculation is performed using a k×k convolution kernel and a c-dimensional feature pool is output. The c-dimensional feature pool is converted into a feature map of the same size P×P, and a fully connected layer is connected to calculate the fusion loss.

Images