Model training method, target detection method, storage medium and computing device

Abstract

The present invention relates to the technical field of machine learning, and provides a model training method, a target detection method, a storage medium and a computing device. The model training method comprises: obtaining a plurality of target images, wherein the plurality of target images comprise a source domain image with a label and a target domain image without a label; for each of the plurality of target images: substituting the target image into a feature extractor for feature extraction processing to obtain a first feature map; inputting the first feature map into a classifier to obtain a classification result of the target image; inputting the first feature map into a first discriminator to obtain a first probability value that the target image belongs to a source domain; based on the target detection classification results, the first probability value and the labels of each source domain image corresponding to each of the plurality of target images, training the feature extractor, the classifier and the first discriminator, so that the feature extractor learns the feature distribution that can be shared by the source domain image and the target domain image, so that the classifier can classify the target domain image more accurately.

Description

Technical Field

[0001] The present invention relates to the field of machine learning technology, and in particular to a model training method, a target detection method, a storage medium and an electronic device. Background Art

[0002] Large-scale tree survey is a key research problem. Nowadays, the rapid development of rich remote sensing images and deep learning algorithms has brought new opportunities for large-scale tree detection such as oil palm trees. However, large-scale tree counting and detection may face the problem that remote sensing images with different acquisition conditions, such as different sensors, seasons, and environments, lead to different distributions between images. For example, Figure 1 As shown, image A and image B are two different satellite images. Here, we assume that image A is an image with sufficient labels, where there are 4 categories of labels, namely, the area between oil palm trees, oil palm trees, other vegetation or bare land, impermeable layer or cloud, and image B is an image without labels; due to the differences in sensors, acquisition dates and regional locations, it can be seen that there are obvious differences between image A and image B in the histograms of the 4 categories (used to characterize the distribution of image pixel values); even if the feature extractor and classifier have excellent detection accuracy and classification accuracy under labeled images, when it is directly applied to images without any labels, such as image B, the performance of the feature extractor and classifier may drop sharply.

[0003] Therefore, how to improve the performance of feature extractors and classifiers on unlabeled images has become an urgent problem to be solved. Summary of the invention

[0004] The present invention provides a model training method, a target detection method, a computer-readable storage medium and an electronic device. By training a feature extractor to enable it to learn the feature distribution that can be shared by source domain images and target domain images, a classifier can classify the target domain image more accurately, thereby realizing the migration of the label of the source domain image to the target domain image; in addition, since the forest areas corresponding to the source domain image and the target image are different, cross-regional forest detection can be realized.

[0005] In a first aspect, the present invention provides a model training method, comprising:

[0006] Based on remote sensing images taken from different forest areas, a plurality of target images are obtained, wherein the plurality of target images include labeled source domain images and unlabeled target domain images, and the source domain images and the target domain images correspond to different forest areas;

[0007] For each image of the plurality of target images:

[0008] Substituting the target image into a feature extractor for feature extraction processing to obtain a first feature map;

[0009] Inputting the first feature map into a classifier to obtain a classification result of the target image;

[0010] Inputting the first feature map into a first discriminator to obtain a first probability value that the target image belongs to a source domain;

[0011] Based on the target detection classification results corresponding to each of the plurality of target images and the first probability value, a first loss is obtained, which indicates the uncertainty of the classification result;

[0012] Based on the labels of each image in the source domain and the corresponding classification results, a second loss is obtained, which indicates the classification error of the classifier;

[0013] Based on the first probability values ​​corresponding to each of the plurality of target images, a third loss is obtained, which indicates an error in performing source domain classification based on the first feature map;

[0014] The feature extractor, the classifier and the first discriminator are trained based on the first loss, the second loss and the third loss.

[0015] In a second aspect, the present invention provides a method for target detection, comprising:

[0016] Acquire the target image to be detected;

[0017] Segmenting the target image to determine a plurality of sub-images;

[0018] By means of a feature extractor and a classifier, the plurality of sub-images are respectively detected and classified to determine a detection and classification result; the feature extractor and the classifier are obtained by training through any one of the methods of the first aspect above, and the detection and classification result includes a plurality of target frames and respective categories of the plurality of target frames;

[0019] The target frames belonging to the same category are merged to determine the target detection result of the target image.

[0020] In a third aspect, the present invention provides a model training device, comprising:

[0021] An image acquisition module, configured to obtain a plurality of target images based on remote sensing images taken of different forest areas, wherein the plurality of target images include labeled source domain images and unlabeled target domain images, and the source domain images and the target domain images correspond to different forest areas;

[0022] A classification module is used for: for each of the plurality of target images: substituting the target image into a feature extractor for feature extraction processing to obtain a first feature map; inputting the first feature map into a classifier to obtain a classification result of the target image; inputting the first feature map into a first discriminator to obtain a first probability value that the target image belongs to a source domain;

[0023] A first loss calculation module, configured to obtain a first loss indicating the uncertainty of the classification result based on the target detection classification results corresponding to each of the plurality of target images and the first probability value;

[0024] A second loss calculation module, configured to obtain a second loss based on the labels of each image in the source domain and the corresponding classification results, which indicates a classification error of the classifier;

[0025] A third loss calculation module, configured to obtain a third loss based on the first probability values ​​corresponding to each of the plurality of target images, the third loss indicating an error in performing source domain classification based on the first feature map;

[0026] A training module is used to train the feature extractor, the classifier and the first discriminator based on the first loss, the second loss and the third loss.

[0027] In a fourth aspect, the present invention provides a device for target detection, comprising:

[0028] An image acquisition module, used to acquire a target image to be detected;

[0029] A segmentation module, used to segment the target image and determine a plurality of sub-images;

[0030] A classification module, configured to perform detection and classification on the multiple sub-images respectively through a feature extractor and a classifier, and determine a detection and classification result; the feature extractor and the classifier are trained by any one of the methods described in the first aspect above, and the detection and classification result includes multiple target frames and respective categories of the multiple target frames;

[0031] The merging module is used to merge target frames belonging to the same category to determine the target detection result of the target image.

[0032] In a fifth aspect, the present invention provides a computer-readable storage medium, comprising execution instructions. When a processor of an electronic device executes the execution instructions, the processor executes the method as described in any one of the first aspect or the second aspect.

[0033] In a sixth aspect, the present invention provides an electronic device, comprising a processor and a memory storing execution instructions, wherein when the processor executes the execution instructions stored in the memory, the processor executes the method described in any one of the first aspect or the second aspect.

[0034] The present invention provides a model training method, device, computer-readable storage medium and electronic device. The method obtains multiple target images based on remote sensing images taken from different forest areas, wherein the multiple target images include labeled source domain images and unlabeled target domain images, and the source domain images and the target domain images correspond to different forest areas; for each of the multiple target images: then, the target image is substituted into a feature extractor for feature extraction processing to obtain a first feature map; the first feature map is input into a classifier to obtain a classification result of the target image; the first feature map is input into a first discriminator to obtain The target image belongs to the first probability value of the source domain; then, based on the target detection classification results and the first probability value corresponding to each of the multiple target images, a first loss is obtained, which indicates the uncertainty of the classification result; then, based on the labels of each image in the source domain and the corresponding classification results, a second loss is obtained, which indicates the classification error of the classifier; then, based on the first probability values ​​corresponding to each of the multiple target images, a third loss is obtained, which indicates the error of source domain classification based on the first feature map; then, based on the first loss, the second loss and the third loss, the feature extractor, the classifier and the first discriminator are trained. In summary, the technical solution of the present invention trains the feature extractor to learn the feature distribution that can be shared by the source domain image and the target domain image, so that the classifier can classify the target domain image more accurately, and realize the migration of the label of the source domain image to the target domain image; in addition, since the forest areas corresponding to the source domain image and the target image are different, cross-regional forest detection can be achieved.

[0035] The further effects of the above-mentioned non-conventional preferred manner will be described below in conjunction with specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the existing technical solutions, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0037] Figure 1 A schematic diagram of a histogram of an image under different acquisition conditions provided by an embodiment of the present invention;

[0038] Figure 2A schematic diagram of the structure of the recognition model provided by an embodiment of the present invention;

[0039] Figure 3 A flowchart of a model training method provided by an embodiment of the present invention;

[0040] Figure 4 A schematic diagram of a target detection method provided by an embodiment of the present invention Figure 1 ;

[0041] Figure 5 A schematic diagram of a target detection method provided by an embodiment of the present invention Figure 2 ;

[0042] Figure 6 A schematic diagram of the structure of a model training device provided by an embodiment of the present invention;

[0043] Figure 7 A schematic diagram of the structure of a target detection device provided by an embodiment of the present invention;

[0044] Figure 8 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] First, we introduce the source domain and the target domain. .

[0047] Transfer learning is specifically about applying the knowledge or patterns learned in the source domain to a different but related target domain. The source domain refers to a domain that is different from the test sample, but has rich supervision information; the target domain refers to the domain where the test sample is located, which has no labels or only a small number of labels. The data distribution of the source domain and the target domain is different, but the tasks are the same. Here, the task is what needs to be done, such as tree identification and classification.

[0048] In an embodiment of the present invention, model training is performed through source domain and target domain to achieve the task of tree detection and classification, such as the detection and classification of oil palm trees. It should be noted that the trees here are trees of a certain category. In practical applications, multiple categories of trees can also be considered. The embodiment of the present invention is described by taking a certain category (referred to as the target category for the convenience of description and distinction) of trees as an example. Among them, the source domain includes multiple images with labels, and the target domain includes multiple images without labels. Here, due to the high similarity between trees and areas between trees, in order to improve the accuracy of tree identification, the labels of the embodiment of the present invention include areas between trees and tree categories. Other labels can be designed based on actual conditions, such as other vegetation or bare land, impermeable layers or clouds, etc.

[0049] In practical applications, the images of the source domain and the target domain are usually remote sensing images, and remote sensing images usually contain large tracts of trees. Therefore, the images of the source domain and the target domain are images after segmentation of remote sensing images. In a specific implementation, when constructing the source domain, multiple labels are pre-set; then, remote sensing images taken from one or more regions (including trees of the target category) are selected, and these remote sensing images are manually labeled, for example, the regions and the labels to which the regions belong are marked, and then the marked remote sensing images are segmented, and the segmented images (carrying labels, for example, the image is part of the marked region, and the label of the marked region can be used as the label of the image; if the image has two labels, that is, including the dividing line between the marked different regions, the label of the region with a larger area is used as the label of the image, and if the area is similar, that is, it is difficult to determine the label to which the image belongs, the image can be discarded) form the source domain; in practical applications, the label can also be segmented first. Afterwards, remote sensing images taken from one or more regions (including trees of the target category) other than the above regions are segmented, and the segmented images form the target domain. For example, the size of images in the source and target domains may be 17×17 pixels.

[0050] The construction of the above target domain and source domain is only an example and does not constitute a specific limitation. As long as the acquisition conditions of the source domain and the target domain are different, such as different sensors, seasons, and regions, different regions are preferred.

[0051] It should be noted that the data distribution of the source domain and the target domain in the embodiment of the present invention is different. If a large number of labeled source domains are used for training, the trained classifier will not perform well in the target domain. Based on this, the embodiment of the present invention proposes to train the model in an adversarial transfer learning manner.

[0052] Next, we will introduce adversarial transfer learning. .

[0053] Figure 2The structure of the recognition model provided by the embodiment of the present invention is shown. In the embodiment of the present invention, Figure 2 As shown, the recognition model includes a feature extractor, a classifier, and a first discriminator.

[0054] Adversarial transfer learning is a form of unsupervised deep transfer learning, which uses a feature extractor, a classifier, and a first discriminator to form an adversarial transfer learning network model. The purpose of adversarial transfer learning is to extract features from source domain images and target domain images so that the first discriminator cannot distinguish whether the extracted features are from the source domain or the target domain.

[0055] Among them, the feature extractor is used to map the image to a specific feature space, so that the classifier can distinguish the label from the source domain image, while the first discriminator cannot distinguish which source domain or target domain the image comes from.

[0056] The feature extractor includes N convolution blocks, for example, N=5, and the following description takes N=5 as an example. In one example, the convolution block includes a convolution layer, a batch normalization (Batch Normalization, BN) layer, an instance normalization (Instance Normalization, IN) layer and an activation layer. The convolution layer implements image processing through a convolutional neural network (CNN); the activation layer is processed through an activation function, for example, the activation function can be ReLU (Rectified Linear Units); it should be noted that although the BN layer can effectively accelerate the convergence of the model, it will make CNN insensitive to changes in the image. Therefore, the IN layer is added to eliminate the differences between different individuals, thereby enhancing the generalization of the network. In addition, the BN layer and the IN layer are prior art, and the embodiments of the present invention will not be repeated here.

[0057] Furthermore, the feature extractor further includes a pooling layer located between the jth convolution block and the j+1th convolution block, where j is a positive integer greater than 1. For example, j=2. For example, the pooling method adopted by the pooling layer can be maximum pooling or average pooling. Correspondingly, on the basis that the feature extractor includes the pooling layer, the feature extractor can also include a second discriminator. The second discriminator is used to output the probability that the image belongs to the source domain. For example, when When it is greater than or equal to 0.5, it means that the feature map belongs to the source domain. When it is less than 0.5, it means that the feature map belongs to the target domain. Furthermore, the feature map output by the convolution layer of the last convolution block of the feature extractor is processed by the formula shown in the following formula (1) to obtain a new feature map h i :

[0058]

[0059] Among them, f i h is the feature map output by the last convolutional layer; i It is a new feature map that contains information about migration ability; Represents the feature-level attention value. Features with stronger image transfer ability can be assigned larger feature-level attention values.

[0060] It should be noted that the purpose of feature-level attention is to find features of images that are more mobile between the source domain and the target domain, thereby mapping the source domain and the target domain from the original feature space to the new feature space (the source domain and the target domain have the same data distribution), so that the second discriminator cannot distinguish whether the image is from the target domain or the source domain. In order to measure this mobility, the embodiment of the present invention describes uncertainty by the method of information entropy. Information entropy is also called Shannon entropy, which is calculated by the following formula (2).

[0061] E(p)=-∑ d P d ·log(P d ) (2)

[0062] Where, d = 0, P d Represents the probability that the image belongs to the target domain; d = 1, P d represents the probability that the image belongs to the source domain; the present embodiment only considers the case of d = 1. According to information theory, the greater the entropy, the greater the amount of information, and the stronger the image migration. Correspondingly, the feature extractor can calculate the feature level attention value (V) through the following formula (3): i F ):

[0063]

[0064] Among them, the result output by E(·) is the information entropy.

[0065] In this way, the feature extractor can effectively measure the transfer ability of feature maps, know which feature maps are more suitable for classification, and which feature maps have a negative effect on classification. Therefore, a connection is established between the feature map output by the convolution layer of the last convolutional block of the feature extractor and the feature level attention value to generate a new feature map, which contains the transfer ability information.

[0066] Among them, the classifier classifies the source domain image and tries to assign the correct label as much as possible.

[0067] Among them, the first discriminator classifies the image in the feature space and tries to distinguish whether the image comes from the source domain or the target domain.

[0068] In addition, the feature extractor and the classifier constitute the classification detection part, and the feature extractor and the first discriminator constitute the domain discrimination part. The optimization goals of the feature extractor and the first discriminator are opposite. The first discriminator attempts to determine whether the image comes from the source domain or the target domain, and the feature extractor attempts to make the first discriminator unable to determine the source of the image. The confrontation reflects the optimization goals of both. Through the confrontation, the feature distribution of the source domain image and the target domain image output by the feature extractor is finally made similar, that is, the source domain and the target domain with different distributions are mapped to the same feature space, and a certain measurement criterion is found to make their "distance" in this space as close as possible; thus, the classifier can accurately classify the source domain image and the target domain image at the same time.

[0069] Next, the loss function of the recognition model provided by the embodiment of the present invention is introduced.

[0070] The loss function is shown in the following formula (4).

[0071]

[0072] Among them, L S represents the shallow loss domain loss; L D represents the deep feature domain loss; L E represents entropy loss; μ, α, and β represent hyperparameters for balancing shallow feature domain loss, deep feature domain loss, and entropy loss. represents the classification loss of the labeled source domain images.

[0073] in, Calculated by the following formula (5):

[0074]

[0075] Among them, L y (·) represents the cross entropy loss function; G y (·) represents a classifier, and the output is the predicted source domain i-th image belongs to y i The probability of the class.

[0076] Among them, the shallow feature domain loss is used to enable the feature extractor to learn features with stronger transferability between the source domain and the target domain, so that the second discriminator in the feature extractor cannot distinguish whether the image is from the target domain or the source domain. The shallow feature domain loss is calculated by the following formula (6):

[0077]

[0078] Among them, G d (·) represents the second discriminator, L d (·) indicates G d (·) binary cross entropy loss, for the image in the source domain, Equal to 1, for images in the target domain =0; F′ S Represents the feature map of the i-th image in the source domain output by the j-th convolutional block; F′ T To represent the feature map of the i-th image in the target domain output by the j-th convolutional block. It is worth noting that L S Represents the error of source domain classification based on shallow features.

[0079] It should be noted that the feature map output by the feature extractor is generated through the pooling layer, which leads to the loss of shallow feature information. In addition, considering that the transferability of each image is different, for images that are not similar in the feature space, it will have a negative effect on the migration of the source domain features to the target domain, thereby affecting the classification performance of the classifier. Considering that the feature extractor needs to obtain features with good transferability, the second discriminator cannot distinguish whether the image is from the target domain or the source domain. At the same time, the classifier can complete the classification task well by using the features output by the feature extractor. Figure 2 As shown, the embodiment of the present invention designs a shallow feature domain loss before the pooling layer.

[0080] The deep feature domain loss is used to enable the feature extractor to learn the deep features of the image with stronger transferability between the source domain image and the target domain image, and is calculated by the following formula (7):

[0081]

[0082] Among them, g d (·) represents the first discriminator, L d (·) indicates g d (·) is the binary cross entropy loss. For the source domain image, Equal to 1, for the target domain image is equal to 0. To explain further, g d The output of (·) is the probability that the image belongs to the source domain In this way, our domain loss includes the shallow feature domain loss L S and the domain loss L of the deep features D .

[0083] It should be noted that, considering that the transferability of each image is different, for images that are not similar in feature space, it will have a negative effect on the transfer of knowledge from the source domain to the target domain, thereby affecting the classification performance of the classifier. Figure 2 As shown, the embodiment of the present invention designs a deep feature domain loss before classification.

[0084] Among them, L E Calculated by the following formula (8):

[0085]

[0086] Among them, L E V represents entropy loss, which is used to explain the uncertainty of the classification results of the classifier; i F represents the entropy level attention value; C represents the number of categories, for example, 4 categories; p i，c Represents the probability that the predicted category corresponding to the i-th image is c.

[0087] It should be noted that entropy level attention is similar to feature level attention, which is used to describe the degree of attention paid to the information loss of an image. Images with less mobility may cause the feature extractor to fail to learn features with stronger mobility, reducing the classification accuracy of the classifier. Here, images with stronger mobility have a larger entropy level attention value, and images with less mobility have a smaller entropy level attention value, which makes the feature extractor pay more attention to features with stronger mobility. Correspondingly, the entropy level attention value (V) can be calculated by the following formula (9): i E ):

[0088]

[0089] Among them, the output result of E(·) is the information entropy.

[0090] It should be understood that, inspired by the entropy function of information theory, entropy loss is used to reduce the uncertainty of the output category probability. The embodiment of the present invention uses a target image, and the use of entropy loss has two advantages. On the one hand, since the trees and the areas between trees in the image have a large similarity, entropy loss can improve the confidence of the prediction of easily confused samples; on the other hand, for images with poor similarity, if their confidence is forcibly increased, it will have an adverse effect on the classifier; therefore, the weight of the entropy level attention value is assigned to the entropy loss, so that the minimum entropy regularization based on the entropy level attention makes the prediction of our classifier more credible. Entropy loss regularization, as a penalty term of the loss function, imposes some restrictions on certain parameters in the loss function.

[0091] like Figure 3 As shown, a model training method provided by an embodiment of the present invention. The method provided by an embodiment of the present invention can be applied to an electronic device, and can be specifically applied to a server or a general computer. In this embodiment, the method specifically includes the following steps:

[0092] Step 301, based on remote sensing images taken of different forest areas, a plurality of target images are obtained, the plurality of target images include labeled source domain images and unlabeled target domain images, and the target domain images and the source domain images correspond to different forest areas.

[0093] According to a feasible implementation, the first remote sensing image obtained by the first sensor shooting the first forest area is segmented to obtain multiple images; then, labels are added to these images to obtain multiple source domain images. Of course, in actual applications, the first sensor usually shoots multiple first remote sensing images, and the processing process of each first remote sensing image is the same.

[0094] According to a feasible implementation, the second remote sensing image obtained by the second sensor shooting the second forest area is segmented to obtain multiple images; there is no need to add labels to these images to obtain multiple target domain images. Of course, in actual applications, the second sensor usually shoots multiple second remote sensing images, and the processing process of each second remote sensing image is the same.

[0095] For example, the first sensor and the second sensor are different, for example, the first sensor and the second sensor are sensors on different satellites.

[0096] For example, the first sensor and the second sensor may be the same, but the acquisition times are different.

[0097] For example, the first forest area and the second forest area are different, so that cross-area identification is achieved.

[0098] For example, the size of the target image may be 17×17 pixels.

[0099] For example, the first forest area and the second forest area include trees belonging to the target category, such as oil palm trees. Of course, the oil palm tree is only used as an example and does not constitute a specific limitation. The source domain image and the target domain image can be determined in combination with actual needs, and the embodiment of the present invention does not make specific limitations on this. The source domain image is an image in the above-mentioned source domain, and the target domain image is an image in the above-mentioned target domain.

[0100] For the relevant content about the construction of source domain and target domain, please refer to the above text and will not be repeated here.

[0101] Step 302, for each of the multiple target images: substitute the target image into the feature extractor for feature extraction processing to obtain a first feature map; input the first feature map into the classifier to obtain a classification result of the target image; input the first feature map into the first discriminator to obtain a first probability value that the target image belongs to the source domain.

[0102] The detailed description of the feature extractor and classifier is given above and will not be repeated here.

[0103] Step 303 , based on the target detection classification results corresponding to each of the plurality of target images and the first probability value, a first loss is obtained, which indicates the uncertainty of the classification result.

[0104] According to a feasible implementation method, for each image of multiple target images, the Shannon entropy is calculated based on the corresponding first probability value, and the attention value is calculated based on the Shannon entropy; then, the first loss is determined based on the attention value and the attention values ​​corresponding to each of the multiple target images and the probability values ​​of each category in the classification results.

[0105] The first loss corresponds to the entropy loss L mentioned above E It should be noted that the higher the probability of a certain category in the classification result relative to other categories, the lower the uncertainty of the classification result; if the probability difference between multiple categories in the classification result is small, the uncertainty of the classification result is higher. Therefore, the purpose of designing the first loss is to reduce the uncertainty of the classification result output by the classifier.

[0106] Step 304: Based on the labels of the source domain images and the corresponding classification results, a second loss is obtained, which indicates the classification error of the classifier.

[0107] The second loss corresponds to the classification loss mentioned above It should be noted that the smaller the classification loss, the higher the probability of the label of the source domain image predicted by the classifier, and the more accurate the classification result.

[0108] Step 305 , based on the first probability values ​​corresponding to each of the plurality of target images, a third loss is obtained, which indicates an error in performing source domain classification based on the first feature map.

[0109] The third loss corresponds to the above deep feature domain loss L D It should be noted that the deep feature domain loss L D The smaller it is, the smaller the error of source domain classification of the first feature map output by the feature extractor for the source domain image and the target domain image respectively.

[0110] Step 306: training the feature extractor, the classifier, and the first discriminator based on the first loss, the second loss, and the third loss.

[0111] According to a feasible implementation method, the feature extractor includes a first extraction layer, a second extraction layer, and a second discriminator; the first extraction layer is used to extract shallow features of the target image to obtain a second feature map; the second discriminator is used to judge the second probability value that the target image belongs to the source domain image based on the second feature map; the second extraction layer is used to extract deep features based on the second feature map and the second probability value to obtain the first feature map.

[0112] In one example, the feature extractor includes multiple convolution blocks and pooling layers, the first advance layer includes multiple convolution blocks before the pooling layer, the second extraction layer includes the pooling layer and multiple convolution blocks after it; a single convolution block includes a convolution layer, a batch normalization layer, an instance normalization layer, and an activation layer. For details, please refer to the above and will not be repeated here.

[0113] Furthermore, the method further includes: obtaining a fourth loss based on the second probability values ​​corresponding to each of the plurality of target images, which indicates an error in performing source domain classification based on the second feature map.

[0114] Afterwards, based on the first loss, the second loss, the third loss and the fourth loss, the feature extractor, the classifier and the first discriminator are trained to find features with strong mobility between the source domain image and the target domain image, and construct a feature space with strong mobility, so that the feature extractor maps the source domain image and the target domain image into the feature space, and then the classification of the target domain image is realized through the classifier.

[0115] Through the above technical solution, it can be seen that the beneficial effects of this embodiment are:

[0116] By training the feature extractor to learn the feature distribution that can be shared by the source domain image and the target domain image, the classifier can classify the target domain image more accurately and transfer the label of the source domain image to the target domain image. In addition, since the source domain image and the target image correspond to different forest areas, cross-regional forest detection can be achieved.

[0117] like Figure 4 As shown, a method for target detection provided by an embodiment of the present invention. The method provided by an embodiment of the present invention can be applied to an electronic device, and can be specifically applied to a server or a general computer. In this embodiment, the method specifically includes the following steps:

[0118] Step 401: Acquire a target image to be detected.

[0119] According to a feasible implementation, an image captured by a third sensor on the third area is obtained, and the image is used as a target image. Here, the forest area corresponding to the target image and the forest areas corresponding to the above-mentioned multiple target images may be different.

[0120] Step 402: segment the target image to determine a plurality of sub-images.

[0121] According to a feasible implementation, the target image is divided into overlapping parts through a sliding window to obtain multiple sub-images whose sizes meet the model input requirements, for example, 17×17 pixels.

[0122] Step 403, using a feature extractor and a classifier, respectively detect and classify the multiple sub-images to determine the detection and classification results; the feature extractor and the classifier are trained by any of the above methods, and the detection and classification results include multiple target frames and their respective categories.

[0123] like Figure 5 As shown, for each of the multiple sub-images, the sub-image is sequentially input into the feature extractor and the classifier to obtain the detection and classification results of the classifier for the multiple sub-images, including the coordinates and categories of the target boxes of each sub-image. Here, each sub-image can have multiple target boxes. In practical applications, the classifier outputs a probability distribution (including the probability values ​​corresponding to each of the multiple categories), and the category corresponding to the maximum probability value in the probability distribution is used as the category of the target box in the detection and classification results.

[0124] Step 404: merge target frames belonging to the same category to determine the target detection result of the target image.

[0125] According to a feasible implementation, the target frames of the same category are merged using a criterion based on Intersection-Of-Union (IOU). The IOU-based merging method does not require an iterative step, which can improve the merging efficiency. For example, if the IOU of two target frames of the same category is greater than or equal to a given threshold, the coordinates of the two target frames are averaged. In practical applications, multiple target frames whose IOU is greater than or equal to a given threshold can be merged; the calculation formula of the merged target frame is shown in the following formula (10).

[0126]

[0127] Among them, (X lt , Y lt ) represents the coordinate of the upper left corner of the merged target box; (X rb , Y rb ) represents the coordinate of the lower right corner of the merged target box; n represents the number of target boxes with IOU greater than the threshold; (x lt，i ,y lt，i ) represents the coordinate of the upper left corner of the i-th target box among multiple target boxes whose IOU is greater than the threshold; (x rb，i ,y rb，i ) represents the coordinate of the lower right corner of the i-th target box among multiple target boxes whose IOU is greater than the threshold.

[0128] In practical applications, only the target boxes of specified categories, such as the oil palm tree category, can be merged.

[0129] Through the above technical solution, it can be seen that the beneficial effects of this embodiment are:

[0130] The target image is segmented, and each segmented image is detected and classified separately, and then the target frames of the same type are merged to improve the accuracy of the target detection results of the image.

[0131] Based on the same concept as the embodiment of the method of the present invention, please refer to Figure 6 , an embodiment of the present invention further provides a model training device, comprising:

[0132] An image acquisition module 601 is used to obtain a plurality of target images based on remote sensing images taken of different forest areas, wherein the plurality of target images include labeled source domain images and unlabeled target domain images, and the source domain images and the target domain images correspond to different forest areas;

[0133] The classification module 602 is used for: for each of the plurality of target images: substituting the target image into a feature extractor for feature extraction processing to obtain a first feature map; inputting the first feature map into a classifier to obtain a classification result of the target image; inputting the first feature map into a first discriminator to obtain a first probability value that the target image belongs to a source domain;

[0134] A first loss calculation module 603, configured to obtain a first loss indicating the uncertainty of the classification result based on the target detection classification results corresponding to each of the plurality of target images and the first probability value;

[0135] A second loss calculation module 604 is used to obtain a second loss based on the labels of each image in the source domain and the corresponding classification results, which indicates the classification error of the classifier;

[0136] A third loss calculation module 605 is used to obtain a third loss based on the first probability values ​​corresponding to each of the plurality of target images, which indicates an error in performing source domain classification based on the first feature map;

[0137] The training module 606 is used to train the feature extractor, the classifier and the first discriminator based on the first loss, the second loss and the third loss.

[0138] According to a feasible implementation manner, the plurality of target images are obtained by segmenting remote sensing images respectively taken from different forest areas;

[0139] The source domain images have multiple labels, including at least the tree category and the area between the trees;

[0140] The different forest areas respectively include trees belonging to the target forest category;

[0141] The target domain image and the source domain image come from different sensors; and / or,

[0142] The target domain image and the source domain image correspond to different shooting seasons.

[0143] According to a feasible implementation manner, the feature extractor includes a first extraction layer, a second extraction layer, and a second discriminator; the first extraction layer is used to extract shallow features of the target image to obtain a second feature map; the second discriminator is used to judge, based on the second feature map, a second probability value that the target image belongs to the source domain; the second extraction layer is used to extract deep features based on the second feature map and the second probability value to obtain a first feature map.

[0144] In one example, the feature extractor includes a plurality of convolution blocks and a pooling layer, the first advance layer includes a plurality of convolution blocks before the pooling layer, and the second extraction layer includes the pooling layer and a plurality of convolution blocks thereafter;

[0145] A single convolutional block consists of a convolutional layer, a batch normalization layer, an instance normalization layer, and an activation layer.

[0146] In one example, the second extraction layer is used to calculate entropy based on the second probability value, determine a feature attention value based on the calculated entropy value, extract deep features based on the feature attention value and the second feature map, and obtain a first feature map.

[0147] According to a feasible implementation manner, the device further includes: a third loss calculation module; wherein,

[0148] The third loss calculation module is used to obtain a fourth loss based on the second probability values ​​corresponding to each of the multiple target images, which indicates an error in source domain classification based on the second feature map.

[0149] The training module 606 is used to train the feature extractor, the classifier and the first discriminator based on the first loss, the second loss, the third loss and the fourth loss.

[0150] According to a feasible implementation manner, the first loss calculation module 603 includes: an attention calculation unit and a loss calculation unit; wherein,

[0151] The attention calculation unit is used to calculate entropy for each of the multiple target images based on the corresponding first probability value, and obtain an entropy attention value based on the calculated entropy value, wherein the entropy attention value indicates a degree of attention to the target image;

[0152] The loss calculation unit is used to determine the first loss based on the entropy attention values ​​corresponding to each of the multiple target images and the probability values ​​of each category in the classification result.

[0153] Based on the same concept as the embodiment of the method of the present invention, please refer to Figure 7 , an embodiment of the present invention further provides a device for target detection, comprising:

[0154] The image acquisition module 701 is used to acquire the target image to be detected;

[0155] A segmentation module 702 is used to segment the target image to determine a plurality of sub-images;

[0156] A classification module 703 is used to perform detection and classification on the multiple sub-images respectively through a feature extractor and a classifier to determine a detection and classification result; the feature extractor and the classifier are trained by any method described in the first aspect above, and the detection and classification result includes multiple target frames and their respective categories;

[0157] The merging module 704 is used to merge target frames belonging to the same category to determine the target detection result of the target image.

[0158] Figure 8 It is a structural diagram of an electronic device provided by an embodiment of the present invention. At the hardware level, the electronic device includes a processor 801 and a memory 802 storing execution instructions, and optionally also includes an internal bus 803 and a network interface 804. Among them, the memory 802 may include a memory 8021, such as a high-speed random access memory (Random-Access Memory, RAM), and may also include a non-volatile memory 8022 (non-volatile memory), such as at least one disk storage, etc.; the processor 801, the network interface 804 and the memory 802 can be interconnected through an internal bus 803, and the internal bus 803 can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc.; the internal bus 803 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8Only one bidirectional arrow is used in the figure, but it does not mean that there is only one bus or one type of bus. Of course, the electronic device may also include hardware required for other services. When the processor 801 executes the execution instruction stored in the memory 802, the processor 801 executes the method in any embodiment of the present invention, and is at least used to execute the following steps: Figure 3 or Figure 4 The method shown.

[0159] In a possible implementation, the processor reads the corresponding execution instructions from the non-volatile memory into the memory and then runs them, and can also obtain the corresponding execution instructions from other devices to form a model training device or a target detection device at the logical level. The processor executes the execution instructions stored in the memory to implement a model training method or a target detection method provided in any embodiment of the present invention through the execution of the execution instructions.

[0160] The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or instructions in the form of software. The above processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The disclosed methods, steps and logic block diagrams in the embodiments of the present invention can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0161] The embodiment of the present invention further provides a computer-readable storage medium, including an execution instruction. When a processor of an electronic device executes the execution instruction, the processor executes the method provided in any embodiment of the present invention. The electronic device may be specifically as follows: Figure 8 The electronic device shown; the execution instruction is a computer program corresponding to a model training device or a target detection device.

[0162] Those skilled in the art will appreciate that the embodiments of the present invention may be provided as methods or computer program products. Therefore, the present invention may be implemented in the form of a complete hardware embodiment, a complete software embodiment, or a combination of software and hardware.

[0163] The various embodiments of the present invention are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0164] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0165] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A model training method, characterized in that: include: Based on remote sensing images taken from different forest areas, a plurality of target images are obtained, wherein the plurality of target images include labeled source domain images and unlabeled target domain images, and the source domain images and the target domain images correspond to different forest areas; For each image of the plurality of target images: Substituting the target image into a feature extractor for feature extraction processing to obtain a first feature map; Inputting the first feature map into a classifier to obtain a classification result of the target image; Inputting the first feature map into a first discriminator to obtain a first probability value that the target image belongs to a source domain; Based on the target detection classification results corresponding to each of the plurality of target images and the first probability value, a first loss is obtained, which indicates the uncertainty of the classification result; Based on the labels of each source domain image and the corresponding classification results, a second loss is obtained, which indicates the classification error of the classifier; Based on the first probability values ​​corresponding to each of the plurality of target images, a third loss is obtained, which indicates an error in performing source domain classification based on the first feature map; The feature extractor, the classifier and the first discriminator are trained based on the first loss, the second loss and the third loss.

2. The method according to claim 1, characterized in that The multiple target images are obtained by segmenting remote sensing images respectively taken from different forest areas; Each source domain image has multiple labels, including at least a tree category and an area between trees; The different forest areas respectively include trees belonging to the target forest category; The target domain image and the source domain image come from different sensors; and / or, The target domain image and the source domain image correspond to different shooting seasons.

3. The method according to claim 1, characterized in that The feature extractor includes a first extraction layer, a second extraction layer, and a second discriminator; The first extraction layer is used to extract shallow features of the target image to obtain a second feature map; The second discriminator is used to determine a second probability value that the target image belongs to the source domain based on the second feature map; The second extraction layer is used to extract deep features based on the second feature map and the second probability value to obtain a first feature map.

4. The method according to claim 3, characterized in that The feature extractor includes a plurality of convolution blocks and a pooling layer, the first extraction layer includes a plurality of convolution blocks before the pooling layer, and the second extraction layer includes the pooling layer and a plurality of convolution blocks after the pooling layer; A single convolutional block consists of a convolutional layer, a batch normalization layer, an instance normalization layer, and an activation layer.

5. The method according to claim 3, characterized in that The second extraction layer is used to calculate entropy based on the second probability value, determine a feature attention value based on the calculated entropy value, extract deep features based on the feature attention value and the second feature map, and obtain a first feature map.

6. The method according to claim 3, characterized in that The method further comprises: Based on the second probability values ​​corresponding to each of the plurality of target images, a fourth loss is obtained, which indicates an error in performing source domain classification based on the second feature map; The training of the feature extractor, the classifier and the first discriminator based on the first loss, the second loss and the third loss includes: The feature extractor, the classifier and the first discriminator are trained based on the first loss, the second loss, the third loss and the fourth loss.

7. The method according to claim 1, characterized in that The determining of the first loss based on the target detection classification results corresponding to each of the plurality of target images and the first probability value comprises: For each of the plurality of target images, calculating entropy based on the corresponding first probability value, and obtaining an entropy attention value based on the calculated entropy value, wherein the entropy attention value indicates a degree of attention to the target image; A first loss is determined based on the entropy attention values ​​corresponding to each of the multiple target images and the probability values ​​of each category in the classification results.

8. A method for target detection, characterized in that: include: Acquire the target image to be detected; Segmenting the target image to determine a plurality of sub-images; By means of a feature extractor and a classifier, the plurality of sub-images are respectively detected and classified to determine a detection and classification result; the feature extractor and the classifier are obtained by training by any one of the methods in claims 1 to 7 above, and the detection and classification result includes a plurality of target frames and respective categories of the plurality of target frames; The target frames belonging to the same category are merged to determine the target detection result of the target image.

9. A computer-readable storage medium, comprising an execution instruction. When a processor of an electronic device executes the execution instruction, the processor executes the method according to any one of claims 1 to 7, or the method according to claim 8.

10. A computing device, comprising a processor and a memory storing execution instructions, wherein when the processor executes the execution instructions stored in the memory, the processor executes the method according to any one of claims 1 to 7, or the method according to claim 8.

Images