A high-precision multi-label classification method applied to images of unmanned aerial vehicles

Abstract

The application discloses a high-precision multi-label classification method applied to unmanned aerial vehicle images, and comprises the following steps: collecting pictures through an unmanned aerial vehicle, marking different labels for different categories of targets in the pictures, generating corresponding label sequences for each picture, taking one picture and the corresponding label sequence thereof as a sample, and generating a sample data set; calculating the support degree of one sample containing two different labels at the same time, and obtaining a frequent item set corresponding to each label; the high-precision multi-label classification method applied to unmanned aerial vehicle images adopts a convolutional neural network to classify the collected pictures according to labels, and combines the correlation degree between the labels to weight the two, so that the precision of picture classification is improved; and the high-precision multi-label classification method applied to unmanned aerial vehicle images can better maintain important feature information of the images by weighting the maximum pooling and average pooling of the pooling layer of the residual neural network.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of image classification, and particularly relates to a high-precision multi-label classification method applied to images of unmanned aerial vehicles. BACKGROUND

[0002] With the continuous development of artificial intelligence, computer vision and visual sensor technology, various image-based technologies are increasingly widely applied to unmanned aerial vehicles. Multi-label classification of all targets on an image is an important prerequisite for the perception function of an unmanned aerial vehicle. For multi-label classification of an image, existing methods convert a multi-classification problem into multiple binary classification problems, but such methods have poor generalization ability and do not take into account the correlation between labels, and thus have low recognition accuracy. SUMMARY

[0003] The application aims to solve the problems in the background art, and provides a high-precision multi-label classification method applied to images of unmanned aerial vehicles.

[0004] To achieve the above object, the application adopts the following technical scheme:

[0005] The high-precision multi-label classification method applied to images of unmanned aerial vehicles provided by the application comprises the following steps: collecting pictures by an unmanned aerial vehicle, labeling different targets in the pictures with different labels, generating a label sequence corresponding to each picture, taking one picture and its corresponding label sequence as a sample, and generating a sample data set.

[0006] The support of a sample containing two different labels at the same time is calculated to obtain a frequent item set corresponding to each label.

[0007] The trained convolutional neural network is used to output the probability of each sample belonging to each label, and a probability subset is selected from the probabilities greater than a specified first probability.

[0008] Each probability in the probability subset is weighted using the frequent item set to calculate the accurate probability of each sample belonging to each label. When the accurate probability is greater than a specified second probability, it is considered that the sample belongs to the corresponding label, and the label classification is completed.

[0009] Preferably, the convolutional neural network is a residual neural network.

[0010] Preferably, when the convolutional neural network is trained, a plurality of pictures are collected, the pictures are subjected to data enhancement, and then divided into a training set and a validation set.

[0011] Then, the training set is input into the convolutional neural network to train the convolutional neural network, and a loss function is defined to verify the training result of the convolutional neural network, and the verification result is fed back to the training of the convolutional neural network.

[0012] Preferably, the calculation formula of the loss function is as follows:

[0013] L c ={L 1，c ,l 2，c ,...,L n，c} T

[0014] and,

[0015] L n，c =W n，c [R c y n，c ·logσ(x n，c )+(1-y n，c )·log(1-σ)(x n，c )]

[0016] Wherein, c represents a label, L c represents the loss value of the label c, l n，c represents the loss value of sample n on the label c, T represents the matrix transpose, n represents the sample number, W n，c and σ respectively represent the super parameter, R c represents the weight adjustment coefficient of label c taking value 1, R c > 1 indicates increasing recall, R c < 1 indicates increasing precision, y n，c represents the attribute corresponding to the label c in the nth sample, and represents the output value of the label c residual error neural network in the n samples.

[0017] Preferably, the data enhancement includes:

[0018] Flip, scale, translate or color transform the collected pictures.

[0019] Preferably, the support of a sample containing two different labels is calculated, including:

[0020] The FP-tree is constructed by using the FP-growth algorithm, and the support formula is used to calculate the support of a sample containing two labels, and the support formula is as follows:

[0021]

[0022] Wherein, bi and bj represent two different labels, sup(bi→bj) represents the support of label bj when label bi appears, num(bi,bj) represents the number of samples containing label bi and label bj, and nums represents the total number of samples.

[0023] Preferably, the frequent item set corresponding to each label is obtained, including:

[0024] A threshold is set, if the support is greater than the threshold, it is called that the label bi and the label bj are related to the association degree, and the frequent item set with the support greater than the threshold is selected, and the frequent item set is as follows:

[0025]

[0026] Wherein, The frequent item set is represented by A, and A represents all the labels related to the label b j The set of labels related to the association degree is represented by A, and sup(A→b j ) represents the set of support of all labels in A and the label b j .

[0027] Preferably, the trained convolutional neural network outputs the probability that each sample belongs to each label respectively, including:

[0028] Each sample is input into the residual neural network, and the residual neural network includes a pooling layer. After the sample image is subjected to maximum pooling and average pooling through the pooling layer, 1x1 convolution operation is performed, and two features of the sample image are output. After the two features of the sample image are weighted and fused, they are input into an activation function, and finally the probability that each sample belongs to each label respectively is output.

[0029] Preferably, a probability subset is selected, including:

[0030] The set of all labels is defined as {b1, b2,... bi... bj... b n}, wherein a sample is represented by y i , and the probability set of the sample y i belonging to each label is represented by Wherein, The probability that the sample y i belongs to the label b n is represented by p(y i |b ), and the first probability is 0.5.

[0031] The probability set of the sample y i belonging to each label is selected, and the probability subset is formed.

[0032] Preferably, the frequent item set is weighted for each probability in the probability subset, and the accurate probability that each sample belongs to each label is calculated, including:

[0033] The accurate probability calculation formula is as follows:

[0034]

[0035] Wherein, The sample yi the precision probability of the sample y j belonging to the label b i is denoted as j the probability of the sample y belonging to the label b denotes the element with the maximum support in the frequent item set.

[0036] When the precision probability is greater than a specified second probability, the sample is considered to belong to the corresponding label, and the label classification is completed.

[0037] Compared with the prior art, the application has the beneficial effects that:

[0038] 1. The high-precision multi-label classification method applied to unmanned aerial vehicle images adopts a convolutional neural network to classify collected pictures according to labels, and combines the correlation degrees between the labels to weight the two, thereby improving the precision of picture classification.

[0039] 2. The high-precision multi-label classification method applied to unmanned aerial vehicle images can better maintain important feature information of an image by weighting the maximum pooling and average pooling of the pooling layer of the residual neural network. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 is a module block diagram of the high-precision multi-label classification method applied to unmanned aerial vehicle images.

[0041] Figure 2 is a flowchart of the convolutional neural network outputting the probability of each sample belonging to each label. DETAILED DESCRIPTION

[0042] The technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only some of the embodiments of the application, but not all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the application.

[0043] It should be noted that when a component is referred to as being “connected” with another component, it can be directly connected with the other component or there can be a component in the middle. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description of the application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0044] In one embodiment, as shown in Figure 1 , a high-precision multi-label classification method applied to unmanned aerial vehicle images comprises:

[0045] The pictures are collected by the unmanned aerial vehicle, different labels are marked for different categories of targets in the pictures, a corresponding label sequence is generated for each picture, and a picture and the corresponding label sequence thereof are taken as a sample to generate a sample data set.

[0046] Specifically, the unmanned aerial vehicle flies at a height of 15 m, 50000 pictures are collected by using an on-board camera, label making is performed on the collected pictures (for example, the labels can be people, bags, shoes or clothes, etc.), at least one target exists in each picture, a certain target is marked as 1 at the corresponding position, and no target is marked as 0, and a corresponding label sequence is generated for each picture.

[0047] The support of a sample containing two different labels is calculated to obtain a frequent item set corresponding to each label.

[0048] Specifically, an FP tree is constructed by using an FP-growth algorithm, and a support formula is used to calculate the support of a sample containing two labels appearing at the same time, and the support formula is as follows:

[0049]

[0050] Wherein, bi and bj represent two different labels, sup(bi→bj) represents the support of the label bj appearing when the label bl appears, num(bi,bj) represents the number of samples containing the label bi and the label bj at the same time, and nums represents the total number of samples.

[0051] A threshold is set, if the support is greater than the threshold, the label bi and the label bj are said to have a correlation degree, and the frequent item set formed by the support greater than the threshold is selected, and the frequent item set is as follows:

[0052]

[0053] Wherein, represents the frequent item set, A represents a set of all labels related to the label b j i, sup(A→bi) represents a set of supports of all labels in A with the label b j i respectively. j

[0054] The correlation degree between labels is constructed, which can reduce redundant information and improve the accuracy of the model.

[0055] The trained convolutional neural network outputs the probability of each sample belonging to each label, and a probability subset is selected by selecting a specified first probability.

[0056] Specifically, the convolutional neural network is a residual neural network. ​

[0057] The 50000 collected pictures are divided into a training set and a validation set after data enhancement, and 80% is used as the training set and 20% is used as the validation set.

[0058] Data enhancement includes flipping, scaling, translation or color transformation of the collected pictures. The purpose of data enhancement is to enrich the samples and prevent fitting during the training process.

[0059] Then the training set is input into the convolutional neural network, the convolutional neural network is trained, and the loss function is defined to test the result of the convolutional neural network training, and the test result is fed back to the convolutional neural network training, and the calculation formula of the loss function is as follows:

[0060] L c ={l 1，c ,l 2，c ,...,l n，c} T

[0061] and,

[0062] l n，c =W n，c [R c y n，c ·logσ(x n，c )+(1-y n，c )·log(1-σ)(x n，c )]

[0063] Wherein, c represents the label, L c represents the loss value of label c, l n，c represents the loss value of sample n on label c, T represents matrix transposition, n represents sample number, W n，c and σ respectively represent super parameter, R c represents the weight adjustment coefficient of label c with value 1, R c > 1 represents increasing recall rate, R c < 1 represents increasing precision, y n，c represents the label c corresponding attribute in the nth sample, represents the output value of label c residual error neural network in n samples.

[0064] Each sample is input into the residual error neural network, which includes a pooling layer. After maximum pooling and average pooling (which can better maintain important feature information of the image) of the sample image through the pooling layer, 1x1 convolution operation is performed, and two features of the sample image are output. After weighting and fusing the two features of the sample image, it is input into the activation function, and finally the probability of each sample belonging to each label is output.

[0065] Define the set of all labels as {b1, b2, ...b i ...b j ..., b n}, the set of all samples is represented as {y1, y2, ...y i ..., y n}, sample y i The probability set belonging to each label is expressed as in, Represents sample y i Belongs to label b n The probability of, the first probability is 0.5 (generally stipulated).

[0066] Select sample y i The elements in the probability set belonging to each label are greater than 0.5, forming a probability subset.

[0067] Each probability in the probability subset is weighted by the frequent item set, and the precise probability of each sample belonging to each label is calculated. When the precise probability is greater than the specified second probability, the sample is considered to belong to the corresponding label, and the label classification is completed.

[0068] Specifically, the precise probability calculation formula is as follows:

[0069]

[0070] in, Represents sample y i Belongs to label b j The exact probability of , γ represents the weight, Represents sample y i Belongs to label b j The probability of Represents the element with the largest support in the frequent item set.

[0071] When the exact probability is greater than the specified second probability (the second probability can be obtained through experimental data), the sample is considered to belong to the corresponding label, and multi-label classification is completed.

[0072] This high-precision multi-label classification method for drone images uses a convolutional neural network to classify the collected images by label, and combines the correlation between the labels to weight the two to improve the accuracy of image classification; this high-precision multi-label classification method for drone images can better maintain the important feature information of the image by weighting the maximum pooling and average pooling of the pooling layer of the residual neural network.

[0073] Any combination of the technical features in the above-described embodiments can be made, and for the sake of brevity, not all possible combinations are described, however, as long as there is no conflict, any combination of the technical features should be considered within the scope of the present disclosure.

[0074] The above-described embodiments are merely representative and detailed embodiments described in the present application, but should not be construed as limiting the scope of the patent application. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of the present application. Therefore, the scope of the patent of the present application should be subject to the appended claims.

Claims

1. A high-precision multi-label classification method for drone images, characterized by: The high-precision multi-label classification method applied to drone images includes: Use drones to collect images, label different categories of objects in the images with different labels, generate a corresponding label sequence for each image, and use an image and its corresponding label sequence as a sample to generate a sample dataset; Calculate the support of a sample containing two different labels at the same time, and obtain the frequent item set corresponding to each label; Use the trained convolutional neural network to output the probability of each sample belonging to each label, and select the probability subset with a probability greater than the specified first probability; Each probability in the probability subset is weighted by the frequent item set to calculate the exact probability that each sample belongs to each label. When the exact probability is greater than the specified second probability, the sample is considered to belong to the corresponding label and the label classification is completed. The method of weighting each probability in the probability subset using a frequent item set to calculate the precise probability that each sample belongs to each label includes: The formula for calculating the precise probability is as follows: ； in, Represents a sample Belong to the label The exact probability of represents the weight, Represents a sample Belong to the label The probability of Represents the element with the largest support in the frequent item set; When the exact probability is greater than the specified second probability, the sample is considered to belong to the corresponding label and the label classification is completed.

2. The high-precision multi-label classification method for drone images according to claim 1, wherein: The convolutional neural network is a residual neural network.

3. The high-precision multi-label classification method for drone images according to claim 2, characterized in that: When the convolutional neural network is trained, pictures are collected, and after data enhancement is performed on each picture, the pictures are divided into a training set and a validation set; The training set is then input into the convolutional neural network to train the convolutional neural network, and a loss function is defined to test the results of the convolutional neural network training, and the test results are fed back to the convolutional neural network training.

4. The high-precision multi-label classification method for drone images according to claim 3, wherein: The calculation formula of the loss function is as follows: ； and, ； in, Indicates a label, Representation Label The loss value, Represents a sample In the label The loss value on represents the matrix transpose, represents the number of samples, and Represent the hyper parameters, Representation Label The value of 1 is the weight adjustment coefficient, > Indicates an increase in recall rate, < Indicates increased precision, Indicates the labels in samples The corresponding attribute indicates labels in samples Output value of the residual neural network.

5. The high-precision multi-label classification method for drone images according to claim 3, wherein: The data enhancement includes: Flip, zoom, translate, or change the color of the captured images.

6. The high-precision multi-label classification method for drone images according to claim 1, wherein: The calculation of the support of a sample containing two different labels at the same time includes: The FP-growth algorithm is used to construct the FP tree, and the support formula is used to calculate the support of a sample containing two labels at the same time. The support formula is as follows: ； in, 、 Represents two different labels, Representation Label support, Indicates that it also contains tags and tags The number of samples, Indicates the total number of samples.

7. The high-precision multi-label classification method for drone images according to claim 6, characterized in that: Obtaining the frequent item set corresponding to each tag includes: Set a threshold, if the support is greater than the threshold, the label is called and tags There is a correlation, and the frequent item sets with support greater than the threshold are selected, and the frequent item sets are as follows: ； in, represents a frequent item set, Represents all tags A collection of related tags, express All tags in the The set of support.

8. The high-precision multi-label classification method for drone images according to claim 2, wherein: The trained convolutional neural network is used to output the probability that each sample belongs to each label, including: Input each of the samples into the residual neural network, which includes a pooling layer. After performing maximum pooling and average pooling on the sample image through the pooling layer, a 1x1 convolution operation is performed to output two features of the sample image. After weighted fusion of the two features of the sample image, the two features are input into the activation function, and finally the probability of each sample belonging to each label is output; 9. The high-precision multi-label classification method for drone images according to claim 1, wherein: The forming probability subset whose selection probability is greater than the specified first probability includes: Define the set of all tags as , where a sample is represented as ,sample The probability set belonging to each label is expressed as ,in, Represents a sample Belong to the label The probability of , the first probability is 0.5; Select samples The elements in the probability set belonging to each label are greater than 0.5, forming a probability subset.

Images