Weakly supervised multi-label long-tail distribution recognition method and product enhanced by positioning data

Abstract

The embodiment of the application relates to the field of multi-label recognition, and provides a weakly supervised positioning data enhanced multi-label long-tail distribution recognition method and product, which comprises the following steps: inputting an input image and all candidate frames corresponding to the input image into an optimized weakly supervised deep positioning network to obtain a detection frame of the input image, the detection frame of the input image being an optimal candidate frame of a corresponding category of the input image; copying the detection frame of the input image of a tail category in the input image and pasting the detection frame into a random position in another input image in the same batch to obtain a data-enhanced input image, the data-enhanced data image being used for training a multi-label classification neural network. The application improves the number of tail categories existing in an input image set, thereby relieving the semantic imbalance problem of a long-tail distribution, and uses the same as an input image for data enhancement to train a classification neural network, thereby improving the performance of the classification neural network in a multi-label recognition scene under a long-tail distribution.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of multi-label recognition, in particular, to a weakly supervised positioning data enhanced multi-label long-tail distribution recognition method and product. BACKGROUND

[0002] When a deep learning algorithm trains a neural network and completes a corresponding computer vision task, a large and balanced data set is often needed. However, semantic concepts in nature are often long-tail distributed, that is, a small number of semantics have a large number of sample sizes, while a large number of semantic concepts have a very low frequency of occurrence. This results in an unbalanced class distribution of training data when collecting the training data for the neural network. If direct collection or use of web crawlers and other technologies without additional restrictions is used, a long-tail distribution is often introduced, resulting in an unbalanced class distribution of training data.

[0003] Current long-tail distribution recognition methods are designed to solve the problem of single-label long-tail distribution recognition, mainly including re-weighting method, re-sampling method and two-stage method. However, these methods often fail to achieve good results in the face of multi-label long-tail distribution recognition problems. For example, for the re-weighting method, the semantic association between different classes is ignored, so the problem of long-tail distribution imbalance cannot be completely solved. For the re-sampling method, because many classes have a common occurrence relationship in multi-label recognition problems, the long-tail distribution still exists after re-sampling. Therefore, how to improve the performance of neural networks in the multi-label recognition scene under the long-tail distribution has become a problem to be solved. SUMMARY

[0004] Embodiments of the present application provide a weakly supervised positioning data enhanced multi-label long-tail distribution recognition method and product, aiming to solve the problem of performance degradation of neural networks in the multi-label recognition scene under the long-tail distribution.

[0005] The first aspect of the embodiments of the present application provides a weakly supervised positioning data enhanced multi-label long-tail distribution recognition method, comprising:

[0006] inputting an input image and all candidate boxes corresponding to the input image into an optimized weakly supervised deep positioning network to obtain a detection box of the input image, the detection box of the input image being an optimal candidate box of a corresponding class of the input image;

[0007] copying the detection box of the input image of the tail class in the input image and pasting it to a random position in another input image in the same batch to obtain a data enhanced input image, the data enhanced data image being used to train a multi-label classification neural network.

[0008] Optionally, after obtaining the data enhanced input image, comprising:

[0009] training the multi-label classification neural network with the image-enhanced input image as training data;

[0010] obtaining confidence of the input image of the tail category, the confidence of the input image being an optimal score of a detection frame of the input image obtained by inputting the input image and all candidate frames corresponding to the input image into the optimized weakly supervised deep positioning network;

[0011] using the confidence of the input image of the tail category as a weight of a loss function of the multi-label classification neural network, optimizing the loss function of the multi-label classification neural network to obtain an optimized loss function of the multi-label classification neural network;

[0012] training and optimizing parameters of the multi-label classification neural network through the optimized loss function of the multi-label classification neural network to obtain an optimized multi-label classification neural network.

[0013] Optionally, inputting the input image and all candidate frames corresponding to the input image into the optimized weakly supervised deep positioning network to obtain a detection frame of the input image, comprising:

[0014] processing the input image to obtain all candidate frames corresponding to the input image;

[0015] inputting each candidate frame corresponding to the input image into a pre-trained sub neural network of the initialized weakly supervised deep positioning network to obtain a classification output of each candidate frame and a detection output of each candidate frame;

[0016] calculating the classification output of each candidate frame and the detection output of each candidate frame to obtain a classification score of each candidate frame and a detection score of each candidate frame;

[0017] calculating a score of each candidate frame according to the classification score of each candidate frame and the detection score of each candidate frame;

[0018] adding the scores of all candidate frames and calculating a weakly supervised deep positioning loss function;

[0019] training and optimizing parameters of the initialized weakly supervised deep positioning network by optimizing the calculated weakly supervised deep positioning loss function to obtain an optimized weakly supervised deep positioning network;

[0020] inputting all input images in the input image set into the optimized weakly supervised deep positioning network to obtain a detection frame of each input image and a confidence of each input image, the detection frame of each input image being an optimal candidate frame corresponding to a category of each input image, and the confidence of each input image being an optimal score of the detection frame of each input image.

[0021] Optionally, the bounding box of the input image of the tail category in the input image is copied and pasted to a random position in another input image in the same batch to obtain a data-augmented input image, comprising:

[0022] An input image of a tail category in an input image is obtained as a first input image, and a bounding box of the first input image is selected for copying;

[0023] Another input image in the same batch is randomly selected as a second input image, and the copied bounding box of the first input image is pasted and overlaid in a random position in the second input image;

[0024] The category and position of the bounding box of the first input image are added to the label of the second input image to obtain the data-augmented input image.

[0025] Optionally, the classification output of each candidate box and the detection output of each candidate box are calculated to obtain a classification score of each candidate box and a detection score of each candidate box, which are calculated according to the following formula:

[0026]

[0027] Wherein, r is a candidate box, c is a category of a candidate box, is a classification output of a candidate box r belonging to a category c, and is a detection output of a candidate box r belonging to a category c; is a classification score of a candidate box r, and is a detection score of a candidate box r.

[0028] Optionally, the copied bounding box of the first input image is pasted and overlaid in a random position in the second input image, comprising:

[0029] A random position in the second input image is generated;

[0030] It is judged whether the random position in the second input image contains a bounding box of the second input image;

[0031] If the area proportion of the bounding box of the second input image contained in the random position in the second input image exceeds a preset threshold, a new random position in the second input image is regenerated until the area proportion of the bounding box of the second input image contained in the random position in the second input image does not exceed the preset threshold;

[0032] The copied bounding box of the first input image is pasted and overlaid in the random position in the second input image.

[0033] The second aspect of the present application provides a weakly supervised positioning data-augmented multi-label long-tail distribution recognition device, comprising:

[0034] The weakly supervised positioning module is configured to input the input image and all candidate boxes corresponding to the input image into the optimized weakly supervised deep positioning network to obtain a detection box of the input image, the detection box of the input image being an optimal candidate box of the input image corresponding to a category;

[0035] The data enhancement module is configured to copy the detection box of the input image of the tail category in the input image and paste the detection box into a random position in another input image in the same batch to obtain a data-enhanced input image, the data-enhanced data image being used for training the multi-label classification neural network.

[0036] The apparatus further includes:

[0037] The classification neural network training module is configured to train the multi-label classification neural network by taking the image-enhanced input image as training data.

[0038] The confidence obtaining module is configured to obtain a confidence of the input image of the tail category, the confidence of the input image being an optimal score of the detection box of the input image obtained by inputting the input image and all candidate boxes corresponding to the input image into the optimized weakly supervised deep positioning network.

[0039] The weighting module is configured to take the confidence of the input image of the tail category as a weight of a loss function of the multi-label classification neural network, and optimize the loss function of the multi-label classification neural network to obtain an optimized loss function of the multi-label classification neural network.

[0040] The classification neural network optimization module is configured to train and optimize parameters of the multi-label classification neural network by using the optimized loss function of the multi-label classification neural network to obtain an optimized multi-label classification neural network.

[0041] The weakly supervised positioning module includes:

[0042] The candidate box obtaining submodule is configured to process the input image to obtain all candidate boxes corresponding to the input image.

[0043] The candidate box output submodule is configured to input each candidate box corresponding to the input image into a pre-trained sub-neural network of the initialized weakly supervised deep positioning network to obtain classification output of each candidate box and detection output of each candidate box.

[0044] The first candidate box score submodule is configured to calculate the classification output of each candidate box and the detection output of each candidate box to obtain a classification score of each candidate box and a detection score of each candidate box.

[0045] a second candidate box score calculation module, configured to calculate a score of each candidate box according to the classification score of the candidate box and the detection score of the candidate box;

[0046] a weakly supervised deep positioning loss function calculation sub-module, configured to add up the scores of all candidate boxes and calculate a weakly supervised deep positioning loss function;

[0047] a weakly supervised deep positioning network optimization sub-module, configured to train and optimize the parameters of the initialized weakly supervised deep positioning network by optimizing the calculated weakly supervised deep positioning loss function, to obtain an optimized weakly supervised deep positioning network;

[0048] a detection box output sub-module, configured to input all input images in an input image set into the optimized weakly supervised deep positioning network, to obtain a detection box of each input image and a confidence of each input image, the detection box of each input image being an optimal candidate box of a corresponding category of each input image, and the confidence of each input image being an optimal score of the detection box of each input image.

[0049] The data augmentation module comprises:

[0050] a copying sub-module, configured to obtain an input image of a tail category in the input image as a first input image, and copy a detection box of the first input image;

[0051] a pasting sub-module, configured to randomly select another input image in the same batch as a second input image, and paste the copied detection box of the first input image to a random position in the second input image;

[0052] a data augmentation sub-module, configured to add the category and position of the detection box of the first input image to the label of the second input image, to obtain the data-augmented input image.

[0053] The pasting sub-module further comprises:

[0054] a first random position generation sub-unit, configured to generate a random position in the second input image;

[0055] a judgment sub-unit, configured to judge whether the random position in the second input image contains a detection box of the second input image;

[0056] a final random position generation sub-unit, configured to, if the area proportion of the detection box of the second input image contained in the random position in the second input image exceeds a preset threshold, re-generate a new random position in the second input image, until the area proportion of the detection box of the second input image contained in the random position in the second input image does not exceed the preset threshold;

[0057] pasting sub-unit configured to paste the bounding box of the copied first input image to a random position in the second input image.

[0058] The third aspect of the present application provides an electronic device, comprising a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the weakly supervised positioning data enhanced multi-label long-tail distribution identification method in any one of the first aspect.

[0059] The fourth aspect of the present application provides a computer readable storage medium, which stores a computer program / instruction, wherein the computer program / instruction is executed by a processor to implement the steps of the weakly supervised positioning data enhanced multi-label long-tail distribution identification method in any one of the first aspect.

[0060] The fifth aspect of the present application provides a computer program product, comprising a computer program / instruction, wherein the computer program / instruction is executed by a processor to implement the steps of the weakly supervised positioning data enhanced multi-label long-tail distribution identification method in any one of the first aspect.

[0061] Advantages:

[0062] The embodiments of the present application provide a weakly supervised positioning data enhanced multi-label long-tail distribution identification method and product. An input image and all candidate boxes corresponding to the input image are input into an optimized weakly supervised deep positioning network to obtain a detection box of the input image, the detection box of the input image being an optimal candidate box of a category corresponding to the input image. The detection box of the input image of a tail category in the input image is copied and pasted to a random position in another input image in the same batch to obtain a data enhanced input image, and the data enhanced data image is used to train a multi-label classification neural network. The method of the present application copies and pastes the detection box of the tail category determined by the weakly supervised deep positioning network, improves the number of tail categories existing in the input image set, and thus alleviates the semantic imbalance problem of the long-tail distribution. The input image is used as a data enhanced input image to train the classification neural network. The optimized multi-label classification neural network is trained based on the data enhanced input image for multi-label long-tail distribution identification, and thus the performance of the multi-label classification neural network in the multi-label identification scene of the long-tail distribution is improved. BRIEF DESCRIPTION OF DRAWINGS

[0063] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the description of the embodiments of the present application will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0064] Figure 1 is a weakly supervised positioning data enhancement multi-label long-tail distribution identification method flowchart proposed by an embodiment of the present application;

[0065] Figure 2 is a weakly supervised deep positioning network training optimization flowchart proposed by an embodiment of the present application;

[0066] Figure 3 is a weakly supervised positioning data enhancement multi-label long-tail distribution identification method example diagram proposed by an embodiment of the present application;

[0067] Figure 4 is a weakly supervised positioning data enhancement multi-label long-tail distribution identification device schematic diagram proposed by an embodiment of the present application;

[0068] Figure 5 is an electronic device schematic diagram proposed by an embodiment of the present application. DETAILED DESCRIPTION

[0069] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0070] In related technologies, long-tail distribution identification methods are designed to solve the identification problem of single-label long-tail distribution, mainly including re-weighting method, re-sampling method and two-stage method. However, these methods often fail to achieve good results in the face of multi-label long-tail distribution identification problem. For example, for the re-weighting method, since the semantic association between different categories is ignored, the imbalance problem of long-tail distribution cannot be completely solved; for the re-sampling method, since many categories in the multi-label identification problem have common occurrence relationship, the long-tail distribution still exists after re-sampling.

[0071] Therefore, an embodiment of the present application proposes a weakly supervised positioning data enhancement multi-label long-tail distribution identification method, Figure 1 shows a weakly supervised positioning data enhancement multi-label long-tail distribution identification method flowchart, as Figure 1 shown, comprising:

[0072] S101, input the input image and all candidate boxes corresponding to the input image into the optimized weakly supervised depth positioning network to obtain a detection box of the input image.

[0073] Input the input image and all candidate boxes corresponding to the input image into the optimized weakly supervised depth positioning network to obtain a detection box of the input image, wherein the detection box of the input image is the optimal candidate box of the input image corresponding to the category.

[0074] S102, copy the detection box of the input image of the tail category in the input image and paste it to a random position in another input image in the same batch to obtain a data enhanced input image.

[0075] Copy the detection box of the input image of the tail category in the input image and paste it to a random position in another input image in the same batch to obtain a data enhanced input image, wherein the data enhanced data image is used for training a multi-label classification neural network.

[0076] S103, use the data enhanced input image for training and optimization of the multi-label classification neural network to obtain an optimized multi-label classification neural network.

[0077] Use the data enhanced input image for training and optimization of the multi-label classification neural network to obtain an optimized multi-label classification neural network, wherein the obtained optimized multi-label classification neural network can process classification problems in a multi-label recognition scene under a long-tail distribution.

[0078] When S101 is implemented, the initialized weakly supervised depth positioning network needs to be trained and optimized first to obtain an optimized weakly supervised depth positioning network. The optimized weakly supervised depth positioning network is used to obtain the optimal candidate box corresponding to the category from the input image as the detection box. Figure 2 The training and optimization flowchart of the weakly supervised depth positioning network is shown in FIG. Figure 2 The process of training and initializing the weakly supervised depth positioning network includes the following steps:

[0079] S101-1, process the input image to obtain all candidate boxes corresponding to the input image.

[0080] In implementation, the input image in the input image set of the classification neural network is used as the training data of the initialized weakly supervised deep positioning network. The input image in each training process is processed to obtain all candidate boxes in the input image of this training. The candidate box is all possible image regions in the input image. Since the candidate box is all possible image regions in the input image, it includes candidate boxes containing the corresponding category, candidate boxes partially containing the corresponding category, and candidate boxes not containing the corresponding category at all. Through training of the initialized weakly supervised deep positioning network, the obtained optimized weakly supervised deep positioning network has the function of selecting the optimal candidate box from all candidate boxes of an input image.

[0081] For example, Figure 3 An example diagram of a weakly supervised positioning data augmentation multi-label long-tail distribution recognition method is shown. Figure 3 As shown in the diagram, if an image containing seagulls in Figure 3 is used as an input image, all possible candidate boxes are obtained from the seagull image. Among these candidate boxes, there may be image data containing all seagulls, image data containing part of the seagulls, and image data not containing seagulls at all.

[0082] S101-2, input each candidate box corresponding to the input image into the pre-trained sub neural network of the initialized weakly supervised deep positioning network to obtain the classification output of each candidate box and the detection output of each candidate box.

[0083] In implementation, for the input image I and all candidate boxes R = {r1, r2, … r n} corresponding thereto, and all categories C = {c1, c2, … c m}, the pre-trained sub neural network is used to obtain the classification output f(c, r) and the detection output g(c, r) of each candidate box r, where c is a category and r is a candidate box. The pre-trained sub neural network is used to calculate the output of the candidate box in the category direction and the number direction. For details, refer to the prior art, which will not be described here.

[0084] S101-3, calculate the classification output of each candidate box and the detection output of each candidate box to obtain the classification score of each candidate box and the detection score of each candidate box.

[0085] In implementation, based on the classification output f(c, r) and the detection output g(c, r) obtained in S101-2, the classification score s(r|c) of each candidate box and the detection score s(c|r) of each candidate box are further calculated and output, where s(r|c), s(c|r) ∈ R |C|×|R|, the classification score of each candidate box and the detection score of each candidate box are calculated according to the following formula:

[0086]

[0087] wherein r is a candidate box, c is a category of a candidate box, f(c, r) is a classification output of a candidate box r belonging to a category c, g(c, r) is a detection output of a candidate box r belonging to a category c; s(r|c) is a classification score of a candidate box r, s(c|r) is a detection score of a candidate box r.

[0088] S101-4, calculate the score of each candidate box according to the classification score of each candidate box and the detection score of each candidate box.

[0089] In specific implementation, the probability of a candidate box r belonging to a category c is calculated according to the following formula, and the score of the candidate box is calculated according to the following formula:

[0090] s(c, r) = s(c|r) s(r|c)

[0091] wherein s(r|c) is a classification score of a candidate box r, s(c|r) is a detection score of a candidate box r, and s(c, r) is a score of a candidate box r.

[0092] The score reflects the probability that the candidate box contains the corresponding category, and the greater the score, the greater the probability that the candidate box contains the corresponding category. Figure 3 Taking the seagull image in the above figure as an example, the training direction is that the greater the area of the candidate box containing the seagull, the higher the score.

[0093] S101-5, add the scores of all candidate boxes and calculate the weakly supervised deep positioning loss function.

[0094] In specific implementation, the score s(c, r) of each candidate box r in all candidate boxes R is added, and the weakly supervised deep positioning loss function is calculated according to the following formula:

[0095] φ(c) = ∑ r s(c, r); L img = -∑ c y(c) log φ(c);

[0096] wherein φ(c) is the sum of the scores s(c, r) of each candidate box r, L img is a weakly supervised deep positioning loss function, and y(c) is a class label corresponding to a category c.

[0097] S101-6, train and optimize the parameters of the initialized weakly supervised deep positioning network by optimizing the weakly supervised deep positioning loss function, to obtain an optimized weakly supervised deep positioning network.

[0098] The optimized weakly supervised depth localization network is obtained by inputting each input image in the input image set and all the corresponding candidate boxes into the initialized weakly supervised depth localization network and performing the training process of S101-1 to S101-6 until convergence. The optimized weakly supervised depth localization network can select the optimal candidate box with the highest score from all the candidate boxes of the input image as a detection box for data augmentation in the subsequent steps. The detection box is the candidate box with the highest probability of containing the corresponding category among all the candidate boxes. Meanwhile, the score corresponding to the detection box (the highest score of all candidate boxes) is output as the confidence of the input image.

[0099] Then, all the input images in the input image set are input into the optimized weakly supervised depth localization network to obtain the detection box of each input image and the confidence of each input image. The detection box of each input image is the optimal candidate box of the corresponding category of each input image, and the confidence of each input image is the optimal score of the detection box of each input image. For example, Figure 3 For example, an image containing a seagull is input into the optimized weakly supervised depth localization network as an input image to obtain a detection box containing the highest probability of a seagull and the score corresponding to the highest probability detection box as the confidence.

[0100] At this point, the detection box and the confidence corresponding to each input image in the input image set are obtained. Next, the detection box and the confidence corresponding to the image of the tail category are selected from the input images for data augmentation.

[0101] When performing step S102, first, the input image of the tail category in the input image is obtained as the first input image, and the detection box r of the first input image I obtained in step S101 is copied.

[0102] Subsequently, another input image in the same batch is randomly selected as the second input image I', and the copied detection box of the first input image is pasted and covered in a random position in the second input image. To prevent the copied and pasted detection box r from covering the original detection box r' in the second input image I' during the pasting and covering process, causing the subsequent classification network training to be unable to output the original category of the second input image, in an optional embodiment, a pre-judgment is performed in the second input image. First, a random position in the second input image is generated, and then it is judged whether the random position in the second input image contains the detection box of the second input image:

[0103] if the proportion of the area of the bounding box of the second input image in the random position in the second input image does not exceed the preset threshold, it is determined that the random position meets the condition, and the bounding box of the copied first input image is pasted and covered to the random position in the second input image;

[0104] if the proportion of the area of the bounding box of the second input image in the random position in the second input image exceeds the preset threshold, a new random position in the second input image is regenerated until the proportion of the area of the bounding box of the second input image in the random position in the second input image does not exceed the preset threshold, and then the bounding box of the copied first input image is pasted and covered to the random position in the second input image that finally meets the condition.

[0105] In the embodiment of the application, the default proportion of the area of the preset threshold is 50%, and the specific preset threshold is not limited in the application and can be set according to the situation.

[0106] After pasting and covering the bounding box of the copied first input image to the random position in the second input image that finally meets the condition, the category and position of the bounding box of the first input image are added to the annotation of the second input image, and the data-enhanced input image is obtained.

[0107] For example, the input image set in Figure 3 is taken as an example, the input image of the tail category "seagull image" in the input image set in Figure 3 is selected, the bounding box containing the seagull in the "seagull image" is obtained by inputting the "seagull image" into the optimized weakly supervised deep positioning network (the weakly supervised positioning module in Figure 3 ), and then the "seagull image" is taken as the first input image, the "human and dog image" in the input image set is taken as the second input image (the bounding box of which is the bounding box containing the human and the bounding box containing the dog), the bounding box containing the seagull in the "seagull image" is copied and pasted to the random position in the "human and dog image" that does not contain the bounding box of the human and the bounding box of the dog, and the image containing the seagull, the human and the dog is obtained as the data-enhanced image.

[0108] By copying and pasting the bounding box of the input image of the tail category positioned by the weakly supervised deep positioning network optimized by S101 to other input images, the number of the tail category (such as the "seagull" in Figure 3 ) in the input image set is effectively improved, thereby alleviating the problem of semantic imbalance caused by the long-tail distribution, and obtaining the data-enhanced input image.

[0109] It should be noted that the first input image is the image of the tail category in the input image set, and the second input image can be any image in the input image set, which can be a head image or a tail category.

[0110] After obtaining the data-augmented input image, step S103 is performed to use the data-augmented input image for training optimization of the multi-label classification neural network.

[0111] In the implementation of step S103, first, the image-augmented input image is used as training data to train the multi-label classification neural network; then, based on the multi-label classification neural network, the loss function of the multi-label classification neural network is calculated.

[0112] In the calculation of the loss function of the multi-label classification neural network, since the bounding box of the tail category input image is obtained by step S101, although the bounding box is the optimal candidate box with the highest score among all candidate boxes of the corresponding input image, the optimized weakly supervised deep positioning network may have errors in positioning and cannot accurately position the bounding box, or some bounding boxes may be incorrect. To prevent the data-augmented input image obtained from this part of the bounding box from having a negative impact on the multi-label classification neural network, in an implementable manner, the confidence s(c, r) of the bounding box of the tail category input image is introduced as a training weight into the loss function of the multi-label classification neural network by further improving the adaptive weight, and the confidence adaptive weight is introduced to optimize the loss function of the multi-label classification neural network, and the steps are as follows:

[0113] The confidence of the input image is obtained, and the confidence of the input image of the tail category is obtained. The confidence of the input image is obtained by inputting the input image and all candidate boxes corresponding to the input image into the optimized weakly supervised deep positioning network to obtain the corresponding optimal score of the bounding box of the input image.

[0114] The confidence of the input image of the tail category is used as a weight of the loss function of the multi-label classification neural network, and the loss function of the multi-label classification neural network is optimized to obtain an optimized loss function of the multi-label classification neural network, and the calculation method of the optimized loss function of the multi-label classification neural network is as follows:

[0115] L' cls =s(c, r) L cls

[0116] Where L' cls is the loss function of the optimized multi-label classification neural network, s(c, r) is the confidence of the bounding box r of the tail category input image, and L cls is the loss function of the multi-label classification neural network.

[0117] Finally, the parameters of the multi-label classification neural network are trained and optimized through a loss function of the optimized multi-label classification neural network, to obtain an optimized multi-label classification neural network.

[0118] The embodiment of the application provides a weakly supervised positioning data enhanced multi-label long-tail distribution identification method and product, input images and all candidate boxes corresponding to the input images are input into an optimized weakly supervised deep positioning network, to obtain detection boxes of the input images, the detection boxes of the input images are optimal candidate boxes of corresponding categories of the input images; the detection boxes of the input images of tail categories in the input images are copied and pasted to random positions in another input image in the same batch, to obtain data enhanced input images, and the data enhanced data images are used for training a multi-label classification neural network. The method of the application improves the number of detection boxes of tail categories in the input image set by copying and pasting the detection boxes of tail categories determined by the weakly supervised deep positioning network, thereby relieving the semantic imbalance problem of the long-tail distribution, and the optimized multi-label classification neural network is trained based on the data enhanced input images for multi-label long-tail distribution identification, and therefore the performance of the multi-label classification neural network in the multi-label identification scene of the long-tail distribution is improved.

[0119] Based on the same inventive concept, the embodiment of the application discloses a weakly supervised positioning data enhanced multi-label long-tail distribution identification device, Figure 4 A weakly supervised positioning data enhanced multi-label long-tail distribution identification device is shown, as shown in Figure 4 The device comprises:

[0120] A weakly supervised positioning module is configured to input input images and all candidate boxes corresponding to the input images into an optimized weakly supervised deep positioning network, to obtain detection boxes of the input images, and the detection boxes of the input images are optimal candidate boxes of corresponding categories of the input images.

[0121] A data enhancement module is configured to copy and paste the detection boxes of the input images of tail categories in the input images to random positions in another input image in the same batch, to obtain data enhanced input images, and the data enhanced data images are used for training a multi-label classification neural network.

[0122] The device further comprises:

[0123] A classification neural network training module is configured to use the image enhanced input images as training data to train a multi-label classification neural network.

[0124] The confidence acquisition module is configured to acquire a confidence of an input image of a tail category in the confidence of the input image, the confidence of the input image being an optimal score of a detection frame of the input image obtained by inputting the input image and all candidate frames corresponding to the input image into the optimized weakly supervised deep positioning network.

[0125] The weighting module is configured to take the confidence of the input image of the tail category as a weight of a loss function of the multi-label classification neural network, optimize the loss function of the multi-label classification neural network, and obtain an optimized loss function of the multi-label classification neural network.

[0126] The classification neural network optimization module is configured to train and optimize parameters of the multi-label classification neural network by using the optimized loss function of the multi-label classification neural network, and obtain the optimized multi-label classification neural network.

[0127] The weakly supervised positioning module includes:

[0128] The candidate frame acquisition submodule is configured to process the input image to obtain all candidate frames corresponding to the input image.

[0129] The candidate frame output submodule is configured to input each candidate frame corresponding to the input image into a pre-trained sub neural network of the initialized weakly supervised deep positioning network to obtain classification output of each candidate frame and detection output of each candidate frame.

[0130] The first candidate frame score submodule is configured to calculate the classification output of each candidate frame and the detection output of each candidate frame to obtain a classification score of each candidate frame and a detection score of each candidate frame.

[0131] The second candidate frame score submodule is configured to calculate a score of each candidate frame according to the classification score of each candidate frame and the detection score of each candidate frame.

[0132] The weakly supervised deep positioning loss function submodule is configured to add the scores of all candidate frames and calculate a weakly supervised deep positioning loss function.

[0133] The weakly supervised deep positioning network optimization submodule is configured to train and optimize parameters of the initialized weakly supervised deep positioning network by optimizing the calculation of the weakly supervised deep positioning loss function, and obtain the optimized weakly supervised deep positioning network.

[0134] The detection frame output sub-module is configured to input all input images in the input image set into the optimized weakly supervised deep positioning network to obtain a detection frame of each input image and a confidence of each input image, the detection frame of each input image being an optimal candidate frame of a corresponding category of each input image, and the confidence of each input image being an optimal score of the detection frame of each input image.

[0135] The data augmentation module includes:

[0136] The copying sub-module is configured to obtain an input image of a tail category in the input image as a first input image, and select a detection frame of the first input image for copying.

[0137] The pasting sub-module is configured to randomly select another input image in the same batch as a second input image, and paste the copied detection frame of the first input image to a random position in the second input image.

[0138] The data augmentation sub-module is configured to add a category and a position of the detection frame of the first input image to a label of the second input image to obtain the data-augmented input image.

[0139] The pasting sub-module further includes:

[0140] The first random position generation sub-unit is configured to generate a random position in the second input image.

[0141] The judgment sub-unit is configured to judge whether the random position in the second input image contains a detection frame of the second input image.

[0142] The final random position generation sub-unit is configured to, if an area proportion of the detection frame of the second input image contained in the random position in the second input image exceeds a preset threshold, regenerate a new random position in the second input image until the area proportion of the detection frame of the second input image contained in the random position in the second input image does not exceed the preset threshold.

[0143] The pasting sub-unit is configured to paste the copied detection frame of the first input image to the random position in the second input image.

[0144] Based on the same inventive concept, the embodiments of the present application disclose an electronic device, Figure 5 The electronic device disclosed by the embodiments of the present application is shown in a schematic diagram as Figure 5As shown, the electronic device 100 includes a memory 110 and a processor 120, the memory 110 and the processor 120 are connected through a bus in communication, and the memory 110 stores a computer program, the computer program is executable on the processor 120 to realize the steps in the weakly supervised positioning data enhanced multi-label long tail distribution identification method disclosed in the embodiments of the application.

[0145] Based on the same inventive concept, the embodiments of the application disclose a computer readable storage medium, which stores a computer program / instruction, and the computer program / instruction realizes the steps in the weakly supervised positioning data enhanced multi-label long tail distribution identification method disclosed in the embodiments of the application when executed by a processor.

[0146] Based on the same inventive concept, the application also provides a computer program product, which includes a computer program / instruction, and the computer program / instruction realizes the steps in the weakly supervised positioning data enhanced multi-label long tail distribution identification method disclosed in the embodiments of the application when executed by a processor.

[0147] Each of the embodiments in the specification is described in a progressive manner, and each embodiment focuses on the difference from other embodiments, and the same and similar parts between each embodiment can be referred to each other.

[0148] The embodiments of the application are described with reference to flowcharts and / or block diagrams of the methods, devices, electronic devices and computer program products according to the embodiments of the application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing terminal equipment to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal equipment produce a machine that implements the steps in the flowcharts Figure 1 The functions specified in one flow or multiple flows and / or blocks Figure 1 The functions specified in one flow or multiple flows and / or blocks

[0149] These computer program instructions can also be stored in a computer readable storage medium that can guide the computer or other programmable data processing terminal equipment to work in a specific way, so that the instructions stored in the computer readable storage medium produce a product including instruction devices, which realize the steps in the flowcharts Figure 1 The functions specified in one flow or multiple flows and / or blocks Figure 1 The functions specified in one flow or multiple flows and / or blocks

[0150] These computer program instructions can also be loaded into a computer or other programmable data processing terminal device, so that a series of operational steps are performed on the computer or other programmable terminal device to generate a computer-implemented process, so that the instructions executed on the computer or other programmable terminal device provide a process for implementing the functions specified in the flowchart Figure 1 one flow or multiple flows and / or the functions specified in the block Figure 1 one block or multiple blocks.

[0151] Although the preferred embodiments of the present application have been described, those skilled in the art can make additional changes and modifications to these embodiments once they know the basic inventive concept. Therefore, the appended claims are intended to cover all changes and modifications falling within the scope of the embodiments of the present application.

[0152] Finally, it should also be noted that, in this document, relational terms such as first and second and the like can only be used to distinguish one entity or operation from another entity or operation, without necessarily requiring or implying any such actual relationship or order between or among the entities or operations. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or terminal device. An element proceeded by "comprises a... " does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or terminal device that comprises the element.

[0153] The above provides a weakly supervised positioning data enhanced multi-label long-tail distribution identification method and product, and the principle and implementation manner of the present application are described by using specific examples. The above description of the embodiments is only used to help understand the method and core idea of the present application. Meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manner and application range can be changed, and the above description of the present application should not be understood as a limitation.

Claims

1. A weakly supervised localization data augmentation multi-label long-tail distribution identification method, characterized in that, The application relates to an image processing method and device. The input image and all candidate frames corresponding to the input image are input into an optimized weakly supervised depth positioning network to obtain a detection frame of the input image, wherein the detection frame of the input image is the optimal candidate frame of the input image corresponding to a category; The detection frame of the input image of the tail category in the input image is copied and pasted into a random position in another input image in the same batch to obtain a data-enhanced input image, wherein the data-enhanced data image is used for training a multi-label classification neural network. The detection frame of the input image is obtained by: processing the input image to obtain all candidate frames corresponding to the input image; inputting each candidate frame corresponding to the input image into a pre-trained sub neural network of an initialized weakly supervised depth positioning network to obtain a classification output of each candidate frame and a detection output of each candidate frame; calculating the classification output of each candidate frame and the detection output of each candidate frame to obtain a classification score of each candidate frame and a detection score of each candidate frame; calculating a score of each candidate frame according to the classification score of each candidate frame and the detection score of each candidate frame; adding the scores of all candidate frames and calculating a weakly supervised depth positioning loss function; training and optimizing the parameters of the initialized weakly supervised depth positioning network by optimizing the calculated weakly supervised depth positioning loss function to obtain an optimized weakly supervised depth positioning network; inputting all input images in an input image set into the optimized weakly supervised depth positioning network to obtain a detection frame of each input image and a confidence of each input image, wherein the detection frame of each input image is the optimal candidate frame of each input image corresponding to a category, and the confidence of each input image is the optimal score of the detection frame of each input image.

2. The weakly supervised localization data-augmented multi-label long-tail distribution identification method of claim 1, wherein, After obtaining the data-enhanced input image, the following steps are included: training a multi-label classification neural network by taking the data-enhanced input image as training data; obtaining the confidence of the input image of the tail category, wherein the confidence of the input image is the optimal score of the detection frame of the input image obtained by inputting the input image and all candidate frames corresponding to the input image into the optimized weakly supervised depth positioning network; taking the confidence of the input image of the tail category as a weight of a loss function of the multi-label classification neural network to optimize the loss function of the multi-label classification neural network to obtain an optimized loss function of the multi-label classification neural network; training and optimizing the parameters of the multi-label classification neural network by using the optimized loss function of the multi-label classification neural network to obtain an optimized multi-label classification neural network.

3. The weakly supervised localization data-augmented multi-label long-tail distribution identification method of claim 1, wherein, The detection frame of the input image of the tail category in the input image is copied and pasted into a random position in another input image in the same batch to obtain a data-enhanced input image, including: obtaining the input image of the tail category in the input image as a first input image and copying the detection frame of the first input image; Randomly select another input image in the same batch as a second input image, and paste the copied detection box of the first input image to a random position in the second input image; Add the category and position of the detection box of the first input image to the label of the second input image to obtain the data-augmented input image.

4. The weakly supervised localization data-augmented multi-label long-tail distribution identification method of claim 1, wherein, Calculate the classification output of each candidate box and the detection output of each candidate box to obtain the classification score of each candidate box and the detection score of each candidate box, which are calculated according to the following formula: ； ； where r is a candidate box, c is a class of a candidate box, is a classification output that candidate box r belongs to class c, is a detection output that candidate box r belongs to class c; is a classification score of candidate box r, is a detection score of candidate box r.

5. The weakly supervised localization data-augmented multi-label long-tail distribution identification method of claim 3, wherein, Paste the copied detection box of the first input image to a random position in the second input image, including: Generating a random position in the second input image; Judging whether the random position in the second input image contains the detection box of the second input image; If the area proportion of the detection box of the second input image contained in the random position in the second input image exceeds a preset threshold, a new random position in the second input image is generated until the area proportion of the detection box of the second input image contained in the random position in the second input image does not exceed the preset threshold; Paste the copied detection box of the first input image to the random position in the second input image.

6. A weakly supervised localization data augmentation multi-label long-tail distribution identification apparatus, characterized in that, Including: a weakly supervised positioning module, configured to input an input image and all candidate boxes corresponding to the input image into an optimized weakly supervised deep positioning network to obtain a detection box of the input image, the detection box of the input image being an optimal candidate box of a corresponding category of the input image; a data augmentation module, configured to copy a detection box of an input image of a tail category in the input image and paste it to a random position in another input image in the same batch to obtain a data-augmented input image, the data-augmented data image being used for training a multi-label classification neural network; The weakly supervised positioning module is specifically configured to: process the input image to obtain all candidate boxes corresponding to the input image; input each candidate box corresponding to the input image into a pre-trained sub-neural network of an initialized weakly supervised deep positioning network to obtain a classification output of each candidate box and a detection output of each candidate box; calculate the classification output of each candidate box and the detection output of each candidate box to obtain a classification score of each candidate box and a detection score of each candidate box; calculate a score of each candidate box according to the classification score of each candidate box and the detection score of each candidate box; add the scores of all candidate boxes and calculate a weakly supervised deep positioning loss function; train and optimize parameters of the initialized weakly supervised deep positioning network by optimizing the calculated weakly supervised deep positioning loss function to obtain an optimized weakly supervised deep positioning network; input all input images in an input image set into the optimized weakly supervised deep positioning network to obtain a detection box of each input image and a confidence of each input image, the detection box of each input image being an optimal candidate box of a corresponding category of each input image, and the confidence of each input image being an optimal score of the detection box of each input image.

7. An electronic device, comprising: A computer program product comprising a memory, a processor and a computer program stored on the memory, the processor executing the computer program to implement the steps of the weakly supervised localization data augmentation multi-label long-tail distribution identification method of any of claims 1-5.

8. A computer readable storage medium having stored thereon computer programs / instructions, characterized in that, The computer program / instructions, when executed by the processor, implement the steps of the weakly supervised localization data augmentation multi-label long-tail distribution identification method of any of claims 1-5.

9. A computer program product comprising computer programs / instructions, characterized in that, The computer program / instructions, when executed by the processor, implement the steps of the weakly supervised localization data augmentation multi-label long-tail distribution identification method of any of claims 1-5.

Images