Weakly supervised object detection method based on misclassification correction and related device

Abstract

The application discloses a weakly supervised object detection method and system based on misclassification correction, an electronic device and a storage medium, and the method comprises the following steps: extracting the region features of the target candidate position by using a convolutional neural network; assigning the category labels of the extracted region features by using a classifier; designing a misclassification correction driven label assignment module according to the confidence difference between categories, so as to identify the misclassification and correct it, and reassign the labels for the training of the object detection model. The method of the application can correct the error category labels generated in the training stage, and significantly improve the classification performance of the weakly supervised object detection method.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision, and particularly relates to a weakly supervised object detection method based on misclassification correction and related equipment. BACKGROUND

[0002] Under the promotion of national policy, artificial intelligence technology accelerates to penetrate into various industries, and especially shows broad prospects in the fields of industrial manufacturing, intelligent transportation, medical health and cultural education. As an important part of artificial intelligence, computer vision, especially the advanced learning and cognitive technology represented by visual object detection, has become one of the key driving forces to promote the intelligent upgrading of industry.

[0003] Although the visual object detection model based on deep learning has made great progress in accuracy, its high dependence on large-scale artificial annotation data has become a significant bottleneck. This annotation process is time-consuming and labor-intensive, and the cost is high, which directly limits the efficient use of large-scale data and seriously restricts the development potential of related technologies.

[0004] In order to reduce the dependence on high-cost artificial annotation, researchers have proposed weakly supervised visual object detection technology. However, this kind of method often ignores a key problem in design: misclassification. Even if the detector can accurately locate the target area, it cannot accurately judge its category, thereby affecting the improvement of classification performance. SUMMARY

[0005] In view of the above problems, the present application provides a weakly supervised object detection method based on misclassification correction, a system, an electronic device and a storage medium, which aims to significantly improve the classification performance of the weakly supervised object detection method by correcting the error class labels generated in the training stage.

[0006] According to a first aspect of the embodiments of the present disclosure, a weakly supervised object detection method based on misclassification correction is provided, and the method comprises the following steps:

[0007] extracting the region features of the target candidate position by using a convolutional neural network;

[0008] assigning the extracted region features to the category labels by using a classifier;

[0009] According to the confidence difference between categories, a misclassification correction driven label assignment module is designed to identify and correct the misclassification, and the label is re-assigned for training of the object detection model.

[0010] In some embodiments, the convolutional neural network is based on a VGGNet neural network, and a spatial pyramid pooling layer is added at the output end of the VGGNet to generate convolutional features corresponding to the multi-scale combined target candidate regions; and two fully connected layers are added at the output end of the spatial pyramid pooling layer of the VGGNet to generate feature vectors required by the classifier.

[0011] In some embodiments, the region features extracted are assigned class labels by the classifier, and the training process includes the following steps:

[0012] The region feature vectors are respectively input into the two fully connected layers of the classifier to obtain a first matrix and a second matrix;

[0013] The first matrix and the second matrix are respectively input into a class latitude Softmax layer and a region dimension Softmax layer to obtain a third matrix representing the confidence score of each candidate region belonging to each class and a fourth matrix representing the normalized contribution of each candidate region to the image containing each class;

[0014] An element-wise multiplication operation is performed on the third matrix and the fourth matrix to generate a candidate region score matrix;

[0015] The candidate region score matrix is added along the candidate region dimension to obtain the confidence of the image containing each class.

[0016] The confidence of the image containing each class is input into a multi-class cross-entropy loss function together with the true class.

[0017] In some embodiments, during the training of the misclassification correction driven label assignment module:

[0018] The following steps are iteratively performed until the maximum number of iterations is reached:

[0019] The region feature vector is input into branch k of the Softmax layer containing the fully connected layer and the class dimension to obtain a candidate region score matrix

[0020] From the candidate region score of the previous branch , the candidate region with the highest confidence in each positive class c is selected The corresponding confidence is The candidate region with the highest confidence in all negative classes c' is selected The corresponding confidence is

[0021] If It is considered that a potential misclassification of the class pair c-c' has occurred;

[0022] If the training process is less than 30%, when a potential misclassification of class cc′ occurs, it is recorded in the misclassification memory B and b is updated. cc′ =b cc′ +1, where b cc′ The number of times each category cc' appears; construct a positive class memory P to record the number of times each category appears as a positive class in previous iterations;

[0023] If the training process reaches 30%, calculate the frequency of potential misclassification for each class pair: F = B / P; sort the elements in F in descending order to obtain the sorted class pair frequency vector o, where o i ≥o i+1 And the corresponding ranking matrix S, where s cc′ Represent the frequency ranking of category cc′; calculate the difference between neighboring elements to obtain the frequency difference vector d; find the largest element in d and record its ranking N;

[0024] If the training process exceeds three-tenths, when a potential misclassification pair cc′ occurs, if condition s is satisfied... cc′ If the value is less than or equal to N, then the candidate region with the highest negative class score will be obtained. Mark as a positive example; otherwise, the positive example is the region that receives the highest positive class score.

[0025] The intersection-union ratio (IUU) of other candidate regions with the positive examples is calculated. Candidate regions with an IUU greater than 0.5 are also selected as positive examples, regions with an IUU less than 0.1 are ignored, and the remaining regions are set as negative examples, which are used as supervision information θ. k ;

[0026] Will and θ k The input is fed into the multi-class cross-entropy loss function;

[0027] Calculation after iteration The average value is used as the final prediction result for the confidence level of each candidate region.

[0028] According to a second aspect of the present disclosure, a weakly supervised target detection system based on misclassification correction is provided, the system comprising:

[0029] The region feature acquisition unit is used to extract region features of candidate target locations using a convolutional neural network.

[0030] The category label acquisition unit is used to assign category labels to the extracted region features through a classifier.

[0031] The label assignment module design unit is configured to design a misclassification correction driven label assignment module according to the confidence difference between categories, to identify misclassification and correct it, and to reassign labels for target detection model training.

[0032] In some embodiments, the convolutional neural network in the region feature acquisition unit is based on a VGGNet neural network, and a spatial pyramid pooling layer is added at the output end of the VGGNet to generate convolutional features corresponding to the multi-scale combined target candidate regions; and two fully connected layers are added at the output end of the spatial pyramid pooling layer of the VGGNet to generate feature vectors required by the classifier.

[0033] In some embodiments, the category label acquisition unit assigns the category labels to the extracted region features through the classifier, and the training process includes the following steps:

[0034] The region feature vectors are respectively input into the two fully connected layers of the classifier to obtain a first matrix and a second matrix;

[0035] The first matrix and the second matrix are respectively input into a category latitude Softmax layer and a region dimension Softmax layer to obtain a third matrix representing the confidence scores of each candidate region belonging to each category and a fourth matrix representing the normalized contribution of each candidate region to the image containing each category;

[0036] Element-wise multiplication is performed on the third matrix and the fourth matrix to generate a candidate region score matrix;

[0037] The candidate region score matrix is added along the candidate region dimension to obtain the confidence of the image containing each category;

[0038] The confidence of the image containing each category is input into a multi-class cross-entropy loss function together with the true category.

[0039] In some embodiments, in the label assignment module design unit, during the training of the misclassification correction driven label assignment module:

[0040] The following steps are iteratively performed until the maximum number of iterations is reached:

[0041] The region feature vector is input into branch k of the Softmax layer containing the fully connected layer and the category dimension to obtain a candidate region score matrix

[0042] From the candidate region score of the previous branch , the candidate region with the highest confidence in each positive class c is selected The corresponding confidence is The candidate region with the highest confidence in all negative classes c' is selected The corresponding confidence is

[0043] If regard the class pair c-c' as a potential misclassification case;

[0044] If the training process is less than 3 / 10, when a potential misclassification case occurs in the class pair c-c', record it in the misclassification memory B and update b cc′ = b cc′ + 1, where b cc′ represents the number of times the class pair c-c' appears; construct a positive class memory P to record the number of times each class appears as a positive class in previous iterations;

[0045] If the training process reaches 3 / 10, calculate the frequency of potential misclassification of each class pair: F = B / P; arrange the elements in F in descending order to obtain the sorted class pair frequency vector o, where o i ≥ o i+1 , and the corresponding ranking matrix S, where s cc′ represents the ranking of the frequency of the class pair c-c'; calculate the difference between adjacent elements to obtain the frequency difference vector d; find the largest element in d and record its rank N;

[0046] If the training process exceeds 3 / 10, when a potential misclassification pair c-c' appears, if the condition s cc′ ≤ N is met, the candidate region with the highest negative class score is marked as a positive example; otherwise, the positive example is the region with the highest positive class score ;

[0047] Calculate the intersection-over-union of other candidate regions with the positive example, and the candidate region with an intersection-over-union greater than 0.5 is also selected as a positive example, the region with an intersection-over-union less than 0.1 is ignored, and the remaining regions are set as negative examples as supervision information θ k ;

[0048] Input and θ k to the multi-class cross-entropy loss function;

[0049] After the iteration is completed, calculate the average value of as the final prediction result of the confidence of each candidate region.

[0050] According to a third aspect of the embodiments of the present disclosure, an electronic device is provided, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the weakly supervised object detection method based on misclassification correction when executing the program.

[0051] According to a fourth aspect of the embodiments of the present disclosure, a non-temporary computer readable storage medium is provided, and the storage medium stores computer instructions, and the instructions are executed by a processor to implement the steps of the weakly supervised object detection method based on misclassification correction.

[0052] The weakly supervised object detection method, system, electronic device and storage medium based on misclassification correction provided by the embodiments of the present disclosure are provided, the convolutional neural network is used to extract the region features corresponding to the target candidate position; the classifier classifies the extracted region features, and initially assigns a class label to each candidate region; the label assignment module driven by misclassification correction is used to identify and correct the misclassification, and the re-allocated label is used for model training, so that the features of each class are learned more accurately, which provides a feasible solution to solve the misclassification problem and has great application value. The method of the present application significantly improves the classification performance of the weakly supervised object detection method by correcting the error class labels generated in the training stage.

[0053] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0054] The accompanying drawings, which are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present application and, together with the specification, serve to explain the principles of the present application

[0055] Figure 1 is a weakly supervised object detection method flowchart based on misclassification correction in embodiments of the present application;

[0056] Figure 2 is a weakly supervised object detection method flowchart based on misclassification correction in embodiments of the present application;

[0057] Figure 3 is a weakly supervised object detection system structure diagram based on misclassification correction in embodiments of the present application;

[0058] Figure 4 is a schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0059] The present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, but not to limit the present application. In addition, it should be noted that, for the convenience of description, only the parts related to the present application are shown in the drawings, but not all the structures.

[0060] Before any embodiments of the application are explained in detail, it is to be understood that the example embodiments are described in relation to a process or method which is depicted as a flow diagram. Although the process is described as sequential, many of the steps can be performed in parallel, concurrently or even simultaneously. In addition, the order of the steps can be re-arranged. The process can terminate when its operations are completed, but can also terminate in the middle of one or more steps. The process can correspond in whole or in part to a method, a function, a procedure, a subroutine, a subprogram, etc.

[0061] Embodiments of the present application are directed to a weakly supervised object detection method and system based on misclassification correction, an electronic device and a storage medium, and provide the following embodiments:

[0062] As shown in the figure, the weakly supervised object detection method based on misclassification correction comprises the following steps: Figure 1

[0063] S1, extracting region features of target candidate positions by using a convolutional neural network;

[0064] S2, assigning a class label to the extracted region features by using a classifier; specifically, classifying the extracted region features by using the classifier, and initially assigning a class label to each candidate region;

[0065] S3, designing a misclassification correction driven label assignment module according to the confidence difference between classes, to identify and correct misclassification, and reassign labels for training of the object detection model. Specifically, a misclassification correction driven label assignment module is designed according to the confidence difference between classes, to identify and correct misclassification, and reassign labels for training of the model, so as to more accurately learn the features of each class.

[0066] Overall, as shown in the figure, the convolutional neural network is based on the VGGNet neural network, and a spatial pyramid pooling layer is added at the output end of the VGGNet, for generating convolutional features corresponding to multi-scale combined target candidate regions; and two fully connected layers are added at the output end of the spatial pyramid pooling layer of the VGGNet, for generating feature vectors required by the classifier. Figure 2

[0067] In one specific embodiment, the convolutional neural network in S1 is modified from the original version of VGGNet (Simonyan, Karen, and Andrew Zisserman. "Very deep convolutional networks for large-scale image recognition." ArXiv. 2014.).

[0068] ​​Specifically, the specific establishment method of the convolutional neural network is as follows:

[0069] 1) Adding a spatial pyramid pooling layer at the output end of the VGGNet to generate the convolutional features corresponding to the target candidate region provided by the multi-scale combination grouping method;

[0070] 2) Adding two fully connected layers at the output end of the spatial pyramid pooling layer.

[0071] The establishment of the convolutional neural network realizes the generation of the target candidate region feature vector, and provides an input for the classifier in S2.

[0072] In S2, the classifier is used to assign the class labels to the extracted region features, and the training process includes the following steps:

[0073] The region feature vectors are respectively input into the two fully connected layers of the classifier to obtain a first matrix and a second matrix;

[0074] The first matrix and the second matrix are respectively input into a class latitude Softmax layer and a region dimension Softmax layer to obtain a third matrix representing the confidence score of each candidate region belonging to each class and a fourth matrix representing the normalized contribution of each candidate region to the image containing each class;

[0075] The third matrix and the fourth matrix are subjected to an element-by-element multiplication operation to generate a candidate region score matrix;

[0076] The candidate region score matrix is added along the candidate region dimension to obtain the confidence of the image containing each class.

[0077] The confidence of the image containing each class is input into a multi-class cross-entropy loss function together with the true class.

[0078] In one specific embodiment, the classifier is modified from the original version of WSDDN (Bilen, Hakan, and Andrea Vedaldi. "Weakly supervised deep detection networks." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016.).

[0079] The specific establishment method of the classifier is as follows:

[0080] S21, the feature vectors generated in S1 are respectively input into two fully connected layers to generate two matrices X cls and X det ;

[0081] S22, input the two matrices generated in S21 into a Softmax layer in the category dimension and a Softmax layer in the region dimension respectively to obtain two matrices σ(X cls ) and σ(X det ), which respectively represent the confidence score of each candidate region belonging to each category and the normalized contribution of each candidate region to the image containing each category;

[0082] S23, perform an element-wise multiplication operation on the two matrices generated in S22 to generate a candidate region score matrix

[0083] S24, add the candidate region score matrix in S23 along the candidate region dimension to obtain the confidence of the image containing each category;

[0084] S25, input the image category confidence generated in S24 and the category true value into a multi-class cross-entropy loss function;

[0085] If in the training stage, S21 to S25 are performed; if in the application stage, S21 to S25 are skipped.

[0086] Thus, the establishment of the classifier is completed, and the classifier for classifying the image is trained to provide supervision information for the next branch.

[0087] The misclassification correction driven label assignment module in S3 includes three branches with the same structure

[0088] The construction method of the misclassification correction driven label assignment module is as follows:

[0089] S31, input the feature vector generated in step 1) into branch k with one fully connected layer and one Softmax layer in the category dimension to generate a candidate region score matrix

[0090] S32, from the candidate region score in the last branch, select the candidate region with the highest confidence in each positive class c The corresponding confidence is Select the candidate region with the highest confidence in all negative classes The corresponding confidence is If the difference between the two satisfies regard it as a potential misclassification of the class pair c-c';

[0091] Step S33, if the training process is less than three tenths, when the class pair c-c' has a potential misclassification, record it in the misclassification memory B: b cc′ = bcc′ +1, where b cc′ The frequency of occurrence of class pair cc′ is represented by the number of times it occurs. A positive class memory P is constructed to record the number of times each class appears as a positive class in previous iterations. If the training process reaches 30%, the frequency of potential misclassification for each class pair is calculated: F = B / P. Then, the elements in F are sorted in descending order to obtain the sorted class pair frequency vector o, where o i ≥o i+1 And the corresponding ranking matrix S, where s cc′ This represents the frequency ranking of category pair cc′; calculate the difference between neighboring elements to obtain the frequency difference vector d; find the largest element in d and record its ranking N. If the training process exceeds three-tenths, when a potential misclassification pair cc′ occurs, if condition s is satisfied... cc′ If the value is less than or equal to N, then the candidate region with the highest negative class score will be obtained. Mark as a positive example; otherwise, the positive example is the region that receives the highest positive class score.

[0092] S34. Calculate the intersection-union ratio (IUU) of other candidate regions with the positive examples. Candidate regions with an IUU greater than 0.5 are also selected as positive examples, regions with an IUU less than 0.1 are ignored, and the remaining regions are set as negative examples, serving as supervision information θ. k ;

[0093] S35, will and θ k The input is fed into the multi-class cross-entropy loss function;

[0094] S36. If it is in the training phase, repeat S31 to S35 3 times; if it is in the application phase, repeat only step S31 3 times.

[0095] S37, Calculation The average value is used as the final prediction result for the confidence level of each candidate region.

[0096] This completes the creation of the misclassification correction-driven label allocation module.

[0097] Another embodiment illustrates a weakly supervised target detection system based on misclassification correction, such as... Figure 3 As shown, system 300 includes:

[0098] The region feature acquisition unit 310 is used to extract region features of the target candidate location using a convolutional neural network.

[0099] The category label acquisition unit 320 is used to assign category labels to the extracted region features through a classifier.

[0100] The label assignment module design unit 330 is configured to design a misclassification correction driven label assignment module according to the confidence difference between categories, to identify misclassification and correct it, and to reassign labels for target detection model training.

[0101] The convolutional neural network in the region feature acquisition unit 310 is based on a VGGNet neural network, and a spatial pyramid pooling layer is added at the output end of the VGGNet, to generate convolutional features corresponding to multi-scale combined target candidate regions; and two fully connected layers are added at the output end of the spatial pyramid pooling layer of the VGGNet, to generate feature vectors required by the classifier.

[0102] The category label acquisition unit 320 assigns a category label to the extracted region features through the classifier, and the training process includes the following steps:

[0103] The region feature vectors are respectively input into the two fully connected layers of the classifier, to obtain a first matrix and a second matrix;

[0104] The first matrix and the second matrix are respectively input into a category latitude Softmax layer and a region dimension Softmax layer, to obtain a third matrix representing the confidence score of each candidate region belonging to each category, and a fourth matrix representing the normalized contribution of each candidate region to the image containing each category;

[0105] The third matrix and the fourth matrix are subjected to an element-by-element multiplication operation, to generate a candidate region score matrix;

[0106] The candidate region score matrix is added along the candidate region dimension, to obtain the confidence of the image containing each category;

[0107] The confidence of the image containing each category is input into a multi-class cross-entropy loss function together with the true category.

[0108] In the label assignment module design unit 330, during the training of the misclassification correction driven label assignment module:

[0109] The following steps are iteratively performed until the maximum number of iterations is reached:

[0110] The region feature vector is input into branch k of the Softmax layer containing the fully connected layer and the category dimension, to obtain a candidate region score matrix k .

[0111] From the candidate region score k-1 of the previous branch, the candidate region with the highest confidence in each positive class c is selected , and the corresponding confidence is The candidate region with the highest confidence in all negative classes c' is selected , and the corresponding confidence is

[0112] If a potential misclassification case of class pair c-c' occurs;

[0113] If the training process is less than 3 / 10, when a potential misclassification case of class pair c-c' occurs, record it in the misclassification memory B to update b cc′ = b cc′ +1, where b cc′ represents the number of times class pair c-c' occurs; construct a positive class memory P to record the number of times each class appears as a positive class in previous iterations;

[0114] If the training process reaches 3 / 10, calculate the frequency of potential misclassification of each class pair: F = B / P; arrange the elements in F in descending order to obtain the sorted class pair frequency vector o, where o i ≥ o i+1 , and the corresponding ranking matrix S, where s cc′ represents the ranking of the frequency of class pair c-c'; calculate the difference between adjacent elements to obtain the frequency difference vector d; find the largest element in d and record its rank N;

[0115] If the training process exceeds 3 / 10, when a potential misclassification pair c-c' occurs, if the condition s cc′ ≤ N is met, the candidate region with the highest negative class score is marked as a positive example; otherwise, the positive example is the region with the highest positive class score

[0116] Calculate the intersection-over-union of other candidate regions with the positive example, and the candidate region with an intersection-over-union greater than 0.5 is also selected as a positive example, the region with an intersection-over-union less than 0.1 is ignored, and the remaining regions are set as negative examples as supervision information θ k ;

[0117] After the iteration is completed, φ k and θ k are input into a multi-class cross-entropy loss function;

[0118] After the iteration is completed, the average value of φ k is calculated as the final prediction result of the confidence of each candidate region.

[0119] In addition to the above module, the unit 300 can also include other components, however, since these components are irrelevant to the content of the embodiments of the present disclosure, their illustrations and descriptions are omitted here.

[0120] ​The other specific working process of the weakly supervised object detection system 300 based on misclassification correction is described with reference to the above-described weakly supervised object detection method based on misclassification correction, and will not be described again.

[0121] Another embodiment is used to illustrate that the system of the present application can also be implemented by means of Figure 4 the architecture of the computing device shown. Figure 4 The architecture of the computing device is shown. As shown in Figure 4 the computer system 410, the system bus 430, the one or more CPUs 440, the input / output 420, the memory 450, etc. The memory 450 can store various data or files used by the computer processing and / or communication and the program instructions executed by the CPU, including the program instructions of the embodiment of the weakly supervised object detection method based on misclassification correction. Figure 4 The architecture shown is only exemplary, and when implementing different devices, one or more components in Figure 4 may be adjusted according to actual needs. The memory 450 as a kind of computer readable storage medium can be used to store software programs, computer executable programs and modules, such as the program instructions / modules corresponding to the weakly supervised object detection method based on misclassification correction in the embodiment of the present application (for example, the region feature acquisition unit 310, the class label acquisition unit 320 and the label assignment module design unit 330 in the weakly supervised object detection system 300 based on misclassification correction). The one or more CPUs 440 execute various functional applications and data processing of the system of the present application by running the software programs, instructions and modules stored in the memory 450, that is, implement the weakly supervised object detection method based on misclassification correction described above, which includes the following steps:

[0122] extracting the region features of the target candidate position using a convolutional neural network;

[0123] assigning the extracted region features to the class labels by the classifier;

[0124] According to the confidence difference between the classes, a misclassification correction driven label assignment module is designed to identify and correct the misclassification and reassign the labels for the training of the object detection model.

[0125] Of course, the processor of the server provided by the embodiment of the present application is not limited to performing the method operations as described above, but can also perform the related operations in the weakly supervised object detection method based on misclassification correction provided by any embodiment of the present application.

[0126] The memory 450 can include primarily a program storage area and a data storage area, where the program storage area can store an operating system, application programs, etc. required by at least one function; the data storage area can store data, etc. created according to usage of the terminal. Furthermore, the memory 450 can include a high-speed random access memory, and also can include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid state memory device. In some examples, the memory 450 can further include a memory that is remotely located with respect to the one or more CPUs 440, which can be connected to the device through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0127] The input / output 420 can be used to receive inputted digital or character information, and to generate key signal inputs related to user settings and function controls of the device. The input / output 420 can further include a display device, such as a display screen.

[0128] The embodiment of the present application further provides a non-transitory computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the weakly supervised object detection method based on misclassification correction described in the above embodiment. The computer readable storage medium of the embodiment of the present application can adopt any combination of one or more computer readable media. The computer readable medium can be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination thereof. More specific examples (non-exhaustive list) of the computer readable storage medium include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, the computer readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus or device.

[0129] The computer readable signal medium can include a data signal that is propagated in baseband or that is propagated as part of a carrier wave. Such propagated data signals can take a wide variety of forms, including but not limited to electro-magnetic signals, optical signals, or any suitable combination thereof. Computer readable signal media can also be any computer readable medium that is not a storage medium, and that can communicate, propagate or transport programming for use by or in connection with an instruction execution system, apparatus or device.

[0130] The program code contained on the storage medium can be transmitted in any suitable format, including but not limited to wireless, wire line, optical cable, RF, etc., or any suitable combination of the foregoing.

[0131] In addition, other specific working processes of the non-temporary computer readable storage medium are described with reference to the above description of the weakly supervised object detection method based on misclassification correction, and will not be repeated.

[0132] In summary, the technical solutions provided by the above embodiments are a weakly supervised object detection method, system, electronic device and storage medium based on misclassification correction. The convolutional neural network is used to extract the region features corresponding to the target candidate position. The classifier classifies the extracted region features, and preliminarily assigns each candidate region to a class label. The misclassification correction driven label assignment module is used to identify and correct the misclassification, and reassign the label for model training, so as to more accurately learn the features of each class, and provide a feasible solution to solve the misclassification problem, which has great application value. The method of the present application corrects the error class label generated in the training stage, and significantly improves the classification performance of the weakly supervised object detection method.

[0133] In this document, the terms "comprise", "contain", or any other variant thereof are intended to cover non-exclusive inclusion, so that the steps, methods comprising a series of elements not only include those elements, but also include other elements not explicitly listed, or include elements inherent to such steps or methods.

[0134] The above is a further detailed description of the present application in combination with specific preferred embodiments, and cannot be considered as limiting the specific implementation of the present application to these descriptions. For ordinary skilled persons in the technical field to which the present application belongs, without departing from the concept of the present application, a number of simple deductions or replacements can be made, which should be considered as belonging to the protection scope of the present application.

Claims

1. A weakly supervised object detection method based on misclassification correction, characterized in that, The method comprises the following steps: For the input image, a region feature of a target candidate position is extracted by using a convolutional neural network; A classifier is used to assign a class label to the extracted region feature; According to the confidence difference between classes, a misclassification correction driven label assignment module is designed to identify and correct misclassification and reassign labels for target detection model training; During the training of the misclassification correction driven label assignment module: The following steps are iteratively performed until the maximum number of iterations is reached: inputting the region feature vector into a branch containing a fully connected layer and a softmax layer of class dimension , to obtain a candidate region score matrix ; from the candidate regions of the previous branch for each positive class the candidate region with the highest confidence , the corresponding confidence is for each negative class the candidate region with the highest confidence , the corresponding confidence is ​ If , consider the pair of classes as a potential misclassification case; If the training process is less than three-tenths, the class pair is recorded in the misclassification memory bank is updated where represents the number of times the class pair occurred; a positive class memory bank is constructed to record the number of times each class occurred as a positive class in previous iterations; If the training process reaches three-tenths, calculate the frequency of each class pair to potentially misclassify: ; rank the elements in descending order in , obtaining the sorted class pair frequency vector , where , and the corresponding ranking matrix , where represents the frequency rank of the class pair ; calculate the difference between adjacent elements, obtaining the frequency difference vector ; find the largest element in and record its rank ; If the training process exceeds three tenths, when a potential misclassification occurs, if the condition is met, then the candidate region that obtains the highest negative class score is labeled as a positive example; otherwise, the positive example is the region that obtains the highest positive class score ;​​ The intersection over union of other candidate regions and positive examples is calculated, and candidate regions greater than 0.5 are selected as positive examples, regions less than 0.1 are ignored, and the remaining regions are set as negative examples as supervision information ; Will and The input is fed into the multi-class cross-entropy loss function; The average of the values of the final prediction results of the confidence of each candidate region is calculated after the iteration is completed. the average value of the final prediction results of the confidence of each candidate region.

2. The weakly supervised object detection method based on misclassification correction according to claim 1, characterized in that, The convolutional neural network is based on a VGGNet neural network, and a spatial pyramid pooling layer is added at the output end of the VGGNet to generate convolutional features corresponding to multi-scale combined target candidate regions; And two fully connected layers are added at the output end of the spatial pyramid pooling layer of the VGGNet to generate feature vectors required by the classifier.

3. The weakly supervised object detection method based on misclassification correction of claim 1, wherein, The classifier is used to assign a class label to the extracted region feature, and the training process includes the following steps: The region feature vector is input into the two fully connected layers of the classifier to obtain a first matrix and a second matrix; The first matrix and the second matrix are input into a class latitude Softmax layer and a region dimension Softmax layer respectively to obtain a third matrix representing the confidence score of each candidate region belonging to each class and a fourth matrix representing the normalized contribution of each candidate region to the image containing each class; Element-wise multiplication is performed on the third matrix and the fourth matrix to generate a candidate region score matrix; The candidate region score matrix is added along the candidate region dimension to obtain the confidence of the image containing each class; The confidence of the image containing each class is input into a multi-class cross-entropy loss function together with the true class.

4. A weakly supervised object detection system based on misclassification correction, characterized in that, The system comprises: A region feature acquisition unit for extracting a region feature of a target candidate position from an input image by using a convolutional neural network; A class label acquisition unit for assigning a class label to the extracted region feature by using a classifier; A label assignment module design unit for designing a misclassification correction driven label assignment module according to the confidence difference between classes to identify and correct misclassification and reassign labels for target detection model training; During the training of the misclassification correction driven label assignment module: The following steps are iteratively performed until the maximum number of iterations is reached: inputting the region feature vector into a branch containing a fully connected layer and a softmax layer of class dimension , to obtain a candidate region score matrix ; from the candidate regions of the previous branch for each positive class the candidate region with the highest confidence for which the corresponding confidence is for each negative class the candidate region with the highest confidence for which the corresponding confidence is ; If , consider the pair of classes as a potential misclassification case; If the training process is less than three-tenths, the class pair is recorded in the misclassification memory bank is updated where represents the number of times the class pair occurred; a positive class memory bank is constructed to record the number of times each class occurred as a positive class in previous iterations; If the training process reaches three-tenths, calculate the frequency of each class pair occurring potential misclassification: ; arrange the elements in descending order in , get the sorted class pair frequency vector , where , and the corresponding ranking matrix , where represents the frequency ranking of class pair ; calculate the difference between adjacent elements to get the frequency difference vector ; find the largest element in and record its rank ; If the training process exceeds three-tenths, when the potential misclassification is... When it occurs, if the conditions are met Then the candidate region with the highest negative class score will be obtained. Mark as a positive example; otherwise, the positive example is the region that receives the highest positive class score. ; The intersection over union of other candidate regions and positive examples is calculated, and candidate regions greater than 0.5 are selected as positive examples, regions less than 0.1 are ignored, and the remaining regions are set as negative examples as supervision information ; Will and The input is fed into the multi-class cross-entropy loss function; The average of the values of the completed iterations is calculated as the final prediction result for the confidence of each candidate region. The average of the values of the completed iterations is calculated as the final prediction result for the confidence of each candidate region.

5. The weakly supervised object detection system based on misclassification correction of claim 4, wherein, The convolutional neural network in the region feature acquisition unit is based on a VGGNet neural network, and a spatial pyramid pooling layer is added at the output end of the VGGNet to generate convolutional features corresponding to multi-scale combined target candidate regions; And two fully connected layers are added at the output end of the spatial pyramid pooling layer of the VGGNet to generate feature vectors required by the classifier.

6. The weakly supervised object detection system based on misclassification correction of claim 4, wherein, The classifier is used to assign a class label to the extracted region feature, and the training process includes the following steps: The region feature vector is input into the two fully connected layers of the classifier to obtain a first matrix and a second matrix; The first matrix and the second matrix are input into a class latitude Softmax layer and a region dimension Softmax layer respectively to obtain a third matrix representing the confidence score of each candidate region belonging to each class and a fourth matrix representing the normalized contribution of each candidate region to the image containing each class; Element-wise multiplication is performed on the third matrix and the fourth matrix to generate a candidate region score matrix; The candidate region score matrix is added along the candidate region dimension to obtain the confidence of the image containing each class; The confidence of the image containing each class is input into a multi-class cross-entropy loss function together with the true class. inputting the first matrix and the second matrix into a category latitude Softmax layer and a region dimension Softmax layer respectively, to obtain a third matrix representing confidence scores of each candidate region belonging to each category and a fourth matrix representing normalized contributions of each candidate region to each category in the image; performing an element-wise multiplication operation on the third matrix and the fourth matrix to generate a candidate region score matrix; adding the candidate region score matrix along the candidate region dimension to obtain confidence of the image containing each category; inputting the confidence of the image containing each category and the true category into a multi-class cross-entropy loss function.

7. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the steps of the weakly supervised object detection method based on misclassification correction according to any one of claims 1 to 3 when executing the program.

8. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The instructions implement the steps of the weakly supervised object detection method based on misclassification correction according to any one of claims 1 to 3 when executed by the processor.

Images