Method, apparatus, system, and electronic medium for recognizing an incompletely labeled image

Abstract

Embodiments of the present disclosure disclose a method, device, system and electronic medium for recognizing annotated incomplete images. A specific embodiment of the method includes: annotating an annotated incomplete image with a single label, inputting the annotated incomplete image into a feature extraction backbone network to extract a plurality of features at different levels; constructing a semantic mask module to generate a semantic mask according to a class activation map of the image and extract features of a single object from the features at different levels; constructing a mask optimization module to filter a certain amount of significant semantic masks for each class of object from all images; and constructing a bootstrap learning framework based on an expectation maximization algorithm to gradually recover and supplement constraint information in the network training stage by comparing semantic masks of objects of different labels, so that the network can more accurately recognize annotated incomplete images. The embodiment enhances the ability to distinguish, locate and recognize different objects in complex images, and improves accuracy through bootstrap.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to the field of computer vision, in particular to a recognition method, device, system and electronic medium for labeling incomplete images. BACKGROUND

[0002] Given an image containing multiple objects, the purpose of image recognition is to find the labels (such as categories) of the appearing objects from the image and output them. This is a basic task in the field of computer vision and is relied on by many downstream tasks.

[0003] Compared with images containing only a single prominent object, multi-label images are more complex, often involving more label categories and being more difficult to recognize. At the same time, due to the complexity of the images and the large number of label categories, the cost of manually labeling the multi-label image dataset for training is even more unbearable. In view of this difficulty, existing deep learning methods attempt to use non-complete labeled datasets to train image multi-label recognition models. Generally, existing methods often assume that only part of the labels of all images in the dataset are completely labeled, or that only part of the labels of all images in the dataset are labeled. Moreover, these methods rely on the association relationship of labels or additional prior information, use graph networks, Transformers and other structures and algorithms to explore the correlation between objects, thereby enhancing the accuracy under such weak supervision and semi-supervision. However, this assumption requires that the images in the dataset must be labeled with at least two or more labels, ignoring the more practically meaningful single-label dataset in reality.

[0004] According to the disclosed method, features are extracted using a deep network, a bootstrap learning framework based on semantic mask comparison is proposed on a single-label incomplete image dataset, a recognition method for weakly supervised learning is provided, and a device, system and electronic medium based on the method are provided. Compared with the above method, it has three beneficial features: 1) reduces the cost of image labeling, can make full use of incomplete labeled data, save labor costs, and improve the practicality of the method; 2) semantic mask comparison further enhances the ability of the method to distinguish, locate and recognize different objects in complex images; 3) without relying on additional prior information, the accuracy is improved through bootstrap. SUMMARY

[0005] The summary of the present disclosure is used to introduce the concepts in a brief form, which will be described in detail in the specific embodiments section. The summary of the present disclosure is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0006] Some embodiments of the present disclosure propose an incomplete image annotation recognition method, device, system and electronic medium to solve one or more of the technical problems mentioned in the background section.

[0007] According to the actual needs and key problems mentioned above, the purpose of the present disclosure is to propose an incomplete image annotation recognition method, device, system and electronic medium, input incomplete image data annotated with only a single label, extract multi-level features, generate semantic masks using a semantic mask module and a mask optimization module, and then optimize network parameters based on a bootstrap learning framework using semantic masks, and finally obtain a neural network that can identify the labels of all known objects in the image.

[0008] In a first aspect, some embodiments of the present disclosure provide an incomplete image annotation recognition method, which comprises: step S100, given an incomplete annotation image annotated with only a single label, inputting the incomplete annotation image into a feature extraction backbone network to extract multiple features of different levels; step S200, constructing a semantic mask module to generate semantic masks according to image class activation maps, and extracting independent features of single objects in the image from features of different levels; step S300, constructing a mask optimization module to filter a certain amount of significant semantic masks for each type of object from the semantic masks generated in S200; and step S400, constructing a bootstrap learning framework based on an expectation maximization algorithm, gradually recovering and supplementing constraint information in the network training stage by comparing semantic masks of objects of different labels, and using a contrastive learning loss function, a regularization online label estimation loss function and an exponential moving average method to optimize the network under weak supervision.

[0009] In a second aspect, some embodiments of the present disclosure provide an incomplete image annotation recognition device, which comprises: an input unit configured to give an incomplete annotation image annotated with only a single label, input the incomplete annotation image into a feature extraction backbone network to extract multiple features of different levels, and preliminarily identify the image to obtain a confidence; a first construction unit configured to construct a semantic mask module to generate semantic masks according to image class activation maps, and extract independent features of single objects in the image from features of different levels; a second construction unit configured to construct a mask optimization module to filter a certain amount of significant semantic masks for each type of object from the generated semantic masks; and a third construction unit configured to construct a bootstrap learning framework based on an expectation maximization algorithm, gradually recover and supplement constraint information in the network training stage by comparing semantic masks of objects of different labels, and use a contrastive learning loss function, a regularization online label estimation loss function and an exponential moving average method to optimize the network under weak supervision.

[0010] In a third aspect, some embodiments of the present disclosure provide a system, comprising: one or more processors; a memory device having stored thereon one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the method described in any of the above implementation manners.

[0011] In a fourth aspect, some embodiments of the present disclosure provide an electronic medium having stored thereon a computer program, wherein the program, when executed by a processor, implements the method described in any of the above implementation manners. BRIEF DESCRIPTION OF DRAWINGS

[0012] The above and other features, aspects and advantages of embodiments of the present disclosure will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings. Throughout the drawings, the same or like reference numerals are used to represent the same or similar elements. It should be understood that the drawings are schematic and elements and features are not necessarily drawn to scale.

[0013] Figure 1 is a flowchart of some embodiments of the method for labeling incomplete images according to the present disclosure;

[0014] Figure 2 is a general framework diagram of some embodiments of the method for labeling incomplete images according to the present disclosure;

[0015] Figure 3 is a structure diagram of the mask selection optimization module according to some embodiments of the method for labeling incomplete images according to the present disclosure;

[0016] Figure 4 is a flowchart of the S200 step according to some embodiments of the method for labeling incomplete images according to the present disclosure;

[0017] Figure 5 is a flowchart of the S300 step according to some embodiments of the method for labeling incomplete images according to the present disclosure;

[0018] Figure 6 is a flowchart of the S400 step according to some embodiments of the method for labeling incomplete images according to the present disclosure;

[0019] Figure 7 is a structure diagram of some embodiments of the device for labeling incomplete images according to the present disclosure. DETAILED DESCRIPTION

[0020] Embodiments of the present disclosure will be described below in greater detail with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be embodied in various forms and should not be construed as being limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be thoroughly and completely understood. It should be understood that the drawings of the present disclosure are only for illustrative purposes and should not be construed as limiting the scope of protection of the present disclosure.

[0021] It should also be noted that, for ease of description, only parts related to the present application are shown in the drawings. The embodiments in the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0022] It should be noted that the terms “first”, “second”, and the like mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not intended to limit the order or interdependence of the functions performed by these devices, modules or units.

[0023] It should be noted that the terms “one”, “multiple” mentioned in the present disclosure are illustrative and not restrictive, and those skilled in the art should understand that, unless otherwise explicitly stated in the context, it should be understood as “one or more”.

[0024] The names of the messages or information exchanged between the devices in the embodiments of the present disclosure are only for illustrative purposes, and are not intended to limit the scope of the messages or information.

[0025] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0026] Figure 1 Flow 100 of some embodiments of the method for recognizing imperfectly labeled images according to the present disclosure is shown. The method for recognizing imperfectly labeled images includes the following steps:

[0027] Step S100, given an imperfectly labeled image labeled with only a single label, input the imperfectly labeled image into a feature extraction backbone network to extract multiple different levels of features.

[0028] In practice, the multiple different levels of features can be extracted by the following steps:

[0029] Step S110, given an input image, use a strong data augmentation method and a weak data augmentation method to preprocess the image respectively to obtain a strong augmented image and a weak augmented image.

[0030] Step S120, use the feature extraction backbone network to extract the multi-level features of the images generated in S110. The network is composed of an online network and a target network with the same structure, and the parameters are denoted as θ on , θtar respectively, are used to extract multi-level features of the two enhanced images in S110. For example, for the convolutional neural network ResNet (a total of 4 stages), the method extracts the output features of the last layer of multiple network blocks in the last two stages in the two sets of networks (the stage and the number can be dynamically adjusted according to specific circumstances) as a multi-level feature group, which can be denoted as F. For this design, different architectures of ResNet and various variant networks such as ResNeXt, SE-Net, etc. can be used to extract the last layer features of the corresponding blocks in the corresponding stages.

[0031] In step S200, a semantic mask module is constructed to generate a semantic mask according to the class activation map of the image, and independent features of a single object in the image are extracted from features at different levels.

[0032] The semantic mask module has the following characteristics:

[0033] The image features F of S100 are calculated based on the gradient-based class activation map to obtain an activation map G of the confidence p of the given image for the labeled label C; the activation map G is divided into a grid map according to a variable size l, the average activation of each grid is calculated, and the average activation is binarized with a variable threshold γ cam to obtain a semantic mask M that labels the objects in the image associated with the label:

[0034]

[0035] Where ind is an indicator function, which takes 1 when the condition P of ind[P] is true, and 0 otherwise, and l and J represent the element in the lth row and Jth column of the semantic mask M (matrix).

[0036] The semantic mask M is used to mask the image features and the Transformer based on the self-attention mechanism is used to obtain the features H of the object associated with the label c:

[0037]

[0038]

[0039] Where W qr y, W key and W val are three learnable parameter matrices, t represents the tth training iteration stage, Δ represents the position encoding, T represents the matrix transpose, and F represents the image features.

[0040] The feature generation process of the object has the following characteristics:

[0041] The object features are formed by fusing multiple features at different levels and scales:

[0042]

[0043] where H o is the independent feature of different levels, n is the number of layers, is the vector addition operation, O represents the number of layers of the independent feature H.

[0044] Step S300, a mask selection module is constructed to filter a certain amount of significant semantic masks for each object from the semantic masks generated in S200. The structure diagram of the mask selection module is shown in Figure 3 .

[0045] The mask selection module has the following characteristics:

[0046] The object feature H obtained from the semantic mask M in S200 is used to calculate the confidence p' of the feature belonging to the label C, forming a quadruple (H, M, c, p'). The tree data structure is used to maintain the top k quadruples with the maximum p' value in each label. The top k quadruples of each label are used as the most significant semantic mask of the label.

[0047] Step S400, a bootstrap learning framework based on the expectation maximization algorithm is constructed to gradually recover and supplement the constraint information in the network training stage by comparing the semantic masks of objects of different labels, and to optimize the network using the contrast learning loss function, the regularization online label estimation loss function and the exponential moving average method.

[0048] The network is composed of an online network and a target network with similar structures. Both networks are composed of a feature extraction backbone network, a semantic mask module and a classification network. The difference between the online network and the target network is that the parameters of the online network are updated by backpropagation, and the parameters of the target network are updated by the exponential moving average method.

[0049] The bootstrap learning framework based on the expectation maximization algorithm has the following characteristics:

[0050] Let the parameters of the network at the t-th iteration be vector Θ t , M is the semantic mask, x is the input image, Z is the associated label of the image, indicates the probability, and the log-likelihood function of the bootstrap learning framework can be described as:

[0051]

[0052] In the E-step of the bootstrap framework, the semantic mask M is optimized, and the semantic mask with the maximum probability of the object of the image associated label for each image is obtained according to S200.

[0053] In the M-step of the bootstrap framework, the model parameters Θ are updated, using the object features generated by the semantic mask selected by S300, to optimize the network parameters by comparing the contrastive learning loss function and the regularized online label estimation loss function.

[0054] Figure 4 S200 step flow of some embodiments of the disclosed method for recognizing annotated incomplete images is shown:

[0055] In step S210, class activation maps are calculated for the multi-level features of the feature extraction network. The image features F of dimension KxHxW from S100 are subjected to gradient-based class activation map calculation to obtain an activation map G for the given input image with a confidence p for the labeled label C of the input image;

[0056] In step S220, the class activation map is divided and binarized to generate a semantic mask. The activation map G is divided into a two-dimensional grid with a variable size l, and binarized with a variable threshold γ cam to obtain a semantic mask M associated with the label-related object in the image:

[0057]

[0058] where ind is an indicator function that takes 1 when the condition P of ind[P] is true, and 0 otherwise. The semantic mask M labels the foreground semantics associated with the label in the image with 1, and other background semantics with 0.

[0059] In step S230, the semantic mask is used to generate object image features, and a self-attention mechanism is used to generate object features. Let t be the t-th iteration in training. The semantic mask M obtained in S220 is used to mask the image features from the target network in S100 and a Transformer based on the self-attention mechanism is used to obtain the features H of the label-related object:

[0060]

[0061]

[0062] By masking the image features, irrelevant semantics in the image are removed, and only the object semantics related to the label are retained. Δ is the position encoding, W qry , W key and W val are three learnable parameter matrices, which are multiplied by the feature matrix to obtain query vectors, key vectors and value vectors. After a certain number of layers of multi-attention mechanism operation, multi-level object features H are obtained. Here, F refers to the features of a certain level in S100, and the calculated H is also the object image features of that level.

[0063] Step S240, fuse the multi-level object features to generate the final object feature. Fuse the multi-level H obtained in S230.

[0064]

[0065] where H is the object feature of different levels in S230, n is the number of levels used in S100, i is the vector concatenation operation. All levels of H are added and averaged to obtain the final object image feature. The dimension of H is K x (H x W). Global average pooling is applied to H, and the final object feature is obtained.

[0066] Figure 5 The S300 step flow chart of some embodiments of the image annotation incomplete recognition method of the present disclosure is shown.

[0067] Step S310, calculate the confidence and normalized feature using the object feature, and construct a quadruple. The object feature obtained in S200 is recognized using a deep network to calculate the confidence p' of the feature belonging to the label C. The object feature is then normalized to have a 2-norm of 1 to obtain H'. The quadruple (H', M, c, p') is constructed.

[0068] Step S320, construct a data structure for maintaining the most significant semantic mask of each label. Before each training round (Epoch), a plurality of fixed-size tree data structures equal to the number of label categories are constructed, which satisfy the properties of a heap. The quadruple (H, M, c, p') is maintained in the heap, and the "insert", "pop", and "adjust" operations that meet the definition of the heap are defined. The quadruple of S310 is added to the heap.

[0069] Step S330, add all the quadruples generated in the training iteration to the data structure in turn. All the quadruples generated in the training process are added to the heap of the corresponding label through the "insert" operation, and a fixed number of quadruples are retained according to the descending order of p.

[0070] Step S340, in each training iteration, the most significant semantic mask is taken out of the data structure and output. In each batch of training, the quadruple is taken out of all the heaps as the significant semantic mask of the corresponding label.

[0071] Step S350, after completing a full data training, empty the elements in the data structure. After completing a training iteration on all data, the elements in the heap are emptied before the next round begins.

[0072] Figure 6 ​A S400 step flow chart is shown to illustrate some embodiments of the annotation-incomplete image recognition method of the present disclosure.

[0073] Step S410, obtain the salient semantic mask, and use the contrastive learning loss function to perform contrastive learning with the mapped online network object features. In each training iteration, first perform the E-step: use the salient semantic mask features of each label obtained in S340 as negative samples, and use the object features obtained in S200 and mapped by the mapping network using the online network image features in S100 to form positive samples. The mapping network is formed by a full connection layer, an activation function, and a full connection layer in series. The contrastive learning loss function is used to perform contrastive learning on the positive and negative samples, and the online network parameters are optimized.

[0074] Step S420, input the online network image features into the network, calculate the confidence, and optimize the network using the regularized online label estimation loss function. In each training iteration, perform the M-step again: input the image features obtained by the online network in S100 into the classification network to obtain the confidence of the image associated with each label, and optimize the online network using the regularized online label estimation loss function (Elijah Cole et al., 2021).

[0075] Step S430, update the parameters of the target network using the exponential moving average method. Using the exponential moving average method, take a value a that can be flexibly adjusted according to actual needs, and update the target network parameters according to the following formula:

[0076] θ tar ←αθ tar +(1-α)θ on .

[0077] Further referring to Figure 7 , as an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of an annotation-incomplete image recognition device, which device embodiments correspond to the method embodiments shown in Figure 1 , and the annotation-incomplete image recognition device can be specifically applied to various electronic devices.

[0078] As Figure 7As shown, the incomplete-labeled image labeling recognition device 700 of some embodiments includes a receiving unit 701, an analyzing unit 702, an information selecting unit 703, and a generating unit 704. The input unit 701 is configured to input an incomplete-labeled image labeled with only a single label into a feature extraction backbone network, extract multiple features at different levels, and preliminarily recognize the image to obtain a confidence score. The first constructing unit 702 is configured to construct a semantic mask module, generate a semantic mask according to an image class activation map, and extract independent features of a single object in the image from the features at different levels. The second constructing unit 703 is configured to construct a mask selection module, and select a certain amount of significant semantic masks for each type of object from the generated semantic masks. The third constructing unit 704 is configured to construct a bootstrap learning framework based on an expectation maximization algorithm, gradually recover and supplement the constraint information in the network training stage by comparing the semantic masks of objects of different labels, and optimize the network using a contrastive learning loss function, a regularization online label estimation loss function, and an exponential moving average method.

[0079] It can be understood that the units described in the incomplete-labeled image labeling recognition device 700 correspond to the respective steps in the method described above. Figure 1 The operations, features, and advantages described above for the method also apply to the incomplete-labeled image labeling recognition device 700 and the units included therein, and will not be repeated here.

[0080] The above description is merely some preferred embodiments of the present disclosure and a description of the principles of the technology used. Those skilled in the art should understand that the scope of the application involved in the embodiments of the present disclosure is not limited to the technical solutions formed by the specific combinations of the technical features described above, and should also cover other technical solutions formed by any combination of the above technical features or equivalent features without deviating from the above inventive concept. For example, the above features and the technical features disclosed in the embodiments of the present disclosure (but not limited to) having similar functions are replaced with each other to form a technical solution.

Claims

1. A method for identifying an incomplete image, comprising: S100, for a given incomplete annotated image with only a single label, inputting the incomplete annotated image into a feature extraction backbone network, extracting multiple features at different levels, and preliminarily identifying the image to obtain a confidence; S200, constructing a semantic mask module, generating a semantic mask according to an image class activation map, and extracting independent features of a single object in the image from the features at different levels; S300, constructing a mask optimization module, and selecting a preset number of significant semantic masks for each type of object from the generated semantic masks; S400, constructing a bootstrap learning framework based on an expectation maximization algorithm, gradually recovering and supplementing constraint information in a network training stage by comparing semantic masks of objects with different labels, and using a contrastive learning loss function, a regularization online label estimation loss function, and an exponential moving average method to optimize the network for weakly supervised optimization, and generating a network capable of identifying images.

2. The method of claim 1, wherein, The network is composed of a similar online network and a target network, both of which are composed of a feature extraction backbone network, a semantic mask module, and a classification network; the difference between the online network and the target network is that the parameters of the online network are updated by back propagation, and the parameters of the target network are updated by the exponential moving average method.

3. The method of claim 2, wherein, The semantic mask module has the following features: The image feature F of S100 is calculated by gradient-based class activation map, obtaining the activation map G of the confidence p of the given image for the labeled label c; the activation map G is divided into a grid map according to the variable size l, the average activation of each grid is calculated, and the variable threshold γ cam The binaryzation processing is performed to obtain the semantic mask M which labels the objects associated with the label in the image: Wherein, ind is an indicator function, which takes 1 when the condition P of ind[P] is true, otherwise it takes 0, i and j represent the element of the i-th row and j-th column of the semantic mask M (matrix); Masking image features using semantic mask M and get the feature H of the object associated with label C using Transformer based on self-attention mechanism: where W qry , W key and W val are three learnable parameter matrices, t denotes the t-th training iteration stage, Δ denotes the position encoding, T denotes the matrix transpose, and F denotes the image feature.

4. The method of claim 1, wherein, The independent feature generation process of the object has the following features: Object The features are formed by fusing features at multiple levels and different scales: where H o is an independent feature at different levels, n is the number of levels, is a vector addition operation, O represents the number of levels of the independent feature H.

5. The method of claim 1, wherein, The mask optimization module has the following features: Using the object feature H obtained from the semantic mask M in S200, the confidence p' of the feature belonging to the label c is calculated, forming a four-tuple (H, M, c, p'); using a tree data structure to maintain the top k four-tuples with the largest p' value in each label; and using the top k four-tuples of each label as the most significant semantic mask of the label.

6. The method of claim 1, wherein, The bootstrap learning framework based on the expectation maximization algorithm has the following features: Let Θtbe the vector of all parameters of the network at the t-th iteration t The log-likelihood function of the bootstrap learning framework can be described as: In the E-step of the bootstrap framework, the semantic mask M is optimized, and the semantic mask with the maximum probability of the object associated with the label of the image is obtained according to S200; In the M-step of the bootstrap framework, the model parameters Θ are updated, the object features generated by the semantic masks selected by S300 are used, and the network parameters are optimized by the contrastive learning loss function and the regularization online label estimation loss function.

7. An identification device for incomplete annotated images, comprising: an input unit configured to, for a given incomplete annotated image with only a single label, input the incomplete annotated image into a feature extraction backbone network, extract multiple features at different levels, and preliminarily identify the image to obtain a confidence; a first construction unit configured to construct a semantic mask module, generate a semantic mask according to an image class activation map, and extract independent features of a single object in the image from the features at different levels; A second constructing unit configured to construct a mask preference module to filter a certain amount of significant semantic masks for each object class from the generated semantic masks; A third constructing unit configured to construct a bootstrap learning framework based on an expectation maximization algorithm, to gradually recover and supplement the constraint information in the network training stage by comparing the semantic masks of objects with different labels, and to optimize the network using a contrastive learning loss function, a regularized online label estimation loss function, and an exponential moving average method.

8. A system comprising: one or more processors; storage having stored thereon one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1-6.

9. An electronic medium having stored thereon a computer program, wherein, The program is executed by the processor to implement the method according to any one of claims 1-6.

Images