Multi-label model training method for partially annotated pictures

Abstract

The invention discloses a multi-label model training method for partially labeling pictures, and the method specifically comprises the steps: carrying out the partial labeling of an image in a sample data set, and obtaining a training set; a ViT-L / 16 neural network model is adopted to extract a two-dimensional feature map of the images in the training set, and the occurrence probability of each label at each position is calculated; and the optimized Asymetric Loss asymmetric loss function is adopted to adjust the loss weight of the image in the training set. By adopting the training method, the difficulty in multi-label model training of part of labeled pictures can be effectively solved, so that the model can obtain high precision and high performance at the same time.

Description

Technical Field

[0001] The present application relates to the field of image processing, and mainly to a multi-label model training method for partially annotated images. Background Art

[0002] There are three difficulties in training multi-label classification models:

[0003] 1. Some labels in the training data set are missing. Due to the time-consuming and labor-intensive labeling and the increase in the number of labels, some labels in the data set may be missing;

[0004] 2. Positive and negative samples are unevenly distributed. In practice, positive samples are always rare and difficult to collect, while collecting negative samples is relatively easy, which inevitably leads to an imbalance in positive and negative samples;

[0005] 3. Different labels appear in different areas of the picture. For example, the subject often appears in the center, while the TV station logo often appears in the upper right corner of the picture. It is necessary to adaptively extract labels from different areas.

[0006] Traditional multi-label training methods can often only solve two of the problems. Here, a method is proposed to solve the above three problems at the same time. Summary of the invention

[0007] In response to the above problems in the prior art, the present application proposes a multi-label model training method for partially annotated images.

[0008] According to one aspect of the present invention, a multi-label model training method for partially annotated images is proposed, comprising:

[0009] Partially annotate the images in the sample data set to obtain a training set;

[0010] The ViT-L / 16 neural network model is used to extract the two-dimensional feature maps of the images in the training set and calculate the probability of each label appearing at each position;

[0011] The optimized Asymmetric Loss asymmetric loss function is used to adjust the loss weights of the images in the training set.

[0012] Furthermore, the partially annotating the images in the sample data set to obtain the training set specifically includes: partially annotating the images, and for the missing labels, counting the probability of the positive samples of the label appearing in the data set, and the probability of the model outputting the label, if both are less than a preset value at the same time, marking them as pseudo labels of the negative samples of the label.

[0013] Annotating only part of an image, rather than the entire image, can greatly reduce the cost and time of manual annotation. Furthermore, through statistical analysis of existing labels, the probability of missing labels can be inferred, thereby reducing the need to rely entirely on manual annotation.

[0014] By calculating the model prediction probability and the probability of the label appearing in the dataset, pseudo labels can be quickly generated for unlabeled areas. This method can automatically fill in the missing labels, thereby improving the overall efficiency of labeling.

[0015] Furthermore, the method of extracting the two-dimensional feature map of the image in the training set using the neural network model and calculating the probability of each label appearing at each position specifically includes:

[0016] The ViT-L / 16 neural network is used to extract the two-dimensional feature map of the image. The features at each position on the two-dimensional feature map are used as the key and value of the Transformer decoder structure.

[0017] Based on the AutoEncoder model, the word features of each label are extracted as the query of the Transformer decoder structure to query whether each position in the image contains the corresponding label;

[0018] Based on the Key, Value, and Query in the Transformer decoder structure, the probability of each label appearing at each position is calculated.

[0019] The AutoEncoder model uses 1 layer of autoencoder and 1 layer of autodecoder.

[0020] Among them, in the Transformer model, the decoder structure is mainly used for sequence-to-sequence tasks, such as machine translation. The decoder module is usually composed of multiple layers of identical sub-layers, each of which contains a multi-attention mechanism and a feedforward neural network.

[0021] To help the model focus on different parts of the input sequence during decoding, three parameters are used: Query, Key, and Value.

[0022] Key and Value usually come from the output of the encoder, while Query comes from the output of the decoder itself.

[0023] The Key vector is generated by the output of the encoder and represents the characteristics of each position of the input sequence. In the decoding stage, the decoder determines which input positions are most relevant by calculating the similarity between the Query and the Key, thereby deciding the input part to focus on.

[0024] The Value vector is also generated by the output of the encoder, which contains specific information about the input sequence. When the decoder determines which input positions should be focused on, it extracts information from the corresponding Value vector to generate the final output.

[0025] The query vector comes from the output of the decoder. The purpose of this stage is to enable the decoder to focus on the relevant information in the input sequence and extract useful information from it to generate output.

[0026] By calculating the similarity between Query and Key (usually using dot product), the attention weights are obtained, and then the Value vectors are weighted summed according to these weights to obtain the attention output. This design enables Transformer to effectively capture long-distance dependencies between input and output sequences while improving the ability of parallel computing.

[0027] Furthermore, the use of the optimized Asymmetric Loss asymmetric loss function to adjust the loss weights of the images in the training set specifically includes: giving high weights to positive samples, giving medium weights to negative samples, giving the minimum weight of the loss function to pseudo-labeled samples, and not participating in the calculation of the loss function for simple negative samples.

[0028] The specific formula of the Asymmetric Loss asymmetric loss function is as follows:

[0029]

[0030] Where N represents the total number of samples in the training set, C represents the total number of categories, i represents the i-th sample, and j represents the j-th category;

[0031] The Loss calculation formula for each sample in each category is as follows:

[0032] L + =(1-p γ+ )log(p)

[0033]

[0034] Among them, L + , L- and L un Represents the Loss of positive samples, negative samples and pseudo labels respectively;

[0035] γ + , γ - and γ un Represent the weight coefficients of positive samples, negative samples and pseudo labels respectively;

[0036] p represents the calculated probability of a positive sample;

[0037] p m represents the probability after the shift, p m =max(pm,0), where m represents the offset.

[0038] The optimized Asymmetric Loss asymmetric loss function is used. For positive samples, since the label data is usually the least, we hope to better fit the positive samples, so we give them a greater weight when calculating the loss function. For negative samples, we give them a medium to low weight. For pseudo-labeled samples, we give them the minimum weight in the loss function. At the same time, simple negative samples are not included in the calculation of the loss function. After optimizing the loss function, the impact of a large number of negative samples can be greatly reduced, and the recall rate of the model can be improved.

[0039] According to a second aspect of the present invention, a computer-readable storage medium is provided, on which one or more computer programs are stored. When the one or more computer programs are executed by a computer processor, the above method is implemented.

[0040] The above one or more technical solutions in the embodiments of the present application have at least one of the following technical effects:

[0041] By reducing the cost of manual annotation and improving data utilization, the efficiency of image annotation and the performance of the model can be effectively improved. By using pseudo-labels and partial annotation strategies, limited annotation resources can be better utilized and the robustness and generalization ability of the model can be improved when processing complex image data. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated into and constitute a part of this specification. The accompanying drawings illustrate the embodiments and are used together with the description to explain the principles of the present invention. It will be easy to recognize other embodiments and many expected advantages of the embodiments because they become better understood by reference to the following detailed description. The elements of the drawings are not necessarily to scale with each other. The same reference numerals refer to corresponding similar parts.

[0043] Figure 1 A schematic diagram of a multi-label model training method for partially annotating images according to an embodiment of the present invention is shown;

[0044] Figure 2 A schematic diagram of a model network structure according to a specific embodiment of the present invention is shown;

[0045] Figure 3 It is a structural diagram of a computer system suitable for implementing an electronic device of an embodiment of the present application. DETAILED DESCRIPTION

[0046] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It should also be noted that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0047] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0048] Figure 1 FIG. 4 shows a flow chart of a multi-label model training method for partially annotating images according to an embodiment of the present invention. Figure 1 As shown:

[0049] Partially annotate the images in the sample data set to obtain a training set;

[0050] The method of partially labeling the images in the sample data set to obtain the training set specifically includes: partially labeling the images, and for the missing labels, counting the probability of the positive samples of the label appearing in the data set, and the probability of the model outputting the label. If both are less than a preset value at the same time, the labels are marked as pseudo labels of the negative samples of the label.

[0051] In this embodiment, an image is labeled with cats and dogs, which are positive samples, and there are no chickens and ducks, which are negative samples. As demand increases, we need to add the labels of tiger and cow, but we are not sure whether the image contains these two categories. The corresponding labels are missing, resulting in partial labeling of the image. Obtaining the training set specifically includes: partially labeling the image, for the missing labels, taking the tiger label as an example, counting the frequency of positive samples of tiger in the data set, if it is less than the preset frequency value; and calculating the probability value of the model output containing a tiger in the picture, if the value is also less than the preset probability value. If both are less than the preset values ​​at the same time, it is considered that there is no tiger in the picture, and the picture is marked as a negative sample pseudo label of tiger. This method reasonably supplements the missing labels.

[0052] In the process of model construction, the common models fail to consider the problem that different labels may appear in different positions. Therefore, by modifying the model, labels in different regions can be adaptively extracted. Specifically:

[0053] The ViT-L-16 network model is used to extract the two-dimensional feature maps of the images in the training set, and the image features are used as the key and value. At the same time, the AutoEncoder model is used to extract the features of the label text, and the text features are used as the query. The key and value of the image features and the text features are used as the query in the Transformer decoder to calculate the probability of each label appearing in each position.

[0054] The method of extracting the two-dimensional feature map of the image in the training set by using the neural network model and calculating the probability of each label appearing at each position specifically includes:

[0055] The ViT-L / 16 neural network is used to extract the two-dimensional feature map of the image. The features at each position on the two-dimensional feature map are used as the key and value of the Transformer decoder structure.

[0056] Based on the AutoEncoder model, the word features of each label are extracted as the query of the Transformer decoder structure to query whether each position in the image contains the corresponding label;

[0057] Based on the Key, Value, and Query in the Transformer decoder structure, the probability of each label appearing at each position is calculated.

[0058] The AutoEncoder model uses 1 layer of autoencoder and 1 layer of autodecoder, that is, a simple linear autoencoder, which has a simple architecture, relatively low computational cost, and a fast model convergence speed.

[0059] The network structure flow chart is as follows Figure 2 As shown in the figure, ViT-L / 16 is used to extract the two-dimensional feature map of the image, and the features of each position on the two-dimensional feature map are used as the key and value of the Transformer decoder structure; the word features of each label are extracted based on the AutoEncoder model. The AutoEncoder model uses 1 layer of automatic encoder and 1 layer of automatic decoder to extract the word features of each label as the query of the Transformer decoder structure, and query whether each position in the image contains the label, so that the label can be adaptively extracted from the image;

[0060] The optimized Asymmetric Loss asymmetric loss function is used to adjust the loss weights of the images in the training set.

[0061] The specific method of adjusting the loss weights of images in the training set by using the optimized Asymmetric Loss function is as follows: for positive samples, high weights are given; for negative samples, medium weights are given; for samples with pseudo-labels, the minimum weight is assigned to their loss functions. At the same time, for simple negative samples, they do not participate in the calculation of the loss function.

[0062] The specific formula of the Asymmetric Loss function is as follows:

[0063]

[0064] Where N represents the total number of all samples in the training set, C represents the total number of classifications, i represents the i-th sample, and j represents the j-th classification;

[0065] The Loss calculation formula for each sample in each category is as follows:

[0066] L + =(1 - p γ+ )log(p)

[0067]

[0068] Where L + 、L - and L un represent the Loss of positive samples, negative samples, and pseudo-labels respectively;

[0069] γ + 、γ - and γ un represent the weight coefficients of positive samples, negative samples, and pseudo-labels respectively;

[0070] p represents the probability calculated as a positive sample;

[0071] p m represents the offset probability, p m = max(p - m, 0), where m represents the offset.

[0072] In this embodiment, γ + = 0, γ - = 3, and γ un = 4. Since the probability value 0 < p < 1, therefore, that is, positive samples are given the maximum weight, followed by negative samples, and the weight of pseudo-labels is the lowest, which greatly alleviates the uneven distribution of positive and negative samples. At the same time, by the threshold m, the contribution of easy negative samples and pseudo-labels to the loss is reduced. In practice, m = 0.2 is taken. For example, for a negative sample image, the probability p calculated that the image is a positive sample is p = 0.01, p m=max(0.01-0.2,0)=0, which does not participate in the calculation of the loss function. The threshold m is used to reduce the contribution of negative samples and pseudo labels to the loss.

[0073] In summary, the above training method can effectively solve the difficulties in multi-label model training of partially annotated images, so that the model can achieve high performance while achieving high accuracy.

[0074] Reference below Figure 3 , which shows a schematic diagram of the structure of a computer system 300 suitable for implementing an electronic device of an embodiment of the present application. Figure 3 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0075] like Figure 3 As shown, the computer system 300 includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage part 308 into a random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the system 300 are also stored. The CPU 301, the ROM 302, and the RAM 303 are connected to each other through a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0076] The following components are connected to the I / O interface 305: an input section 306 including a keyboard, a mouse, etc.; an output section 307 including a liquid crystal display (LCD), etc. and a speaker, etc.; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LAN card, a modem, etc. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to the I / O interface 305 as needed. A removable medium 311, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 310 as needed, so that a computer program read therefrom is installed into the storage section 308 as needed.

[0077] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 309, and / or installed from the removable medium 311. When the computer program is executed by the central processing unit (CPU) 301, the above functions defined in the method of the present application are executed. It should be noted that the computer-readable storage medium of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, - but not limited to - an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable storage medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wireless, wireline, optical cable, RF, etc., or any suitable combination of the foregoing.

[0078] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0079] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0080] The modules involved in the embodiments of the present application may be implemented by software or by hardware.

[0081] As another aspect, the present application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiment; or it may exist independently without being assembled into the electronic device. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed by the electronic device, the electronic device: partially annotates the images in the sample data set to obtain a training set; uses the ViT-L / 16 neural network model to extract the two-dimensional feature map of the image in the training set, and calculates the probability of each label appearing at each position; uses the optimized Asymmetric Loss asymmetric loss function to adjust the loss weight of the image in the training set.

[0082] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above invention concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. A multi-label model training method for partially annotated images, characterized in that: The following steps are involved: Partially annotate the images in the sample data set to obtain a training set; The ViT-L / 16 neural network model is used to extract the two-dimensional feature maps of the images in the training set and calculate the probability of each label appearing at each position; The optimized Asymmetric Loss asymmetric loss function is used to adjust the loss weights of the images in the training set.

2. The method according to claim 1, characterized in that: The method of partially labeling the images in the sample data set to obtain the training set specifically includes: partially labeling the images, and for the missing labels, counting the probability of the positive samples of the label appearing in the data set, and the probability of the model outputting the label. If both are less than a preset value at the same time, the labels are marked as pseudo labels of the negative samples of the label.

3. The method according to claim 1, characterized in that: The method of extracting the two-dimensional feature map of the image in the training set by using the neural network model and calculating the probability of each label appearing at each position specifically includes: The ViT-L / 16 neural network is used to extract the two-dimensional feature map of the image. The features at each position on the two-dimensional feature map are used as the key and value of the Transformer decoder structure. Based on the AutoEncoder model, the word features of each label are extracted as the query of the Transformer decoder structure to query whether each position in the image contains the corresponding label; Based on the Key, Value, and Query in the Transformer decoder structure, the probability of each label appearing at each position is calculated.

4. The method according to claim 1, characterized in that: The AutoEncoder model uses 1 layer of autoencoder and 1 layer of autodecoder.

5. The method according to claim 1, characterized in that: The optimized Asymmetric Loss asymmetric loss function is used to adjust the loss weights of the images in the training set. Specifically, a high weight is given to the positive samples, a medium weight is given to the negative samples, and the minimum weight of the loss function is given to the pseudo-labeled samples. At the same time, simple negative samples are not involved in the calculation of the loss function.

6. The method according to claim 5, characterized in that: The specific formula of the Asymmetric Loss asymmetric loss function is as follows: Where N represents the total number of samples in the training set, C represents the total number of categories, i represents the i-th sample, and j represents the j-th category; The Loss calculation formula for each sample in each category is as follows: L + =(1-p γ+ )log(p) Among them, L + , L - and L un Represents the Loss of positive samples, negative samples and pseudo labels respectively; γ + , γ - and γ un Represent the weight coefficients of positive samples, negative samples and pseudo labels respectively; p represents the calculated probability of a positive sample; p m represents the probability after the shift, p m =max(pm,0), where m represents the offset.

7. A computer program product, characterized in that A computer program is stored thereon, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

8. A computing system, characterized in that: The method comprises a processor and a memory, wherein the processor is configured to execute the method according to any one of claims 1 to 6.

Images