Image label labeling method and device, computer equipment and storage medium

Abstract

The invention relates to an image label labeling method and device, computer equipment and a storage medium. The method comprises the steps that image information is acquired, the image information comprises image data, text data and an image label, and the image label comprises first category marking information; determining a visual feature sequence of the image data based on the image coding model, and respectively determining a text feature sequence and category features based on the text coding model; determining a similarity matrix based on the visual feature sequence and the text feature sequence, and aligning the similarity matrix to a correction matrix to update the image coding model, the correction matrix being determined based on the image information; determining an updated visual feature sequence based on the updated image coding model, and determining second category marking information of the image tag based on the updated visual feature sequence and the category features; and marking the unknown category in the first category marking information based on the second category marking information. According to the invention, the accuracy and robustness of label labeling are improved.

Description

Technical Field

[0001] The present application relates to the technical field of image data processing, and in particular to an image labeling method, device, computer equipment and storage medium. Background Art

[0002] Multi-label image recognition models have a wide range of applications in the fields of intelligent transportation, medical health, and intelligent security. However, in the training data of multi-label image recognition models, only a small number of images may have known labels, while the labels of most other images are unknown. Therefore, training multi-label image recognition models is a challenging and practical task.

[0003] In traditional technology, pseudo-labeling is a common method to deal with the limited labeling problem. Considering the imbalance between positive and negative labels (the number of negative labels significantly exceeds that of positive labels), a more intuitive method is to extract a part of unknown labels and label them as negative pseudo-labels. However, the pseudo-labeling method in traditional technology may introduce excessive false negative labels, and asymmetric pseudo-labeling will also lead to asymmetric positive and negative pseudo-labels. Obviously, the pseudo-labeling method in traditional technology will introduce a large number of noise labels during the implementation process due to the lack of robust inference, resulting in low accuracy of label annotation. Summary of the invention

[0004] Based on this, it is necessary to provide an image labeling method, device, computer equipment and storage medium that can improve the accuracy of labeling in response to the above technical problems.

[0005] In a first aspect, the present application provides an image labeling method. The method comprises:

[0006] Acquire image information, the image information including image data, text data and image labels, the text data and the image labels respectively correspond to the image data, and the image labels include first category label information;

[0007] Determine a visual feature sequence of the image data based on an image coding model, and determine a text feature sequence of the text data and a category feature of the first category tag information based on a text coding model;

[0008] Determining a similarity matrix based on the visual feature sequence and the text feature sequence, and aligning the similarity matrix to a correction matrix to update the image coding model, wherein the correction matrix is ​​determined based on the image information;

[0009] Based on the updated image coding model, determine an updated visual feature sequence, and based on the updated visual feature sequence and the category feature, determine second category marking information of the image label;

[0010] Based on the second category label information, the unknown category in the first category label information is labeled.

[0011] In one of the embodiments, determining the visual feature sequence of the image data based on the image coding model comprises:

[0012] Segment the image data based on a preset segmentation rule to generate a plurality of image segments;

[0013] The multiple image segments are input into the image coding model, and the visual feature sequence is output, where the visual feature sequence includes global visual features and spatial visual features.

[0014] In one embodiment, determining the text feature sequence of the text data based on the text encoding model includes:

[0015] Segment the text data based on preset word segmentation rules to generate multiple word segmentation texts;

[0016] The multiple word segmentation texts are input into the text encoding model, and the text feature sequence is output, where the text feature sequence includes global text features and word segmentation text features.

[0017] In one embodiment, the similarity matrix includes a first similarity matrix, and the correction matrix includes a first identity matrix;

[0018] The determining of a similarity matrix based on the visual feature sequence and the text feature sequence, and aligning the similarity matrix to a correction matrix comprises:

[0019] Determining the first similarity matrix between the global visual features and the global text features;

[0020] The first similarity matrix is ​​aligned to the first identity matrix based on information divergence, and the first identity matrix is ​​determined based on the amount of the image data.

[0021] In one embodiment, the similarity matrix includes a fourth similarity matrix, and the correction matrix includes a label matrix;

[0022] The determining of a similarity matrix based on the visual feature sequence and the text feature sequence, and aligning the similarity matrix to a correction matrix further comprises:

[0023] Determining a second similarity matrix between the spatial visual feature and the category feature, and determining a third similarity matrix between the segmentation text feature and the category feature;

[0024] determining aggregated visual features based on the second similarity matrix, and determining aggregated text features based on the third similarity matrix;

[0025] Determining the fourth similarity matrix based on the aggregated visual features and the aggregated text features;

[0026] The fourth similarity matrix is ​​aligned to the label matrix based on a binary cross entropy loss function, and the label matrix is ​​determined based on the first category label information.

[0027] In one embodiment, determining the aggregated visual features based on the second similarity matrix comprises:

[0028] Determining an image category attention map based on the second similarity matrix and the normalized exponential function;

[0029] Determining an image feature mask based on the image category attention map and a first threshold;

[0030] The aggregated visual feature is determined based on the image feature mask.

[0031] In one embodiment, determining the aggregated text features based on the third similarity matrix includes:

[0032] Determine a text category attention map based on the third similarity matrix and the normalized exponential function;

[0033] Determining a text feature mask based on the text category attention map and a second threshold;

[0034] The aggregated text features are determined based on the text feature mask.

[0035] In one embodiment, the label matrix is ​​determined based on the following method:

[0036] Stacking the first category label information a preset number of times to determine a stacking matrix, where the preset number of times includes the number of the image labels;

[0037] The label matrix is ​​determined based on the product of the stacking matrix and a second identity matrix, wherein the second identity matrix is ​​determined based on the number of the image labels.

[0038] In one embodiment, determining the second category labeling information of the image label based on the updated visual feature sequence and the category feature includes:

[0039] Determining image label similarity based on the updated visual feature sequence and the dot product of the category feature;

[0040] Determining image tag correlation probabilities based on the image tag similarities and a normalized exponential function;

[0041] Based on the image tag related probability, the second threshold and the percentage threshold, second category labeling information of the image tag is determined.

[0042] In one embodiment, determining the second category labeling information of the image label based on the image label correlation probability, the second threshold and the percentage threshold comprises:

[0043] Sort all image tags from large to small based on the probability of the image tags being associated with them;

[0044] Determine the category of the image label with a smaller correlation probability among all the image labels as negative, and the number of image segments with a smaller correlation probability of the image labels is less than the product of the number of all labels and the percentage threshold;

[0045] Determine, among all image labels, the category of the image label whose associated probability is greater than the second threshold as positive;

[0046] Among all image labels, the categories of other image labels other than positive and negative categories are determined as unknown.

[0047] In a second aspect, the present application also provides an image labeling device. The device comprises:

[0048] A data acquisition module, used to acquire image information, wherein the image information includes image data, text data and image labels, wherein the text data and the image labels correspond to the image data respectively, and the image labels include first category label information;

[0049] A feature determination module, used to determine a visual feature sequence of the image data based on an image coding model, and to determine a text feature sequence of the text data and a category feature of the first category tag information based on a text coding model;

[0050] A model updating module, configured to determine a similarity matrix based on the visual feature sequence and the text feature sequence, and align the similarity matrix to a correction matrix to update the image coding model, wherein the correction matrix is ​​determined based on the image information;

[0051] A second category label information determination module, configured to determine an updated visual feature sequence based on the updated image coding model, and determine second category label information of the image label based on the updated visual feature sequence and the category feature;

[0052] A labeling module is used to label the unknown category in the first category labeling information based on the second category labeling information.

[0053] In a third aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of any one of the image labeling methods in the first aspect are implemented.

[0054] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of any one of the image labeling methods in the first aspect are implemented.

[0055] The above-mentioned image labeling method, device, computer equipment and storage medium extract the visual feature sequence of image data, the text feature sequence of text data and the category features of the image label category marking information, and then determine the similarity matrix based on the visual feature sequence and the text feature sequence. Since the text data and the image label correspond to the image data respectively, the similarity matrix can fully reflect the association information between the visual modality and the language modality. Aligning the similarity matrix to the correction matrix determined based on the image information can further extract the image information so that the image coding model can extract the visual features more accurately. After updating the image coding model, the category of the image label is determined based on the updated visual feature sequence and category features, and the second category marking information of the image label can be accurately determined in combination with the known category marking information in the image label. Based on the second category marking information of the image label of the image label, the unknown category in the first category marking information is marked, and then the unknown label in the image label is accurately marked, thereby improving the accuracy, effectiveness and robustness of the label marking.

[0056] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0058] Figure 1 A diagram showing an application environment of an image labeling method in one embodiment;

[0059] Figure 2 A schematic diagram of a process of an image labeling method in one embodiment;

[0060] Figure 3 is a schematic diagram of a process of aligning a similarity matrix to a correction matrix in one embodiment;

[0061] Figure 4 is a structural block diagram of an image labeling device in one embodiment;

[0062] Figure 5 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0064] The terms "module", "unit", etc. used below are a combination of software and / or hardware that can implement a predetermined function. Although the devices described in the following embodiments are preferably implemented in hardware, the implementation of software, or a combination of software and hardware is also possible and conceivable.

[0065] The image labeling method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The terminal 102 obtains image information and sends it to the server 104. The image information includes image data, text data and image tags. The text data and the image tags correspond to the image data respectively, and the known tags in the image tags include the first category tag information. The server 104 determines the visual feature sequence of the image data based on the image coding model, and determines the text feature sequence of the text data and the category features of the first category tag information based on the text coding model. The server 104 determines the similarity matrix based on the visual feature sequence and the text feature sequence, and aligns the similarity matrix to the correction matrix to update the image coding model. The correction matrix is ​​determined based on the image information. The server 104 determines the updated visual feature sequence based on the updated image coding model, and determines the second category tag information of the image tag based on the updated visual feature sequence and the category features and sends it to the terminal 102. The terminal 102 marks the unknown tags in the image tag based on the second category tag information. The implementation steps of the above-mentioned image labeling method can also be performed only by the server 104 or only by the terminal 102, and this application does not make specific restrictions on this. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices, etc. The server 104 can be implemented as an independent server or a server cluster composed of multiple servers.

[0066] In one embodiment, Figure 2 As shown, a method for labeling an image is provided, which is applied to Figure 1 The application scenario in is taken as an example to illustrate, including the following steps:

[0067] S201 acquires image information, where the image information includes image data, text data, and image tags. The text data and the image tags correspond to the image data respectively, and the image tags include first category tag information.

[0068] The image labeling method provided in the embodiment of the present application can be used to label the training set images of the image recognition model, and can also be directly used in application scenarios where image labels need to be labeled. Specifically, the image recognition model may include a medical image recognition model, an intelligent driving image recognition model, etc., and accordingly, the image information may include medical image information, image information obtained by a vehicle-mounted camera, etc. Application scenarios where image labels need to be labeled may include crowdsourced data labeling, image classification labeling, etc., and accordingly, the image information may include crowdsourced image information, image information to be classified, etc.

[0069] In the embodiment of the present application, the image information includes image data, text data and image tags. The image data includes pictures, and the text data includes description text for describing the image data. The text data corresponds to the image data. Specifically, the image data and the text data can be represented as an image-text pair.

[0070] In some specific embodiments, the image-text pair of image data and text data may be (X, T), wherein the size of the image data X may be (H, W)=(336, 336), wherein H represents the height of the image data, and W represents the width of the image data. The length M of the text data T may be 256.

[0071] The image label includes an incomplete label corresponding to the image data, and the image label includes a known label and an unknown label. The known label includes an image label of a marked category, such as a positive label or a negative label; the unknown label includes an image label of an unknown category. The image label includes first category label information, and the first category label information is used to label the label category.

[0072] In some specific embodiments, if the image label L = {a, b, c, d, e}, where a, b, c, d, e represent five labels, that is, the label space size C = 5. Then the first category label information Y may be {a = 1, b = -1, c = 0, d = 0, e = 0}, which respectively indicates that the correlation between the image label a and the target category is positive, the correlation between the image label b and the target category is negative, and the correlation between the image labels c, d, e and the target category is unknown.

[0073] S203: Determine a visual feature sequence of the image data based on the image coding model, and determine a text feature sequence of the text data and a category feature of the first category tag information based on the text coding model.

[0074] In the embodiment of the present application, it is necessary to extract features of the image data, text data and first category tag information respectively. The visual feature sequence of the image data can be determined by the image coding model, the text feature sequence of the text data can be determined based on the text coding model, and the category features of the first category tag information can be determined based on the text coding model.

[0075] In the embodiment of the present application, when extracting the visual feature sequence of image data and the text feature sequence of text data, the image data and the text data can be segmented according to the preset segmentation rules, and then the features of the segmented image data and the segmented text data are respectively extracted to obtain a more detailed and accurate feature sequence. The implementation method of the preset segmentation rules and the feature extraction is described below through the embodiment of the present application.

[0076] In some embodiments, determining a sequence of visual features of the image data based on the image coding model comprises:

[0077] S301: Segment image data based on a preset segmentation rule to generate multiple image segments.

[0078] S303: Input multiple image segments into an image coding model, and output a visual feature sequence, where the visual feature sequence includes global visual features and spatial visual features.

[0079] In an embodiment of the present application, the preset segmentation rule may include a preset fragmentation rule. The image data is segmented according to the preset fragmentation rule to generate multiple image fragments. In some specific embodiments, the image data may be segmented according to the preset fragmentation rule, and the image data X of size (H, W) = (336, 336) may be segmented into N = 441 blocks without overlapping to generate 16 × 16 image fragments X'. In other embodiments, the preset fragmentation rule may also include overlapping and segmenting the image data into a preset number of multiple image fragments.

[0080] Multiple image segments are input into an image coding model, and a visual feature sequence is output, which includes global visual features and spatial visual features.

[0081] In some specific embodiments, the image coding model may include a ViT (Vision Transformer) model, where multiple image segments X' are input into the ViT model, and a visual feature sequence V = {vg, v1, ..., vN}∈R is output. (1+N)×hv Where vg is the global visual feature, V' = {v1, ..., vN} is the spatial visual feature, R represents a real number, N represents the number of image segments, and hv represents the dimension of each vector in the visual feature sequence.

[0082] In some embodiments, determining a text feature sequence of text data based on a text encoding model includes:

[0083] S401: Segment text data based on preset word segmentation rules to generate multiple word segmentation texts.

[0084] S403: Input multiple segmented texts into a text encoding model, and output a text feature sequence, where the text feature sequence includes global text features and segmented text features.

[0085] In the embodiment of the present application, the preset segmentation rule may include a preset word segmentation rule. The text data T is segmented based on the preset word segmentation rule to generate multiple word segmentation texts T'. The multiple word segmentation texts are input into the text encoding model, and a text feature sequence is output, which includes global text features and word segmentation text features.

[0086] In some specific embodiments, the text encoding model may include a Transformer structure model (self-attention model). Input multiple segmented texts T' into the text encoding model, and output a text feature sequence U = {ug, u1, ..., uM}∈R (1+M)×hu . Where ug is the global text feature, U'={u1,...,uM} is the word segmentation text feature, R represents a real number, M represents the number of word segmentation texts, and hv represents the dimension of each vector in the text feature sequence.

[0087] In some embodiments, determining the category feature of the first category tag information based on the text encoding model may include determining category prompt information based on a preset template and the first category tag information, inputting the category prompt information into the text encoding model, and outputting the category feature.

[0088] In some specific embodiments, the preset template may include a preset prompt sentence, for example, "This is a picture containing [category name]", and after determining the target category according to the first category tag information, "[category name]" is inserted into the preset prompt sentence to determine the category prompt information, such as "This is a picture containing A". For the label L = {a, b, c, d, e}, the category prompt information determined based on the preset template and the first category tag information may be {Ta, Tb, Tc, Td, Te}.

[0089] By inputting the category prompt information {Ta, Tb, Tc, Td, Te} into the Transformer structure model, the category feature W = {wa, wb, wc, wd, we} ∈ R C×hw , where R represents a real number, C represents the number of image labels, and hw represents the dimension of each vector in the category feature.

[0090] In the embodiment of the present application, after determining the visual feature sequence V, the text feature sequence U, and the category feature W, the features V, U, and W can be mapped to the same dimension using the visual projection network and the text projection network, respectively. Specifically, three multi-layer perceptrons (FV, FU, and FW) with the same output dimension h can be used to project V, U, and W into an embedding space, respectively, to facilitate subsequent data processing.

[0091] S205: Determine a similarity matrix based on the visual feature sequence and the text feature sequence, and align the similarity matrix to a correction matrix to update the image coding model, where the correction matrix is ​​determined based on the image information.

[0092] In the embodiment of the present application, the image coding model can be updated by coarse-grained semantic alignment. In other embodiments, the image coding model can also be updated by fine-grained semantic alignment. In other embodiments, the image coding model can also be updated by both coarse-grained semantic alignment and fine-grained semantic alignment.

[0093] In some embodiments, in the coarse-grained semantic alignment method, determining the similarity matrix based on the visual feature sequence and the text feature sequence may include determining the similarity matrix based on the visual feature sequence and the text feature sequence, and aligning the similarity matrix to the correction matrix. In the coarse-grained semantic alignment method, the similarity matrix may be determined based on the number of image segments.

[0094] In some embodiments, in the fine-grained semantic alignment method, determining the similarity matrix based on the visual feature sequence and the text feature sequence may include determining the similarity matrix based on the visual feature sequence, the text feature sequence and the category feature, and aligning the similarity matrix to the correction matrix. In the fine-grained semantic alignment method, the similarity matrix may be determined based on the number of image segments.

[0095] In the embodiment of the present application, after the similarity matrix is ​​aligned to the correction matrix, the similarity matrix will change accordingly, and then according to the gradient back propagation, the image coding model for extracting the visual feature sequence and the text coding model for extracting the text feature sequence will also be updated accordingly. Specifically, the slice embedding layer, position embedding layer, coding module layer, and linear classification layer in the image coding model and the text coding model are all updated accordingly.

[0096] In the embodiments of the present application, cross-modal feature alignment is intended to unify the semantic knowledge between the visual modality and the textual modality and realize cross-modal representation learning, so as to make full use of the feature information in the image and text to learn a high-performance multi-label classification model, while providing high-quality labels for subsequent image label annotation.

[0097] S207: Determine an updated visual feature sequence based on the updated image coding model, and determine second category marking information of the image label based on the updated visual feature sequence and the category feature.

[0098] In an embodiment of the present application, based on the updated image coding model, an updated visual feature sequence of the image data can be determined. Based on the updated visual feature sequence and the category feature, second category labeling information of the image label can be determined. The second category labeling information can be used to update the first category labeling information. Specifically, the second category labeling information can be used to mark the category of the unknown label in the image label.

[0099] In some embodiments, determining the second category tag information of the image tag based on the updated visual feature sequence and the category feature includes:

[0100] S501: Determine image label similarity based on the updated visual feature sequence and the dot product of the category feature.

[0101] S503: Determine the image tag correlation probability based on the image tag similarity and the normalized exponential function.

[0102] S505: Determine second category labeling information of the image label based on the image label correlation probability, the second threshold and the percentage threshold.

[0103] In the embodiment of the present application, the image label similarity Sw is determined based on the dot product of the updated visual feature sequence vg' and the category feature W = {wa, wb, wc, wd, we}. The image label similarity Sw is used to represent the similarity between the image data X and the different labels a, b, c, d, e in the image label L = {a, b, c, d, e}. Based on the image label similarity Sw and the normalized exponential function Softmax, the image label similarity Sw is converted into a label correlation probability Pw. Based on the image label correlation probability Pw, the second threshold and the percentage threshold, the second category label information of the image label can be determined

[0104] In some embodiments, based on the image tag correlation probability, the second threshold, and the percentage threshold, determining the second category tag information of the image tag includes:

[0105] S601: Sort all image tags from large to small based on the image tag related probabilities.

[0106] S603: Determine the category of the image label with a smaller correlation probability among all the image labels as negative, and the number of image segments with a smaller correlation probability of the image label is less than the product of the number of all labels and the percentage threshold.

[0107] S605: Determine the category of the image label whose image label correlation probability is greater than the second threshold among all the image labels as positive.

[0108] S607: Determine the categories of other image labels other than the positive and negative categories among all image labels as unknown.

[0109] In the embodiment of the present application, all image tags are sorted from large to small based on the image tag relevance probability Pw. The percentage threshold δ% is set, and the product of the number of image tags C and the percentage threshold δ% is C×δ%. Among all the image tags sorted from large to small with the relevance probability Pw, the category of the image tags with the number of image tags less than or equal to C×δ% and the smaller image tag relevance probability Pw is determined to be negative. For example, if the number of tags is 10, the percentage threshold is 3%, and the 10 tags are sorted from large to small based on the image tag relevance probability Pw as {6, 5, 3, 7, 9, 8, 1, 2, 4, 10}, then the categories of the last 3 image tags {2, 4, 10} with the smaller image tag relevance probability Pw are determined to be negative.

[0110] Among all image labels, the category of the image label whose image label correlation probability is greater than the second threshold θ is determined to be positive. Among all image labels, the categories of other image labels other than the positive and negative categories are determined to be unknown. Still taking the above-mentioned number of labels 10 as an example, if the image label correlation probability Pw of the image label {6,5,3,7} is greater than the second threshold θ, the category of the image label {6,5,3,7} is determined to be positive. The categories of the remaining image labels {9,8,1} are determined to be unknown.

[0111] It can be understood that the second category label information y' can be determined by determining the category of the image label. The dimensions corresponding to the positive, negative, and unknown categories are set to 1, -1, and 0 respectively, then the second category label information y'∈{-1,0,1} C , where C represents the number of image labels.

[0112] S209: Labeling unknown labels in the image labels based on the second category label information.

[0113] In an embodiment of the present application, labeling unknown labels in image labels according to second category labeling information may include converting the first category labeling information of the image labels into category labeling information to be labeled, and labeling the unknown categories in the category labeling information to be labeled according to the second category labeling information.

[0114] In some specific embodiments, the first category labeling information Y may be {a=1, b=-1, c=0, d=0, e=0}, then the first category labeling information Y of the image label can be converted into the category labeling information to be labeled y={1,-1,0,0,0}, if the second category labeling information y'={0,-1,1,1,0}, then the unknown labels in the category labeling information to be labeled y will be labeled, and the updated category labeling information y"={1,-1,1,1,0} will be obtained, so as to realize the labeling of the unknown labels in the image label.

[0115] The image labeling method provided in the embodiment of the present application can be used as an execution method of the image labeling model. After the image information is input into the image labeling model, the labeled image label is output, and the image labeling model can be trained using the target loss function of multi-label learning. Finally, the trained image labeling model can be used to perform multi-label inference on new samples. Predicting high-confidence pseudo-labels aims to use more complete supervisory information to learn a more effective image labeling model, thereby providing more accurate multi-label predictions.

[0116] The present application extracts the visual feature sequence of image data, the text feature sequence of text data, and the category features of the category marking information of the image label, and then determines the similarity matrix based on the visual feature sequence and the text feature sequence. Since the text data and the image label correspond to the image data respectively, the similarity matrix can fully reflect the correlation information between the visual modality and the language modality. Aligning the similarity matrix to the correction matrix determined based on the image information can further extract the image information so that the image coding model can extract visual features more accurately. After updating the image coding model, the category of the image label is determined based on the updated visual feature sequence and category features, and the second category marking information of the image label can be accurately determined in combination with the known category marking information in the image label. Based on the second category marking information of the image label of the image label, the unknown category in the first category marking information is marked, thereby achieving accurate marking of the unknown label in the image label, thereby improving the effectiveness and robustness of the label marking.

[0117] The following describes the implementation methods of aligning the similarity matrix to the correction matrix in the coarse-grained semantic alignment and the fine-grained semantic alignment respectively through the embodiments of the present application.

[0118] In the coarse-grained semantic alignment mode, in some embodiments, the similarity matrix includes a first similarity matrix, and the correction matrix includes a first unit matrix; determining the similarity matrix based on the visual feature sequence and the text feature sequence, and aligning the similarity matrix to the correction matrix includes:

[0119] S701: Determine a first similarity matrix between global visual features and global text features;

[0120] S703: Aligning the first similarity matrix to a first identity matrix based on information divergence, where the first identity matrix is ​​determined based on the amount of image data.

[0121] In the embodiment of the present application, the cosine similarity sg between the global visual feature vg and the global text feature ug is determined, and the first similarity matrix Sg is determined by combining the sg of all image-text pairs.

[0122] Specifically, for an image-text pair (X, T), after extracting the features of the paired image data X and text data T, the global visual features vg∈R and the global text features ug∈R are obtained, and the cosine similarity sg∈R between the two global features is calculated, where R represents a real number. Similarly, for all image-text pairs {(X1, T1), ..., (XK, TK)} in the dataset to be labeled, after determining their global features {(vg1, ug1), ..., (vgK, ugK)}, the global feature cosine similarities between different image-text pairs are determined, and finally a K×K first similarity matrix Sg is obtained.

[0123] The first similarity matrix is ​​aligned to the first identity matrix based on information divergence, wherein the divergence information includes KL divergence (Kullback-Leibler), and the first identity matrix is ​​determined based on the amount of image data. In some specific embodiments, the first identity matrix may include an identity matrix determined by the image data and the text data.

[0124] Specifically, the KL divergence is used to align the similarity matrix Sg to the identity matrix I∈{0,1} of the image-text pair K×K By contrastive learning, paired image features are brought closer to text features, while unpaired image features are pushed further away from text features, so as to learn an effective embedding space and improve the accuracy of image labeling.

[0125] In the fine-grained semantic alignment mode, in some embodiments, such as Figure 3 As shown, the similarity matrix includes a fourth similarity matrix, and the correction matrix includes a label matrix;

[0126] Determining a similarity matrix based on the visual feature sequence and the text feature sequence, and aligning the similarity matrix to the correction matrix also includes:

[0127] S801: Determine a second similarity matrix between spatial visual features and category features, and determine a third similarity matrix between segmented text features and category features.

[0128] S803: Determine aggregated visual features based on the second similarity matrix, and determine aggregated text features based on the third similarity matrix.

[0129] S805: Determine a fourth similarity matrix based on the aggregated visual features and the aggregated text features.

[0130] S807: Align the fourth similarity matrix to the label matrix based on the binary cross entropy loss function, where the label matrix is ​​determined based on the first category label information.

[0131] In the embodiment of the present application, the second similarity matrix SV∈R between the spatial visual feature V′ and the category feature W is determined N×C , and determine the third similarity matrix ST∈R between the segmentation text feature U' and the category feature W M×C , where R represents a real number, N represents the number of image segments, M represents the number of segmented texts, and C represents the number of image labels.

[0132] Based on the second similarity matrix, visual features can be aggregated, and the aggregated visual features can represent aggregated visual features of a specific category. Based on the third similarity matrix, aggregated text features are determined, and the aggregated text features can represent aggregated text features of a specific category.

[0133] In some embodiments, determining the aggregated visual features based on the second similarity matrix includes:

[0134] S901: Determine an image category attention map based on a second similarity matrix and a normalized exponential function.

[0135] S903: Determine an image feature mask based on the image category attention map and the first threshold.

[0136] S905: Determine aggregated visual features based on the image feature mask.

[0137] In the embodiment of the present application, the normalized exponential function Softmax function is performed on the second similarity matrix SV from the column dimension to determine the image category attention map MV∈[0,1] N×C , where N represents the number of image segments and C represents the number of image labels.

[0138] According to the first threshold value η V Select the image feature mask MV' with high correlation with the category feature = (MV>η V ∈R{0,1} N×C ). That is, the image category attention map greater than the first threshold is used as the image feature mask.

[0139] The spatial visual features of the image feature mask MV'=1 are calculated and averaged by column to determine the aggregated visual features Vagg∈R for a specific category C×h , where h represents the dimension of each element in the aggregated visual feature.

[0140] In some embodiments, determining the aggregated text features based on the third similarity matrix includes:

[0141] S1001: Determine a text category attention mapping based on a third similarity matrix and a normalized exponential function.

[0142] S1003: Determine a text feature mask based on the text category attention map and the second threshold.

[0143] S1005: Determine aggregated text features based on the text feature mask.

[0144] In the embodiment of the present application, the normalized exponential function Softmax function is performed on the third similarity matrix ST from the column dimension to determine the text category attention map MT∈[0,1] M×C , where M represents the number of segmented texts and C represents the number of image labels.

[0145] According to the set second threshold η T Select the text feature mask MT' with high correlation with the category feature = (MT>η T ∈R{0,1} M×C ). That is, the text category attention map greater than the second threshold is used as the text feature mask.

[0146] The word segmentation text features with text feature mask MT'=1 are calculated and averaged by column to determine the aggregated text features Uagg∈R for a specific category C×h , where h represents the dimension of each element in the aggregated text feature.

[0147] After determining the aggregated visual features and the aggregated text features, a fourth similarity matrix Sagg∈R is determined based on the aggregated visual features and the aggregated text features. C×C The method of determining the fourth similarity matrix may refer to the method of determining the first similarity matrix between the global visual features and the global text features in step S701 in the above embodiment, which will not be described in detail here.

[0148] The fourth similarity matrix is ​​aligned to the label matrix ML based on the binary cross entropy loss function, thereby constraining the alignment learning of fine-grained semantic features, wherein the label matrix is ​​determined based on the first category label information.

[0149] The following is an example of the method for determining the label matrix. In some embodiments, the label matrix is ​​determined based on the following method:

[0150] S1101: Stacking the first category label information a preset number of times to determine a stacking matrix, where the preset number includes the number of image labels.

[0151] S1103: Determine a label matrix based on the product of the stacking matrix and a second identity matrix, where the second identity matrix is ​​determined based on the number of image labels.

[0152] In the embodiment of the present application, the first category label information is repeatedly stacked a preset number of times to determine a stacking matrix, where the preset number includes the number of image labels C. It can be understood that the stacking matrix is ​​composed of the first category label information as a vector of each row, and the row vectors are repeatedly stacked C times to obtain a stacking matrix.

[0153] The label matrix ML is determined based on the product of the stacking matrix and the second identity matrix, where the second identity matrix is ​​determined based on the number of image labels. In some specific embodiments, the second identity matrix is ​​I∈{0,1} C×C , where C is the number of image labels.

[0154] In the embodiment of the present application, the image labeling can be a continuous and repeated process. In the process of labeling the image for the first time, the label matrix can be determined by stacking the first category label information. In the second and subsequent image labeling processes, the label matrix can also be determined by stacking the second category information a preset number of times, where the preset number includes the number of image labels.

[0155] In an embodiment of the present application, through a fine-grained semantic alignment method, the aggregated visual features and aggregated text features judged as positive categories are brought closer, and the aggregated visual features and aggregated text features judged as negative categories are pushed away, which can further optimize the learning of the embedding space and improve the accuracy of image labeling.

[0156] In the embodiment of the present application, by respectively determining the similarity matrix between the category features of the image label and the spatial visual features and the word segmentation text features, and then based on the similarity measurement, pseudo-labeling is performed for the missing image labels with high confidence. Subsequently, these pseudo-labels are combined with the original image annotation information as the supervision information for training the entire model.

[0157] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0158] Based on the same inventive concept, the embodiment of the present application also provides an image labeling device 1300 for implementing the above-mentioned image labeling method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations of one or more embodiments of the image labeling device 1300 provided below can refer to the limitations of the image labeling method above, and will not be repeated here.

[0159] In one embodiment, Figure 4 As shown, an image labeling device 1300 is provided, comprising:

[0160] The data acquisition module 1301 is used to acquire image information, where the image information includes image data, text data and image labels, where the text data and the image labels correspond to the image data respectively, and the image labels include first category label information;

[0161] A feature determination module 1302, configured to determine a visual feature sequence of the image data based on an image coding model, and to determine a text feature sequence of the text data and a category feature of the first category tag information based on a text coding model;

[0162] A model updating module 1303 is used to determine a similarity matrix based on the visual feature sequence and the text feature sequence, and align the similarity matrix to a correction matrix to update the image coding model, wherein the correction matrix is ​​determined based on the image information;

[0163] A second category label information determination module 1304 is used to determine an updated visual feature sequence based on the updated image coding model, and determine second category label information of the image label based on the updated visual feature sequence and the category feature;

[0164] The labeling module 1305 is used to label the unknown category in the first category labeling information based on the second category labeling information.

[0165] In some embodiments, the feature determination module 1302 is also used to segment the image data based on preset segmentation rules to generate multiple image segments; input the multiple image segments into the image coding model, and output a visual feature sequence, which includes global visual features and spatial visual features.

[0166] In some embodiments, the feature determination module 1302 is also used to segment text data based on preset segmentation rules to generate multiple segmented texts; input the multiple segmented texts into a text encoding model, and output a text feature sequence, which includes global text features and segmented text features.

[0167] In some embodiments, the similarity matrix includes a first similarity matrix, and the correction matrix includes a first unit matrix; the model update module 1303 is also used to determine the first similarity matrix between the global visual features and the global text features; the first similarity matrix is ​​aligned to the first unit matrix based on information divergence, and the first unit matrix is ​​determined based on the amount of image data.

[0168] In some embodiments, the similarity matrix includes a fourth similarity matrix, and the correction matrix includes a label matrix; the model update module 1303 is also used to determine a second similarity matrix between spatial visual features and category features, and to determine a third similarity matrix between word segmentation text features and category features; determine aggregated visual features based on the second similarity matrix, and determine aggregated text features based on the third similarity matrix; determine a fourth similarity matrix based on the aggregated visual features and the aggregated text features; align the fourth similarity matrix to the label matrix based on a binary cross entropy loss function, and the label matrix is ​​determined based on the first category label information.

[0169] In some embodiments, the model update module 1303 is also used to determine the image category attention map based on the second similarity matrix and the normalized exponential function; determine the image feature mask based on the image category attention map and the first threshold; and determine the aggregated visual features based on the image feature mask.

[0170] In some embodiments, the model update module 1303 is also used to determine the text category attention map based on the third similarity matrix and the normalized exponential function; determine the text feature mask based on the text category attention map and the second threshold; and determine the aggregated text features based on the text feature mask.

[0171] In some embodiments, the model update module 1303 is also used to stack the first category labeling information a preset number of times to determine a stacking matrix, where the preset number of times includes the number of image labels; and to determine a label matrix based on the product of the stacking matrix and a second unit matrix, where the second unit matrix is ​​determined based on the number of image labels.

[0172] In some embodiments, the second category label information determination module 1304 is also used to determine the image label similarity based on the updated visual feature sequence and the dot product of the category feature; determine the image label correlation probability based on the image label similarity and the normalized exponential function; and determine the second category label information of the image label based on the image label correlation probability, the second threshold and the percentage threshold.

[0173] In some embodiments, the second category labeling information determination module 1304 is further used to sort all image labels from large to small based on the image label correlation probability; among all image labels, the category of the image label with a smaller correlation probability is determined as negative, and the number of image fragments with a smaller image label correlation probability is less than the product of the number of all labels and a percentage threshold; among all image labels, the category of the image label with an image label correlation probability greater than the second threshold is determined as positive; among all image labels, the category of other image labels outside the positive and negative categories is determined as unknown.

[0174] Each module in the above-mentioned image labeling device 1300 can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module above.

[0175] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an image labeling method is implemented.

[0176] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0177] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps of the image labeling method in any of the above embodiments when executing the computer program.

[0178] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the image labeling method in any of the above embodiments are implemented.

[0179] In one embodiment, a computer program product is provided, including a computer program, which, when executed by a processor, implements the steps of the image labeling method in any of the above embodiments.

[0180] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the", "these" and the like in this application do not represent quantitative restrictions, and they can be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether directly or indirectly. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. Usually, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0181] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0182] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0183] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0184] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for labeling an image, characterized in that: The method comprises: Acquire image information, the image information including image data, text data and image labels, the text data and the image labels respectively correspond to the image data, and the image labels include first category label information; Determine a visual feature sequence of the image data based on an image coding model, and determine a text feature sequence of the text data and a category feature of the first category tag information based on a text coding model; Determining a similarity matrix based on the visual feature sequence and the text feature sequence, and aligning the similarity matrix to a correction matrix to update the image coding model, wherein the correction matrix is ​​determined based on the image information; Based on the updated image coding model, determine an updated visual feature sequence, and based on the updated visual feature sequence and the category feature, determine second category marking information of the image label; Based on the second category label information, unknown labels in the image labels are labeled.

2. The method according to claim 1, characterized in that Determining the visual feature sequence of the image data based on the image coding model comprises: Segment the image data based on a preset segmentation rule to generate a plurality of image segments; The multiple image segments are input into the image coding model, and the visual feature sequence is output, where the visual feature sequence includes global visual features and spatial visual features.

3. The method according to claim 2, characterized in that Determining the text feature sequence of the text data based on the text encoding model includes: Segment the text data based on preset word segmentation rules to generate multiple word segmentation texts; The multiple word segmentation texts are input into the text encoding model, and the text feature sequence is output, where the text feature sequence includes global text features and word segmentation text features.

4. The method according to claim 3, characterized in that: The similarity matrix includes a first similarity matrix, and the correction matrix includes a first unit matrix; The determining of a similarity matrix based on the visual feature sequence and the text feature sequence, and aligning the similarity matrix to a correction matrix comprises: Determining the first similarity matrix between the global visual features and the global text features; The first similarity matrix is ​​aligned to the first identity matrix based on information divergence, and the first identity matrix is ​​determined based on the amount of the image data.

5. The method according to claim 3, characterized in that: The similarity matrix includes a fourth similarity matrix, and the correction matrix includes a label matrix; The determining of a similarity matrix based on the visual feature sequence and the text feature sequence, and aligning the similarity matrix to a correction matrix further comprises: Determining a second similarity matrix between the spatial visual feature and the category feature, and determining a third similarity matrix between the segmentation text feature and the category feature; determining aggregated visual features based on the second similarity matrix, and determining aggregated text features based on the third similarity matrix; Determining the fourth similarity matrix based on the aggregated visual features and the aggregated text features; The fourth similarity matrix is ​​aligned to the label matrix based on a binary cross entropy loss function, and the label matrix is ​​determined based on the first category label information.

6. The method according to claim 5, characterized in that The determining of the aggregated visual features based on the second similarity matrix comprises: Determining an image category attention map based on the second similarity matrix and the normalized exponential function; Determining an image feature mask based on the image category attention map and a first threshold; determining the aggregated visual feature based on the image feature mask; Determining the aggregated text features based on the third similarity matrix includes: Determine a text category attention map based on the third similarity matrix and the normalized exponential function; Determining a text feature mask based on the text category attention map and a second threshold; The aggregated text features are determined based on the text feature mask.

7. The method according to claim 5, characterized in that The label matrix is ​​determined based on the following method: Stacking the first category label information a preset number of times to determine a stacking matrix, where the preset number of times includes the number of the image labels; The label matrix is ​​determined based on the product of the stacking matrix and a second identity matrix, wherein the second identity matrix is ​​determined based on the number of the image labels.

8. The method according to claim 1, characterized in that: The determining the second category labeling information of the image label based on the updated visual feature sequence and the category feature comprises: Determining image label similarity based on the updated visual feature sequence and the dot product of the category feature; Determining image tag correlation probabilities based on the image tag similarities and a normalized exponential function; Based on the image tag related probability, the second threshold and the percentage threshold, second category labeling information of the image tag is determined.

9. The method according to claim 1, characterized in that: The determining the second category labeling information of the image label based on the image label correlation probability, the second threshold and the percentage threshold comprises: Sort all image tags from large to small based on the probability of the image tags being associated with them; Determine the category of the image label with a smaller correlation probability among all the image labels as negative, and the number of image segments with a smaller correlation probability of the image labels is less than the product of the number of all labels and the percentage threshold; Determine, among all image labels, the category of the image label whose associated probability is greater than the second threshold as positive; Among all image labels, the categories of other image labels other than positive and negative categories are determined as unknown.

10. An image labeling device, characterized in that: The device comprises: A data acquisition module, used to acquire image information, wherein the image information includes image data, text data and image labels, wherein the text data and the image labels correspond to the image data respectively, and the image labels include first category label information; A feature determination module, used to determine a visual feature sequence of the image data based on an image coding model, and to determine a text feature sequence of the text data and a category feature of the first category tag information based on a text coding model; A model updating module, configured to determine a similarity matrix based on the visual feature sequence and the text feature sequence, and align the similarity matrix to a correction matrix to update the image coding model, wherein the correction matrix is ​​determined based on the image information; A second category label information determination module, configured to determine an updated visual feature sequence based on the updated image coding model, and determine second category label information of the image label based on the updated visual feature sequence and the category feature; A labeling module is used to label unknown labels in the image labels based on the second category label information.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

Images