A label data processing method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a label data processing method and device, electronic equipment and storage medium, the method comprises: obtaining a set of labels to be processed; the set of labels to be processed comprises labels corresponding to a plurality of images; the labels in the set of labels to be processed are generated by a multi-label classification model for the plurality of images; according to the conditional probability between each label and other labels in the set of labels to be processed, determine the relevance between the each label and other labels; according to the relevance between the each label and other labels, determine the isolated label in the set of labels to be processed; delete the isolated label to obtain a first label set. Embodiments of the present application can determine isolated labels according to relevance, delete isolated labels to obtain a first label set, thereby constructing a more reasonable label system and image multi-label data set, which is conducive to the application of image labels obtained through a deep learning network in actual business.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a tag data processing method, a tag data processing device, an electronic device, and a computer-readable storage medium. Background Technology

[0002] The internet generates massive amounts of image data daily, and with continuous technological advancements, research on corresponding deep learning algorithms for images has become increasingly sophisticated in recent years. Deep learning algorithms for images can be based on supervised learning, which involves training deep learning networks using multi-label image datasets. However, labeling images with pre-trained models results in a large number of isolated labels in the resulting multi-label dataset, hindering the practical application of image labels obtained through deep learning networks in real-world business scenarios. Summary of the Invention

[0003] In view of the above problems, embodiments of this application are proposed to provide a tag data processing method, a tag data processing apparatus, an electronic device, and a computer-readable storage medium that overcome or at least partially solve the above problems.

[0004] To address the aforementioned problems, this application discloses a tag data processing method, the method comprising:

[0005] Obtain a set of tags to be processed; the set of tags to be processed includes tags corresponding to multiple images; the tags in the set of tags to be processed are generated by a multi-label classification model for the multiple images;

[0006] The correlation between each tag and other tags is determined based on the conditional probabilities between each tag in the set of tags to be processed and other tags.

[0007] Based on the correlation between each tag and other tags, the isolated tags in the set of tags to be processed are determined;

[0008] Delete the isolated tags to obtain the first tag set.

[0009] Optionally, determining the association between each tag and other tags based on the conditional probabilities between each tag in the set of tags to be processed includes:

[0010] Based on the conditional probabilities between each tag in the set of tags to be processed and other tags, construct a cross-correlation matrix;

[0011] A symmetry transformation is performed on the cross-correlation matrix to obtain a cross-correlation symmetric matrix;

[0012] The correlation between each tag and other tags is determined based on the elements in the cross-correlation symmetric matrix.

[0013] Optionally, constructing a cross-correlation matrix based on the conditional probabilities between each tag in the set of tags to be processed and other tags includes:

[0014] For any tag in the set of tags to be processed, determine the number of images in which each tag corresponds to the same image as any other tag;

[0015] The number of images that each of the above labels corresponds to the same image along with any other label is divided by the total number of images corresponding to each label to obtain at least one conditional probability for each label.

[0016] Using the conditional probabilities as column vectors, construct a cross-correlation matrix.

[0017] Optionally, the step of performing a symmetry transformation on the cross-correlation matrix to obtain a cross-correlation symmetric matrix includes:

[0018] Perform a transpose operation on the cross-correlation matrix to obtain the transpose matrix of the cross-correlation matrix;

[0019] The average value of the elements of the cross-correlation matrix and the elements in the transpose matrix corresponding to the elements of the cross-correlation matrix is ​​calculated to obtain the cross-correlation symmetric matrix.

[0020] Optionally, a column of elements in the cross-correlation symmetric matrix includes multiple label conditional probabilities for the same label; determining the correlation between each label and other labels based on the elements in the cross-correlation symmetric matrix includes:

[0021] Calculate the average value of the conditional probabilities of the same label based on the sum of the conditional probabilities of multiple labels for the same label;

[0022] The correlation between each tag and other tags is determined based on the average conditional probability of the tags for the same tag;

[0023] The step of determining isolated tags in the set of tags to be processed based on the correlation between each tag and other tags includes:

[0024] When the average conditional probability of at least one label is less than a preset average probability threshold, the at least one label is set as an isolated label.

[0025] Optionally, the method further includes:

[0026] Calculate the semantic similarity between two tags in the first tag set;

[0027] When the word sense similarity score is greater than the preset similarity score threshold, the tag corresponding to the word sense similarity is set as a synonym tag that has a synonym relationship;

[0028] The synonym tags are merged into one tag.

[0029] Optionally, the method further includes:

[0030] For each image, determine the confidence level of the label corresponding to that image.

[0031] Determine the number of target labels for each image, wherein the target labels are the labels corresponding to the image whose confidence scores are greater than a preset confidence threshold;

[0032] The average number of labels per image is obtained by dividing the sum of the number of target labels in each image by the total number of images.

[0033] When the average number of labels in the image is greater than a preset average number of labels threshold, the target labels of each image are used as a second label set.

[0034] Delete rare tags from the second tag set to obtain a tag set to be processed; the rare tags are those whose corresponding image quantity ratio is less than a preset ratio threshold.

[0035] This application also discloses a tag data processing apparatus, the apparatus comprising:

[0036] The acquisition module is used to acquire a set of tags to be processed; the set of tags to be processed includes tags corresponding to multiple images; the tags in the set of tags to be processed are generated by a multi-label classification model for the multiple images.

[0037] The correlation determination module is used to determine the correlation between each tag and other tags based on the conditional probability between each tag in the tag set to be processed and other tags;

[0038] The tag determination module is used to determine isolated tags in the set of tags to be processed based on the correlation between each tag and other tags;

[0039] The deletion module is used to delete the isolated tags to obtain the first tag set.

[0040] Optionally, the correlation determination module includes:

[0041] The matrix construction submodule is used to construct a cross-correlation matrix based on the conditional probabilities between each tag in the set of tags to be processed and other tags;

[0042] The transformation submodule is used to perform a symmetric transformation on the cross-correlation matrix to obtain a cross-correlation symmetric matrix;

[0043] The tag association determination submodule is used to determine the association between each tag and other tags based on the elements in the cross-association symmetric matrix.

[0044] Optionally, the matrix construction submodule includes:

[0045] The image quantity determination unit is used to determine, for any tag in the set of tags to be processed, the number of images that each tag corresponds to the same image with any other tag;

[0046] The conditional probability determination unit is used to divide the number of images that each of the given labels corresponds to the same image with any other label by the total number of images corresponding to each label, to obtain at least one conditional probability for each label.

[0047] The cross-correlation matrix construction unit is used to construct a cross-correlation matrix using the conditional probabilities as column vectors.

[0048] Optionally, the conversion submodule includes:

[0049] The transpose operation unit is used to perform a transpose operation on the cross-correlation matrix to obtain the transpose matrix of the cross-correlation matrix;

[0050] The element average calculation unit is used to calculate the average value between the elements of the cross-correlation matrix and the elements in the transpose matrix corresponding to the element positions of the cross-correlation matrix, to obtain a cross-correlation symmetric matrix.

[0051] Optionally, one column of the cross-correlation symmetric matrix includes multiple label conditional probabilities for the same label; the label correlation determination submodule includes:

[0052] The average value calculation unit is used to calculate the average value of the label conditional probabilities for the same label based on the sum of the conditional probabilities of multiple labels for the same label.

[0053] The association determination unit is used to determine the association between each tag and other tags based on the average conditional probability of the tags for the same tag;

[0054] The label determination module includes:

[0055] The isolated label determination submodule is used to set the at least one label as an isolated label when the average conditional probability of at least one label is less than a preset average probability threshold.

[0056] Optionally, the device further includes:

[0057] The semantic similarity calculation module is used to calculate the semantic similarity between two tags in the first tag set;

[0058] The synonym tag determination module is used to set the tag corresponding to the word sense similarity as a synonym tag that has a synonym relationship when the score of the word sense similarity is greater than a preset similarity score threshold.

[0059] The merging module is used to merge the synonym tags into a single tag.

[0060] Optionally, the device further includes:

[0061] The confidence determination module is used to determine the confidence level of the label corresponding to each image for each image.

[0062] The target label quantity determination module is used to determine the number of target labels for each image, wherein the target labels are the labels corresponding to the image whose confidence scores are greater than a preset confidence threshold.

[0063] The average label count determination module is used to divide the sum of the number of target labels in each image by the total number of images to obtain the average number of labels in the image.

[0064] The confidence threshold determination module is used to take the target labels of each image as a second label set when the average number of labels of the image is greater than a preset average number of labels threshold.

[0065] The rare tag deletion module is used to delete rare tags in the second tag set to obtain a tag set to be processed; the rare tags are tags whose corresponding image quantity ratio is less than a preset ratio threshold.

[0066] This application also discloses an electronic device, including: a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the steps of the data verification method described above.

[0067] This application also discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the data verification method described above.

[0068] The embodiments of this application have the following advantages:

[0069] In this embodiment, the correlation between each label and other labels in the label set to be processed can be determined based on the conditional probability between each label and other labels, and isolated labels can be identified based on the correlation, thereby deleting isolated labels from the label set to obtain a first label set, constructing a more reasonable label system and image multi-label data set; using the first label set to train a deep learning network is beneficial for the application of image labels obtained through the deep learning network in actual business. Attached Figure Description

[0070] Figure 1 This is a flowchart of the steps of a tag data processing method provided in an embodiment of this application;

[0071] Figure 2 This is a flowchart of another tag data processing method provided in an embodiment of this application;

[0072] Figure 3 This is a structural block diagram of a tag data processing device provided in an embodiment of this application. Detailed Implementation

[0073] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, this application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0074] In image deep learning algorithms, multi-label image datasets can be used to train deep learning networks. These datasets are typically constructed using either manual or machine annotation methods. Manual annotation requires manually labeling millions of images with hundreds or thousands of labels, consuming significant time and manpower. Machine annotation, on the other hand, usually involves inputting a large number of unlabeled images into a pre-trained model to obtain an initial label set, then setting a confidence score threshold to filter out labels with high confidence scores, ultimately resulting in the multi-label image dataset. However, machine annotation often results in a large number of isolated labels in the resulting dataset. Training a deep learning network with such a dataset hinders the practical application of the image labels obtained through the deep learning network.

[0075] The core concept of this application embodiment is that, in this application embodiment, based on the conditional probability between each label in the label set to be processed and other labels, the correlation between each label and other labels can be determined, and isolated labels can be identified based on the correlation, thereby deleting isolated labels in the label set to obtain a first label set, constructing a more reasonable label system and image multi-label data set; using the first label set to train a deep learning network is beneficial for the application of image labels obtained through the deep learning network in actual business.

[0076] Reference Figure 1 The diagram illustrates a flowchart of a tag data processing method provided in an embodiment of this application. The method may specifically include the following steps:

[0077] Step 101: Obtain a set of tags to be processed; the set of tags to be processed includes tags corresponding to multiple images; the tags in the set of tags to be processed are generated by a multi-label classification model for the multiple images.

[0078] In this embodiment, an open-source multi-label dataset can be used to pre-train a multi-label classification model. Multiple labels corresponding to each image in the label set to be processed can be generated by the multi-label classification model for multiple unlabeled images. For example, an Open Image open-source dataset with broad coverage and numerous labels can be selected, and a multi-label classification model can be pre-trained.

[0079] Step 102: Determine the correlation between each tag and other tags based on the conditional probabilities between each tag and other tags in the set of tags to be processed.

[0080] The correlation between each tag and other tags can be used to represent the degree of relevance of each tag to other tags within a tag group consisting of all tags. For example, tags A, B, and C can form a tag group. The degree of relevance of tag A to tags B and C within the tag group consisting of tags A, B, and C can be determined based on the conditional probabilities P(A|B) and P(B|A) between tag A and tag B, and the conditional probabilities P(A|C) and P(C|A) between tag A and tag C.

[0081] Step 103: Based on the correlation between each tag and other tags, determine the isolated tags in the set of tags to be processed.

[0082] In this embodiment of the application, each tag can be sorted based on the degree of relevance between each tag and other tags in a tag group consisting of all tags, and isolated tags can be determined based on the sorting results.

[0083] Step 104: Delete the isolated tags to obtain the first tag set.

[0084] In business scenarios, the server identifies isolated labels and removes them from the set of labels to be processed, obtaining a first set of labels. This first set of labels can then be used to train a deep learning network. The trained deep learning network can then be used to perform multi-labeling on unlabeled images, resulting in labels for multiple images. Finally, business applications can be performed based on these labels.

[0085] For example, in a forum post recommendation scenario, a user can browse posts published by other users. The server can use a deep learning network trained with a first set of labels to multi-label all images in the posts viewed by the user, obtaining multiple labels for each image. Based on all image labels, the server matches posts and recommends the post with the highest matching degree to the user. In this embodiment, all image labels do not include isolated labels, which avoids the influence of isolated labels on post matching, thus recommending posts to the user more reasonably.

[0086] In this embodiment, the correlation between each label and other labels in the label set to be processed can be determined based on the conditional probability between each label and other labels, and isolated labels can be identified based on the correlation, thereby deleting isolated labels from the label set to obtain a first label set, constructing a more reasonable label system and image multi-label data set; using the first label set to train a deep learning network is beneficial for the application of image labels obtained through the deep learning network in actual business.

[0087] Reference Figure 2 The diagram illustrates a flowchart of a tag data processing method provided in an embodiment of this application. The method may specifically include the following steps:

[0088] Step 201: Obtain a set of tags to be processed; the set of tags to be processed includes tags corresponding to multiple images; the tags in the set of tags to be processed are generated by a multi-label classification model for the multiple images.

[0089] Step 202: Construct a cross-correlation matrix based on the conditional probabilities between each tag in the set of tags to be processed and other tags.

[0090] In an optional embodiment, step 202 may include the following sub-steps S11-S13:

[0091] Sub-step S11: For any tag in the set of tags to be processed, determine the number of images that each tag corresponds to the same image with any other tag.

[0092] For any tag in the set of tags to be processed, the number of images that each tag corresponds to in the same image with any other tag can be determined. For example, the tags corresponding to image x can be tags A, B, and C; the tags corresponding to image y can be tags A and B; and the tags corresponding to image z can be tags C and D. For tag A, if tag A and tag B both correspond to images x and y, then the number of images that tag A and tag B both correspond to in the same image can be 2; if tag A and tag C both correspond to image z, then the number of images that tag A and tag C both correspond to in the same image can be 1; if tag A and tag D do not have any corresponding images, then the number of images that tag A and tag D both correspond to in the same image can be 0. For tag C, the number of images that tag C corresponds to in the same image with tag A is 1, the number of images that tag C corresponds to in the same image with tag B is 1, and the number of images that tag C corresponds to in the same image with tag D is 1.

[0093] Sub-step S12: Divide the number of images that each of the given labels corresponds to in the same image with any other label by the total number of images corresponding to each label to obtain at least one conditional probability for each label.

[0094] The conditional probability for any label can be represented by P(X|W), where W can represent any label in the set of labels to be processed, and X can represent any other label. For example, for label A, the conditional probability may include P(A|A), P(B|A), P(C|A), P(D|A), etc.; for label C, the conditional probability may include P(A|C), P(B|C), P(C|C), P(D|C), etc.

[0095] The number of images that each label corresponds to in the same image as any other label can be divided by the total number of images corresponding to any given label to obtain at least one conditional probability for that label. For example, the label corresponding to image x can be A, B, or C; the label corresponding to image y can be A or B; and the label corresponding to image z can be C or D. Here, if label A corresponds to both image x and image y, then the total number of images corresponding to label A can be 2; if label C corresponds to both image x and image z, then the total number of images corresponding to label C can also be 2. For label A, P(A|A) is calculated to be 1, P(B|A) is 1, P(C|A) is 0.5, and P(D|A) is 0; for label C, P(A|C) is 0.5, P(B|C) is 0.5, P(C|C) is 1, and P(D|C) is 0.5.

[0096] It should be noted that, based on the meaning of conditional probability, P(A|A) can be determined to be 1. Similarly, P(B|B), P(C|C), and P(D|D) can also be determined to be 1.

[0097] Sub-step S13: Construct a cross-correlation matrix using the conditional probabilities as column vectors.

[0098] In this embodiment, a cross-correlation matrix can be constructed using at least one conditional probability for any label as a column vector. For example, when the set of labels to be processed includes labels A, B, C, D, and E, the constructed cross-correlation matrix can be as follows:

[0099]

[0100] It should be noted that the size of the cross-correlation matrix can be determined based on the number of tags in the tag set to be processed. For example, when the tag set to be processed includes tags A, B, and C, there are 3 tags, and the resulting cross-correlation matrix is ​​a 3×3 matrix; when the tag set to be processed includes tags A, B, C, and D, there are 4 tags, and the resulting cross-correlation matrix is ​​a 4×4 matrix.

[0101] Step 203: Perform a symmetry transformation on the cross-correlation matrix to obtain a cross-correlation symmetric matrix.

[0102] In an optional embodiment, step 203 may include the following sub-steps S21-S23:

[0103] Sub-step S21: Perform a transpose operation on the cross-correlation matrix to obtain the transpose matrix of the cross-correlation matrix.

[0104] Specifically, the rows and columns of the cross-correlation matrix can be interchanged to obtain the transpose of the cross-correlation matrix. The transpose matrix can be represented by Y. T express.

[0105] Sub-step S22: Calculate the average value of the elements of the cross-correlation matrix and the elements in the transpose matrix corresponding to the positions of the elements of the cross-correlation matrix to obtain the cross-correlation symmetric matrix.

[0106] The cross-correlation matrix can be symmetrically transformed by transposing the matrix. The average value of the elements at each position of the cross-correlation matrix and the corresponding elements in the transposed matrix can be calculated. The average value of the elements at each position can then be used as the element in the corresponding position of the cross-correlation symmetric matrix, thus obtaining the cross-correlation symmetric matrix.

[0107] Specifically, it can be done through the calculation formula Y avg =(Y+Y)T Calculate the average of the elements in the i-th row and j-th column of the cross-correlation matrix and the elements in the i-th row and j-th column of the transpose matrix. Use this average as the element in the i-th row and j-th column of the cross-correlation symmetric matrix to obtain the cross-correlation symmetric matrix. For example, if the element in the second row and first column of the cross-correlation matrix is ​​P(B|A) and the element in the second row and first column of the transpose matrix is ​​P(A|B), calculating the average of P(B|A) and P(A|B) gives the element in the second row and first column of the cross-correlation symmetric matrix as (P(B|A)+P(A|B)) / 2.

[0108] For example, when the set of tags to be processed includes tags A, B, and C, the resulting cross-correlation symmetric matrix can be as follows:

[0109]

[0110] Step 204: Determine the association between each tag and other tags based on the elements in the cross-correlation symmetric matrix.

[0111] In an optional embodiment, a column of elements in the cross-correlation symmetric matrix includes multiple label conditional probabilities for the same label, and step 204 may include the following sub-steps S31-S32:

[0112] Sub-step S31: Calculate the average value of the label conditional probabilities for the same label based on the sum of the multiple label conditional probabilities for the same label.

[0113] Sub-step S32: Determine the correlation between each tag and other tags based on the average conditional probability of the tags for the same tag.

[0114] Label conditional probabilities can be used to represent the association between two labels. In a cross-correlation symmetric matrix, each column can include multiple label conditional probabilities for the same label. The average of the multiple label conditional probabilities for the same label in each column can be calculated. Based on the average of the label conditional probabilities for the same label, the association between that label and other labels can be determined. For example, the first column of elements for label A can include label conditional probabilities (P(A|A)+P(A|A)) / 2, (P(B|A)+P(A|B)) / 2, and (P(C|A)+P(A|C)) / 2. Adding (P(A|A)+P(A|A)) / 2, (P(B|A)+P(A|B)) / 2, and (P(C|A)+P(A|C)) / 2 and dividing by the number of label conditional probabilities yields the average label conditional probability for label A: {(P(A|A)+P(A|A)) / 2+(P(B|A)+P(A|B)) / 2+(P(C|A)+P(A|C)) / 2)} / 3. This average label conditional probability for label A can be used to represent the degree of relevance of label A to other labels within a label group consisting of labels A, B, and C.

[0115] Step 205: Based on the correlation between each tag and other tags, determine the isolated tags in the set of tags to be processed.

[0116] In an optional embodiment, step 205 may include: when the average conditional probability of at least one label is less than a preset average probability threshold, setting the at least one label as an isolated label.

[0117] For example, the average probability threshold can be preset to 0.06. When the average conditional probability threshold for label A is less than 0.06, it means that label A has a low degree of correlation with other labels in the label group consisting of labels A, B, and C. Label A is not compact in the label group, and label A can be determined to be an isolated label.

[0118] Step 206: Delete the isolated tags to obtain the first tag set.

[0119] After identifying the isolated tags, the isolated tags in the tag set to be processed can be deleted to obtain the first tag set.

[0120] In an optional embodiment, the method may further include: calculating the semantic similarity between two tags in the first tag set; when the semantic similarity score is greater than a preset similarity score threshold, setting the tag corresponding to the semantic similarity as a synonym tag with a synonym relationship; and merging the synonym tags into one tag.

[0121] Specifically, the WordNet similarity measurement tool can be used to calculate the semantic similarity between every two tags in the first tag set. When the calculated semantic similarity score is greater than a preset similarity score threshold, the two tags can be set as synonyms and merged. For example, if the semantic similarity between the tags "bathroom" and "washroom" is greater than the preset threshold, then the tags "bathroom" and "washroom" are set as synonyms and merged; similarly, if the semantic similarity between the tags "dining table" and "dining table" is greater than the preset threshold, then the tags "dining table" and "dining table" are set as synonyms and merged.

[0122] In an optional embodiment, the method may further include: for each image, determining the confidence level of the corresponding label for the image; determining the number of target labels for each image, wherein the target labels are the labels for the image whose confidence scores are greater than a preset confidence threshold; performing a division operation between the sum of the number of target labels for each image and the total number of images to obtain the average number of labels for each image; when the average number of labels for each image is greater than a preset average number of labels threshold, using the target labels for each image as a second label set; deleting rare labels from the second label set to obtain a label set to be processed; wherein the rare labels are the labels whose corresponding image quantity proportion is less than a preset proportion threshold.

[0123] Specifically, an unlabeled image dataset can be input into a pre-trained multi-label classification model for processing to obtain the labels corresponding to each image and their confidence scores. Different images may have different confidence scores for each label. Then, the number of labels with confidence scores greater than a preset confidence threshold for each image is determined; this is the number of target labels. The sum of the number of target labels for each image is divided by the total number of images to obtain the average number of labels per image. For example, images x and y can be input into the pre-trained multi-label classification model. For image x, the confidence scores for label A are 0.9, label B is 0.95, and label C is 0.1; for image y, the confidence scores for label B are 0.9 and label D is 0.2. A preset confidence threshold of 0.8 can be used to determine that the target labels for image x are labels A and B, with a quantity of 2; and the target label for image y is label B, with a quantity of 1. Therefore, the total number of target labels can be 3.

[0124] When the average number of labels per image exceeds a preset average label count threshold, the target labels of each image can be used as a second label set. The number of images corresponding to each label and the total number of images in the second label set are then statistically determined. Finally, the number of images corresponding to each label is divided by the total number of images to obtain the percentage of images corresponding to each label. For example, the preset average label count threshold can be 6.5.

[0125] When the average number of labels on an image is less than the preset average number of labels threshold, the preset confidence threshold can be adjusted to make the average number of labels on the image greater than the preset average number of labels threshold.

[0126] In this embodiment, a preset average tag count threshold can be set to avoid the problem of insufficient tags corresponding to images hindering business applications. For example, in a blog post recommendation scenario, when matching posts based on all image tags and recommending the post with the highest matching degree to the user, the impact of insufficient tags corresponding to images on post matching can be avoided, resulting in more reasonable post recommendations to the user.

[0127] It can be determined whether the proportion of images corresponding to each label is less than a preset proportion threshold. When the proportion of images corresponding to at least one label is less than the preset proportion threshold, these labels can be set as rare labels and deleted, thereby obtaining a set of labels to be processed. For example, the preset proportion threshold can be 0.001.

[0128] Those skilled in the art should understand that the above-mentioned preset average label quantity threshold and preset percentage threshold are merely examples of this application. Those skilled in the art can set different thresholds according to actual needs, and this application does not impose any restrictions here.

[0129] In this embodiment, after obtaining the tag set using a machine annotation scheme, by determining whether the proportion of images corresponding to each tag is less than a preset proportion threshold, tags with a small proportion can be identified as rare tags. Deleting rare tags yields a tag set to be processed. Based on the correlation between each tag in the tag set and other tags, isolated tags can be identified, and deleting isolated tags yields a first tag set. By calculating the semantic similarity between two tags in the first tag set, synonymous tags can be identified, and deleting identical tags yields the final tag set. By deleting rare and isolated tags and merging synonymous tags in the tag set obtained using the machine annotation scheme, a more reasonable tag system and a multi-tag image dataset can be constructed. Using the final tag set to train a deep learning network is beneficial for the application of image tags obtained through the deep learning network in practical business applications.

[0130] To enable those skilled in the art to better understand the embodiments of this application, an example is provided below to illustrate the embodiments of this application:

[0131] (1) Obtain an unlabeled image dataset. Input the unlabeled image dataset into a pre-trained multi-label classification model for processing to obtain the labels corresponding to the images and the confidence scores of the labels. Then, determine the number of labels for each image whose confidence scores are greater than a preset confidence threshold, i.e., the number of target labels. Divide the sum of the number of target labels for each image by the total number of images to obtain the average number of labels per image. When the average number of labels per image is greater than the preset average number of labels threshold, the target labels of each image are used as a label set.

[0132] (2) Determine the number of images corresponding to each label in the label set. Divide the number of images corresponding to each label by the total number of images to obtain the proportion of images corresponding to each label. When the proportion of images corresponding to a label is less than a preset proportion threshold, the label can be set as a rare label, and the rare labels can be deleted to obtain the label set to be processed. For example, when the proportion of images corresponding to labels F and G is less than 0.001, labels F and G can be deleted to obtain the label set to be processed.

[0133] (3) For any tag in the set of tags to be processed, determine the number of images that each tag corresponds to in the same image as any other tag. For example, for tag A, we can obtain the number of images that tag A corresponds to in the same image as any other tag, such as tags A and B, tags A and C, tags A and D; for tag B, we can obtain the number of images that tag B corresponds to in the same image as any other tag, such as tags B and A, tags B and C, tags B and D.

[0134] (4) Calculate the number of images that each label corresponds to in the same image as any other label, and the total number of images corresponding to any label, to obtain at least one conditional probability for any label. For example, the conditional probability for label A may include P(A|A), P(B|A), P(C|A), P(D|A), etc.; the conditional probability for label B may include P(A|B), P(B|B), P(C|B), P(D|B), etc.

[0135] (5) Construct a cross-correlation matrix using at least one conditional probability for any label as a column vector. For example, if the number of labels in the set of labels to be processed is x, a cross-correlation matrix Y can be constructed using P(A|A), P(B|A), P(C|A), ..., P(X|A) as the elements of the first column of the matrix, P(A|B), P(B|B), P(C|B), ..., P(X|B) as the elements of the second column of the matrix, ..., and P(A|X), P(B|X), P(C|X), ..., P(X|X) as the elements of the x-th column of the matrix.

[0136] (6) Through Y avg =(Y+Y) T ) / 2, calculate the average of the elements in the i-th row and j-th column of the cross-correlation matrix and the elements in the i-th row and j-th column of the transpose matrix, and use the calculated average as the element in the i-th row and j-th column of the cross-correlation symmetric matrix to obtain the cross-correlation symmetric matrix Y. avg .

[0137] (7) Calculate the cross-correlation symmetric matrix Y avg The average conditional probability of multiple tags for the same label in each column is used to determine the association between the label and other labels. When the average conditional probability of one or more labels is less than a preset threshold, one or more labels can be set as isolated labels, and the isolated labels are deleted from the label set to be processed to obtain the first label set.

[0138] (8) Calculate the semantic similarity between two tags in the first tag set. When the semantic similarity score is greater than the preset similarity score threshold, set the corresponding tag as a synonym tag and merge the synonym tags to obtain the final tag set.

[0139] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of this application are not limited to the described order of actions, because according to the embodiments of this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of this application.

[0140] Reference Figure 3 The diagram shows a structural block diagram of a tag data processing device provided in an embodiment of this application, which may specifically include the following modules:

[0141] The acquisition module 301 is used to acquire a set of tags to be processed; the set of tags to be processed includes tags corresponding to multiple images; the tags in the set of tags to be processed are generated by a multi-label classification model for the multiple images.

[0142] The correlation determination module 302 is used to determine the correlation between each tag and other tags based on the conditional probability between each tag in the tag set to be processed and other tags;

[0143] The tag determination module 303 is used to determine isolated tags in the tag set to be processed based on the correlation between each tag and other tags;

[0144] The deletion module 304 is used to delete the isolated tags to obtain the first tag set.

[0145] In this embodiment of the application, the correlation determination module includes:

[0146] The matrix construction submodule is used to construct a cross-correlation matrix based on the conditional probabilities between each tag in the set of tags to be processed and other tags;

[0147] The transformation submodule is used to perform a symmetric transformation on the cross-correlation matrix to obtain a cross-correlation symmetric matrix;

[0148] The tag association determination submodule is used to determine the association between each tag and other tags based on the elements in the cross-association symmetric matrix.

[0149] In this embodiment of the application, the matrix construction submodule includes:

[0150] The image quantity determination unit is used to determine, for any tag in the set of tags to be processed, the number of images that each tag corresponds to the same image with any other tag;

[0151] The conditional probability determination unit is used to divide the number of images that each of the given labels corresponds to the same image with any other label by the total number of images corresponding to each label, to obtain at least one conditional probability for each label.

[0152] The cross-correlation matrix construction unit is used to construct a cross-correlation matrix using the conditional probabilities as column vectors.

[0153] In this embodiment of the application, the conversion submodule includes:

[0154] The transpose operation unit is used to perform a transpose operation on the cross-correlation matrix to obtain the transpose matrix of the cross-correlation matrix;

[0155] The element average calculation unit is used to calculate the average value between the elements of the cross-correlation matrix and the elements in the transpose matrix corresponding to the element positions of the cross-correlation matrix, to obtain a cross-correlation symmetric matrix.

[0156] In this embodiment, a column of elements in the cross-correlation symmetric matrix includes multiple label conditional probabilities for the same label; the label correlation determination submodule includes:

[0157] The average value calculation unit is used to calculate the average value of the label conditional probabilities for the same label based on the sum of the conditional probabilities of multiple labels for the same label.

[0158] The association determination unit is used to determine the association between each tag and other tags based on the average conditional probability of the tags for the same tag;

[0159] The label determination module includes:

[0160] The isolated label determination submodule is used to set the at least one label as an isolated label when the average conditional probability of at least one label is less than a preset average probability threshold.

[0161] In this embodiment of the application, the device further includes:

[0162] The semantic similarity calculation module is used to calculate the semantic similarity between two tags in the first tag set;

[0163] The synonym tag determination module is used to set the tag corresponding to the word sense similarity as a synonym tag that has a synonym relationship when the score of the word sense similarity is greater than a preset similarity score threshold.

[0164] The merging module is used to merge the synonym tags into a single tag.

[0165] In this embodiment of the application, the device further includes:

[0166] The confidence determination module is used to determine the confidence level of the label corresponding to each image for each image.

[0167] The target label quantity determination module is used to determine the number of target labels for each image, wherein the target labels are the labels corresponding to the image whose confidence scores are greater than a preset confidence threshold.

[0168] The average label count determination module is used to divide the sum of the target labels of each image by the total number of images to obtain the average label count of each image; wherein, the labels corresponding to each image are obtained by processing the unlabeled image dataset into a pre-trained multi-label classification model;

[0169] The confidence threshold determination module is used to take the target labels of each image as a second label set when the average number of labels of the image is greater than a preset average number of labels threshold.

[0170] The rare tag deletion module is used to delete rare tags in the second tag set to obtain a tag set to be processed; the rare tags are tags whose corresponding image quantity ratio is less than a preset ratio threshold.

[0171] In this embodiment, the correlation between each label and other labels in the label set to be processed can be determined based on the conditional probability between each label and other labels, and isolated labels can be identified based on the correlation, thereby deleting isolated labels from the label set to obtain a first label set, constructing a more reasonable label system and image multi-label data set; using the first label set to train a deep learning network is beneficial for the application of image labels obtained through the deep learning network in actual business.

[0172] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0173] This application also provides an electronic device, including:

[0174] It includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, it implements the various processes of the above-described tag data processing method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0175] This application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described tag data processing method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0176] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0177] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of this application can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0178] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0179] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0180] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0181] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0182] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0183] The above provides a detailed description of a tag data processing method, apparatus, electronic device, and storage medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A label data processing method characterized by, The method comprises: acquiring a set of labels to be processed; the set of labels to be processed comprises labels corresponding to a plurality of images; the labels in the set of labels to be processed are generated by a multi-label classification model for the plurality of images; determining the relevance between each label and other labels in the set of labels to be processed according to the conditional probability between each label and other labels in the set of labels to be processed, comprising: constructing an inter-relevance matrix according to the conditional probability between each label and other labels in the set of labels to be processed; performing symmetry conversion on the inter-relevance matrix to obtain an inter-relevance symmetric matrix; determining the relevance between each label and other labels according to the elements in the inter-relevance symmetric matrix; determining isolated labels in the set of labels to be processed according to the relevance between each label and other labels; deleting the isolated labels to obtain a first set of labels; determining the confidence score of the label corresponding to each image for the image; determining the number of target labels of each image, the target label being a label in the label corresponding to the image, the confidence score of which is greater than a preset confidence threshold; performing division operation on the sum of the number of target labels of each image and the total number of images to obtain the average number of labels of images; when the average number of labels of images is greater than a preset average number of label threshold, taking the target labels of each image as a second set of labels; deleting rare labels in the second set of labels to obtain a set of labels to be processed; the rare label is a label whose corresponding image quantity proportion is less than a preset proportion threshold; when the average number of labels of images is less than the preset average number of label threshold, adjusting the preset confidence threshold so that the average number of labels of images is greater than the preset average number of label threshold.

2. The method of claim 1, wherein, The method comprises: for any label in the set of labels to be processed, determining the number of images corresponding to the same image in which the any label and any other label coexist; performing division operation on the number of images corresponding to the same image in which the any label and any other label coexist, and the total number of images corresponding to the any label to obtain at least one conditional probability for the any label; constructing an inter-relevance matrix with the conditional probability as a column vector.

3. The method of claim 1, wherein, The method comprises: performing transpose operation on the inter-relevance matrix to obtain a transpose matrix of the inter-relevance matrix; calculating the element average value between the elements of the inter-relevance matrix and the elements in the transpose matrix corresponding to the element positions of the inter-relevance matrix to obtain an inter-relevance symmetric matrix.

4. The method of claim 3, wherein, A column of elements in the inter-relevance symmetric matrix comprises a plurality of label conditional probabilities for the same label; the method comprises: calculating the label conditional probability average for the same label according to the sum of the plurality of label conditional probabilities for the same label; determine the relevance between the respective label and other labels according to the average label conditional probability of the respective label; the determining of the isolated label in the label set to be processed according to the relevance between the respective label and other labels comprises: when the average label conditional probability of at least one label is less than a preset average probability threshold, the at least one label is set as an isolated label.

5. The method of claim 1, wherein, The method further comprises: calculating the word sense similarity between two labels in the first label set; when the score of the word sense similarity is greater than a preset similarity score threshold, the label corresponding to the word sense similarity is set as a synonymous label having a synonym relationship; merging the synonymous labels into one label.

6. A label data processing apparatus characterized by comprising: The device comprises: an acquisition module configured to acquire a label set to be processed; the label set to be processed comprises labels corresponding to a plurality of images; the labels in the label set to be processed are generated by a multi-label classification model for the plurality of images; a relevance determination module configured to determine the relevance between the respective label and other labels according to the conditional probability between the respective label and other labels in the label set to be processed, comprising: a matrix construction submodule configured to construct a mutual relevance matrix according to the conditional probability between the respective label and other labels in the label set to be processed; a conversion submodule configured to perform symmetry conversion on the mutual relevance matrix to obtain a mutual relevance symmetric matrix; and a label relevance determination submodule configured to determine the relevance between the respective label and other labels according to elements in the mutual relevance symmetric matrix; a label determination module configured to determine an isolated label in the label set to be processed according to the relevance between the respective label and other labels; a deletion module configured to delete the isolated label to obtain a first label set; a confidence determination module configured to determine, for each image, the confidence of the label corresponding to the image for the image; a target label number determination module configured to determine the number of target labels of each image, the target label being a label corresponding to the image and having a confidence score greater than a preset confidence threshold; an average label number determination module configured to obtain an average number of labels per image by performing a division operation on the sum of the number of target labels of each image and the total number of images; a confidence threshold judgment module configured to, when the average number of labels per image is greater than a preset average label number threshold, take the target labels of each image as a second label set; a rare label deletion module configured to delete a rare label in the second label set to obtain a label set to be processed; the rare label being a label corresponding to an image number ratio less than a preset ratio threshold; the confidence threshold judgment module is further configured to, when the average number of labels per image is less than a preset average label number threshold, adjust the preset confidence threshold so that the average number of labels per image is greater than the preset average label number threshold.

7. An electronic device, comprising: ​ A processor, a memory, and a computer program stored on the memory and executable on the processor, which, when executed by the processor, implement the steps of the label data processing method according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, A computer program stored on the computer-readable storage medium, which, when executed by a processor, implements the steps of the label data processing method according to any one of claims 1-5.

Images