Image classification methods, apparatus, devices and computer-readable storage media
By generating image verification codes and receiving user labeling results, the problem of inaccurate manual labeling is solved, thus improving the accuracy and effectiveness of image classification.
Patent Information
- Application Number
- CN202111171582.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-10-08
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2041-10-08
AI Technical Summary
In existing technologies, labeling images based on subjective human judgment is prone to inaccurate labeling results, leading to image classification errors and reducing the accuracy of image classification.
An image verification code is generated by acquiring classified images, images to be classified, classification labels, initial labels, and standard labels. The system receives the user's labeling results for the image verification code and verifies the labeling results. If the verification is successful, the classification label of the image to be classified is determined based on the labeling results and the initial labels.
It improves the accuracy of image classification by ensuring the validity and accuracy of the labeling results through multiple labeling and verification processes, thereby reducing the impact of human labeling errors.
Smart Images

Figure CN115964525B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer technology, and in particular to an image classification method, apparatus, device, and computer-readable storage medium. Background Technology
[0002] With the continuous development of network technology, more and more fields require the use of well-classified images, such as machine learning model training, classification of large numbers of images, and login verification. Taking machine learning model training as an example, in image recognition, a set of labeled image samples is needed for the machine learning model to train.
[0003] When labeling images, image samples can be labeled manually. However, relying on subjective human judgment to label images is prone to inaccurate labeling results, leading to incorrect image classification and reducing the accuracy of image classification. Summary of the Invention
[0004] The present invention aims to provide an image classification method, apparatus, device, and computer-readable storage medium. The classified image and classification label in the image verification code can verify the labeling result, thereby improving the effectiveness of the labeling result. The image to be classified is labeled multiple times, and the classification label is determined by combining the labeled label and the initial label, thereby improving the accuracy of image classification.
[0005] The technical solution of this invention is implemented as follows:
[0006] In a first aspect, embodiments of the present invention provide an image classification method, the method comprising: acquiring a classified image, a classification label of the classified image, an image to be classified, an initial label of the image to be classified, and a standard label, wherein the standard label is different from both the classification label and the initial label; generating an image verification code based on the classified image, the image to be classified, the classification label, the initial label, and the standard label; receiving a user's labeling result for the image verification code and verifying the labeling result; if the verification is successful, determining the classification label of the image to be classified based on the labeling label of the image to be classified and the initial label in the labeling result.
[0007] Secondly, embodiments of the present invention provide an image classification device, the device comprising: a storage and calculation module for storing a classified image library, a to-be-classified image library, and a tag library; an image input and output module for obtaining classified images and classification tags of the classified images from the classified image library, obtaining images to be classified and initial tags of the images to be classified from the to-be-classified image library, and obtaining standard tags from the tag library, wherein the standard tags are different from the classification tags and the initial tags; a tag list generation module for generating a tag list based on the classification tags, the initial tags, and the standard tags; an image verification code generation module for generating an image verification code based on the classified images, the to-be-classified images, and the tag list; a verification module for receiving the tagging result of the image verification code by a user and verifying the tagging result; and a storage and calculation module further for determining the classification tag of the to-be-classified image based on the tagging tag of the to-be-classified image and the initial tag in the tagging result if the verification is successful.
[0008] Thirdly, embodiments of the present invention provide an image classification device, the device including a memory for storing executable instructions and a processor for executing the executable instructions stored in the memory to implement the above-described image classification method.
[0009] Fourthly, embodiments of the present invention provide a computer-readable storage medium having executable instructions stored thereon for implementing the above-described image classification method when executed by a processor.
[0010] This invention provides an image classification method, apparatus, device, and computer-readable storage medium. According to the scheme provided by this invention, a classified image, a classification label of the classified image, an image to be classified, an initial label of the image to be classified, and a standard label are obtained. The standard label is different from the classification label and the initial label. The initial label is a label already applied to the image to be classified. An image verification code is generated based on the classified image, the image to be classified, the classification label, the initial label, and the standard label. The classified image and classification label in the image verification code can verify the labeling result, thereby improving the validity and accuracy of the labeling result. The labeling result of the image verification code is received from the user and verified. If the verification is successful, the classification label of the image to be classified is determined based on the label of the image to be classified and the initial label in the labeling result. That is, the image to be classified is labeled multiple times. Compared with the method of determining the classification label after only one labeling of the image to be classified, this invention combines the label and the initial label to determine the classification label, improving the accuracy of image classification. Attached Figure Description
[0011] Figure 1A flowchart illustrating optional steps of an image classification method provided in an embodiment of the present invention;
[0012] Figure 2 This is an exemplary schematic diagram of an image verification code provided in an embodiment of the present invention;
[0013] Figure 3 A flowchart illustrating optional steps of another image classification method provided in an embodiment of the present invention;
[0014] Figure 4 This is a schematic diagram of the structure of an image classification device provided in an embodiment of the present invention;
[0015] Figure 5 This is a schematic diagram of the composition structure of an image classification device provided in an embodiment of the present invention. Detailed Implementation
[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. It should be understood that some embodiments described herein are merely illustrative of the technical solutions of the present invention and are not intended to limit the scope of the present invention.
[0017] The image classification method provided in this invention can be applied, but is not limited to, to the following scenarios: verification before users log in to websites, pay bills, or unlock new games, as well as training samples used for a large number of image classification and machine learning model training.
[0018] Taking machine learning model training as an example, the initial image sample set labeling methods in related technologies can include manual labeling and automatic labeling. Manual labeling relies on subjective human judgment to assign labels to images, which is prone to misclassification and reduces the accuracy of image classification. Automatic labeling methods include two approaches: one is machine labeling that first clusters images through image recognition and then classifies them, which is prone to mislabeling and requires human intervention to be fully automated; the other is using a pre-labeled image library to perform image recognition and then label the images, but this method cannot label unclassified images.
[0019] This invention provides an image classification method, such as... Figure 1 As shown, Figure 1 This is an optional flowchart of an image classification method provided in an embodiment of the present invention. The image classification method includes the following steps:
[0020] S101. Obtain the classified image, the classification label of the classified image, the image to be classified, the initial label of the image to be classified, and the standard label, wherein the standard label is different from the classification label and the initial label.
[0021] In this embodiment of the invention, images selected from a pre-classified image library are used as pre-classified images. The pre-classified image library can be understood as a library of sample images that have been clearly categorized, storing multiple pre-classified images and their respective labels. Images selected from a library of images to be classified are used as images to be classified. The library of images to be classified can be understood as a library of sample images to be classified, storing multiple images to be classified and their respective labels.
[0022] If an image included in the image to be classified has been tagged once, and that tag is used as the image's tag, inaccurate labeling will occur. Therefore, in this embodiment of the invention, when determining the image's tag, it is necessary to tag the image multiple times and comprehensively consider the tags from multiple tags to determine the image's tag, thereby improving the accuracy of image labeling. In this embodiment of the invention, tagging can be understood as labeling the image.
[0023] Therefore, if an image included in the images to be classified has never been labeled, then that image corresponds to zero labels; if an image has been labeled, then that image corresponds to one or more labels. The multiple labels can all be different, and some labels can be the same; this embodiment of the invention does not impose any restrictions on this.
[0024] To facilitate the differentiation of tag types, in this embodiment of the invention, the tags corresponding to classified images are called classification tags, the tags already assigned to images to be classified are called initial tags, and the tags in the tag library are called standard tags.
[0025] In this embodiment of the invention, there is no limit to the number of images included in the classified image and the image to be classified; there can be one or more. Each image included in the classified image corresponds to one label.
[0026] In some embodiments, if the number of images included in the image to be classified is at least two, the labels included in the obtained initial labels may be the same or different; when the initial labels include the same labels, deduplication is not required.
[0027] In this embodiment of the invention, deduplication means removing duplicate labels. When the image to be classified includes only one image, for example, if the image to be classified includes only one image of an apple, and this image has been labeled with multiple "apple" tags, even if multiple tags are deduplicated and only one "apple" tag is retained, the user can still select the "apple" tag when labeling the image, thus allowing the normal labeling process. In other words, for the case where the image to be classified includes only one image, whether or not the initial labels are deduplicated will not affect the labeling result. When the image to be classified includes at least two images, for example, if the image to be classified includes two images of apples, and both of these images have been labeled with multiple "apple" tags, if multiple tags are deduplicated and only one "apple" tag is retained, it is easy for the user to only select the "apple" tag when labeling one of the two images, and the other image cannot be labeled with the "apple" tag, thus preventing the normal labeling process and affecting the labeling result.
[0028] In this embodiment of the invention, when the number of images to be classified is at least two, and the initial labels include the same label, deduplication is not required. This allows users to label multiple images with the same label when they all share the same label, improving the accuracy of the labeling results.
[0029] S102. Generate an image verification code based on the classified image, the image to be classified, the classification label, the initial label, and the standard label.
[0030] A tag list is generated based on the classification tags, initial tags, and standard tags. When generating the image CAPTCHA, for example, the tag list can be arranged around, above, or below the areas containing the classified and unclassified images. This embodiment of the invention does not limit the arrangement of the tag list, as long as it clearly and completely presents the tag list around the classified and unclassified images. The areas containing the images within the classified and unclassified images do not overlap; that is, each image is clearly defined and does not overlap.
[0031] Taking a 'a' image set as an example and b image sets as an example, the following example illustrates the label list. For instance, retrieve the 'a' labels of the selected 'a' images, where each 'a' label belongs to a different category. If the number of labels already assigned to the i-th image in the image set to be classified is Xi, where i is a positive integer less than or equal to b, then retrieve all the assigned labels for each image in the image set to be classified, without deduplication, resulting in a total of X1 + X2 + ... + Xb = Y labels. These Y labels may contain duplicates. For instance, one possibility is that the Xi labels of the i-th image contain duplicates, and another possibility is that the Xi labels of the i-th image and the Xj labels of the j-th image contain duplicates, where j is a positive integer less than or equal to b. Select Z labels from the label library that are different from the selected a+Y labels, and combine these a+Y+Z labels into a label list. As can be seen from the above, the number of tags included in the tag list is greater than or equal to the number of images included in the classified images and the images to be classified, which can ensure that each image can be tagged.
[0032] An image CAPTCHA is generated based on a list of tags consisting of a+Y+Z tags, a images included in the already classified images, and b images included in the images to be classified. An example image CAPTCHA is shown using tags to represent labels, such as... Figure 2 As shown, Figure 2 This is an exemplary schematic diagram of an image verification code provided in an embodiment of the present invention. Figure 2 The example shown is of arranging the tag list on top of the image, and does not represent that the embodiments of the present invention are limited to this.
[0033] S103. Receive the user's labeling result for the image verification code and verify the labeling result.
[0034] When users label images in an image verification code, they can drag the label to the area where the image is located to complete the labeling process for one image. To ensure the uniqueness of the labeling result, an image can only be labeled with one tag. During the labeling process, users can delete, modify, and perform other operations on the labeling results multiple times. When the user has completed labeling all images, they can submit the labeling results by clicking the submit option, or the image classification device can automatically detect the labeling status and automatically submit the labeling results when it detects that all images have been labeled. This embodiment of the invention does not limit the form in which the labeling results are obtained.
[0035] The labeling results include the user's labeling results for classified images and the user's labeling results for images to be classified. Since the labels of the images included in the classified images are known, the validity of the labeling results can be determined by verifying the labeling results of the classified images.
[0036] S104. If the verification is successful, the classification label of the image to be classified is determined based on the labeling label and the initial label of the image to be classified in the labeling results.
[0037] If the verification passes, it indicates that the user has taken the labeling process seriously, the labeling results are highly reliable, and the results are valid and referable. After successful verification, the initial label represents the labels that have already been applied to the image to be classified. Combined with the labels from the current labeling results, for a single image to be classified, multiple labels are considered comprehensively to determine the image's label, which is the classification label, thus improving the accuracy of image classification.
[0038] If the verification fails, it means that the user may have treated the labeling casually, the labeling result has low credibility, and the labeling result is invalid.
[0039] According to the scheme provided by the embodiments of the present invention, a classified image, a classification label of the classified image, an image to be classified, an initial label of the image to be classified, and a standard label are obtained. The standard label is different from the classification label and the initial label. The initial label is a label that has already been applied to the image to be classified. An image verification code is generated based on the classified image, the image to be classified, the classification label, the initial label, and the standard label. The classified image and classification label in the image verification code can verify the labeling result, thereby improving the effectiveness and accuracy of the labeling result. The labeling result of the image verification code is received from the user and verified. If the verification is successful, the classification label of the image to be classified is determined based on the label of the image to be classified and the initial label in the labeling result. That is, the image to be classified will be labeled multiple times. Compared with the method of determining the classification label after only one labeling of the image to be classified, the embodiments of the present invention combine the labeling label and the initial label to determine the classification label, which improves the accuracy of image classification.
[0040] In some embodiments, the verification of the labeling results in S103 above can also be achieved in the following ways: when the number of images included in the classified image is less than or equal to a preset number, if the labeling labels and classification labels of the classified images in the labeling results correspond, then the verification is successful; when the number of images included in the classified image is greater than the preset number, the number of correct labels is determined according to the labeling labels and classification labels of the classified images in the labeling results; the ratio of the number of correct labels to the number of images included in the classified image is calculated; if the ratio is greater than a preset ratio, then the verification is successful.
[0041] When the number of images included in the classified images is less than or equal to the preset number, it indicates that the number of images is insufficient. In this case, all labels on the classified images must match the classification labels for verification to pass. That is, the condition for successful verification is that all labels applied by the user to the images included in the classified images must be correct. Taking PT as an example and Tag as a label, for instance, if the classified images include PT1 and PT2, where PT1 corresponds to Tag1 and PT2 corresponds to Tag2, verification only passes if the label for PT1 is Tag1 and the label for PT2 is Tag2.
[0042] When the number of images included in the classified images exceeds a preset number, it indicates a large number of images. It's necessary to determine the number of correctly labeled images and calculate the ratio of the number of correctly labeled images to the number of images included in the classified images. The verification pass rate is determined based on this ratio. In other words, the user doesn't need to correctly label all the images included in the classified images; only the percentage of correctly labeled images needs to be greater than the preset ratio. Taking PT as an example and Tag as a label, for instance, if the classified images include PT1-PT10, and PT1-PT10 correspond to Tag1-Tag10 respectively, and the label for PTi is Tagi (where i is an integer from 1 to 10), this is considered a correct label, and the number of correct labels is counted. Taking a correct label count of 7 and a preset ratio of 60% as an example, the ratio of the number of correctly labeled images to the number of images included in the classified images is 70%, which is greater than the preset ratio, indicating that the verification passes. Taking a correct labeling count of 4 and a preset ratio of 60% as an example, the ratio of the correct labeling count to the number of images included in the classified images is 40%, which is less than the preset ratio. This indicates that the verification fails and the labeling result is invalid.
[0043] The preset quantity and preset ratio can be appropriately set by those skilled in the art according to actual needs, as long as the marking results can be verified. For example, they can be determined through analysis from a large number of numerical selections.
[0044] In some embodiments, if the number of images included in the classified images is at least two, then images belonging to different categories are obtained, so that the labels included in the classification labels corresponding to the obtained classified images belong to different categories.
[0045] The categorized images and category labels are used to verify the labeling results of image CAPTCHAs, provided that the categorized images include at least two images. If the selected images belong to the same or similar categories, it is difficult for users to distinguish them, which may lead to labeling errors and invalidate the image CAPTCHA labeling results, reducing work efficiency. For example, if the categorized images include two images of road bikes and mountain bikes, and the category labels include two tags for road bikes and mountain bikes, it is difficult for users to distinguish them. When labeling errors occur, it is impossible to tell whether the user labeled randomly or carefully, meaning it is impossible to determine whether the labeling result is valid.
[0046] Choosing images belonging to different categories—that is, the less similar the categories of the images included in the categorized images—makes it easier for users to complete correct labeling. This helps distinguish between random and careful labeling, thus determining the validity of the labeling result. For example, if the categorized images include a mountain bike and an apple, and the category labels include the same two labels, it's very easy for users to identify and differentiate them. If a labeling error occurs, the labeling result can be determined to be invalid.
[0047] In this embodiment of the invention, if the number of images included in the classified images is at least two, then images belonging to different categories are obtained, so that the tags included in the classification labels corresponding to the obtained classified images belong to different categories, making it easier for users to identify and distinguish them, thereby improving work efficiency.
[0048] In some embodiments, S104 above can also be implemented in the following way: Based on the labeling labels and initial labels of the image to be classified in the labeling results, determine the total number of times the image to be classified is labeled, and the number of each type of label included in the labeling labels and initial labels; wherein, the total number of times of labeling is equal to the sum of the number of labels included in the labeling labels and initial labels; calculate the ratio of the number of each type of label to the total number of times of labeling to obtain the label probability of each type of label; if the total number of times of labeling is greater than a preset number of times, and there is a label with a probability greater than the preset probability, then the label corresponding to the maximum label probability in the label probability is taken as the classification label of the image to be classified.
[0049] The initial labels represent the labels already assigned to the image to be classified. For an image to be classified, if it has been labeled *e* times, the initial labels include the same number of labels as the number of times it has been labeled, i.e., *e* labels. After this labeling is completed, the total number of labels, including this labeling, is *e+1* times, and the total number of labels is *e+1*. There may be duplicate labels among the *e+1* labels. By classifying the labels, the number of labels in each category is obtained.
[0050] Taking an image to be classified as an example, after this labeling process, a total of K labels {Tag1,Tag2,...,TagK} are applied. The number of times the i-th label is applied is M_tagi (i∈{1,k}), and the total number of labels is N = M_tag1 + M_tag2 + ... + M_tagk, where M_tagk represents the number of the k-th label. The ratio of each label type to the total number of labels is calculated to obtain the label probability of each label type, i.e., the probability of the label taggi is Pi, where Pi = M_tagi / N (i∈{1,k}). If the total number of labels N is greater than the preset number of labels S, and there is a label among the label probabilities P1-Pk that is greater than the preset probability Q, then the label corresponding to the maximum label probability is taken as the classification label of the image to be classified. For example, if the total number of labels N for an image to be classified is greater than the preset number of labels S, and MAX{P1,P2,...,Pk} = Pi > Q, then the label of the image is determined to be Tagi. For example, the preset number of attempts S can be set to 4, and the preset probability Q can be set to 95%. This can also be understood as the minimum probability Q that an image to be classified can be identified as a certain label.
[0051] The preset probability and preset number of attempts can be appropriately set by those skilled in the art according to actual needs, as long as they are sufficient to determine the classification label of the image. For example, this can be determined through analysis from a large number of numerical selections.
[0052] In this embodiment of the invention, the total number of times the image to be classified is determined based on the labeled labels and initial labels of the image to be classified in the labeling results, as well as the number of each type of label included in the labeled labels and initial labels. The ratio of the number of each type of label to the total number of labels is calculated to obtain the label probability of each type of label. If the total number of labels is greater than a preset number, it indicates that the image has been labeled a sufficient number of times. If there are labels with a higher probability than the preset probability, it means that most of the labels in the label probability are of the same type, with only a few labels being noise. The label corresponding to the maximum label probability can represent the label of the image. The label corresponding to the maximum label probability is used as the classification label of the image to be classified, thereby improving the accuracy of the classification label.
[0053] In some embodiments, prior to S103, the image classification method provided by the present invention further includes the following steps: displaying an image verification code in a display interface, wherein in the image verification code, the image to be classified and the images included in the classified image are located in different regions, and each region does not overlap, and the classification label, initial label, and standard label are distributed outside the region where the image is located; receiving a drag operation from the user in the display interface, wherein the drag operation includes dragging the label included in the classification label, initial label, or standard label to the region where the image included in the classified image or the image to be classified is located, and only one label can be dragged into each region; determining the user's labeling result for the image verification code based on the user's drag operation.
[0054] In this embodiment of the invention, after generating the image verification code, the image verification code is also displayed on the display interface. The image to be classified and the images included in the classification image are located in different regions, and each region does not overlap. This avoids situations where users drag labels into overlapping areas, leading to inaccurate identification of the image corresponding to the label. The classification label, initial label, and standard label are distributed outside the area where the image is located, improving the effectiveness of the user's dragging operation.
[0055] Users perform drag-and-drop operations on the display interface. These operations can include dragging any tag from the category label, initial label, or standard label to an area containing an image within a categorized or uncategorized image. Only one tag can be dragged into each area; that is, one tag is assigned to each image. This embodiment of the invention determines the user's tagging result for the image verification code based on multiple drag-and-drop operations received from the user on the display interface.
[0056] In this embodiment of the invention, the labeling and classification work is performed by all users participating in the Internet. Compared with the automatic labeling method that involves machine clustering and then classification, the labeling work performed manually by Internet users improves the accuracy of the labeling results.
[0057] In some embodiments, the image verification code also includes other tags, and the image classification method further includes: if the tagging tag of the image to be classified in the tagging result is another tag, then the tagging result is invalid.
[0058] In this embodiment of the invention, the tag list also includes an "Other" tag. When a user tags an image CAPTCHA, there may be instances where the tag list does not contain a tag corresponding to the image to be categorized, preventing the user from tagging the image. To facilitate the tagging process for users, this embodiment of the invention also includes an "Other" tag in the image CAPTCHA. When the tag for the image to be categorized is an "Other" tag, it indicates that the tagging result is invalid, meaning that the tagging information is not recorded, and the next image CAPTCHA tagging process is initiated.
[0059] This invention provides an "Other" label, which allows users to easily select an image to be categorized when there is no corresponding label in the label list, thereby improving work efficiency.
[0060] In some embodiments, the image classification method further includes the following steps: classifying the image to be classified into a classified image library according to the classification labels, wherein the classified image library includes classified images and classification labels of the classified images.
[0061] In this embodiment of the invention, after labeling the images to be classified and determining the classification tags, the images to be classified are further categorized into a pre-classified image library based on the classification tags, thereby enriching the data in the pre-classified image library. The data in the pre-classified image library can be used as sample data for the next image CAPTCHA labeling process. For example, pre-classified images and their classification tags are selected from the pre-classified image library.
[0062] In some embodiments, after obtaining the classified image library, the image classification method further includes: obtaining a target label; and selecting an image corresponding to the target label from the classified image library based on the target label.
[0063] Since the classified image library stores classified images and their classification labels, embodiments of the present invention can also receive target labels, thereby selecting images corresponding to the target labels from the classified image library. That is, embodiments of the present invention can output all classified images for a specific label as needed. For example, cluster analysis can be performed on images under the same label, or images under the same label can be used as sample data and input into a machine learning model for recognition processing, etc.
[0064] The following will describe an exemplary application of the embodiments of the present invention in a practical application scenario.
[0065] For ease of description, in this embodiment of the invention, PT represents a classified image, PUT represents an image to be classified, and Tag represents a label. The total number of times each image in the image to be classified has been tagged is N. If the image to be classified has been tagged with K types of labels {Tag1,Tag2,...,TagK}, the number of times the i-th label has been tagged is M_tagi (i∈{1,k}), the total number of tags is N = M_tag1 + M_tag2 + ... + M_tagk, and the probability of the tag is Pi, where Pi = M_tagi / N (i∈{1,k}).
[0066] In this embodiment of the invention, conditions are also set for an image to be tagged and classified, including: the minimum number of times an image to be tagged is S, and the minimum probability that an image to be tagged can be determined to a certain label is Q, for example, Q can be set to 95%. For example, if the total number of tags N for the image to be tagged is greater than S, and MAX{P1,P2,...,Pk}=Pi>Q, then the label of the image to be tagged is determined to be Tagi.
[0067] The image classification method provided in the embodiments of the present invention will be described below by combining various modules, such as Figure 3 As shown, Figure 3 A flowchart illustrating optional steps of another image classification method provided in an embodiment of the present invention. Figure 3 The document illustrates an image input / output module, a label list generation module, an image CAPTCHA generation module, a verification module, and a storage and computation module. It also provides illustrative descriptions of each module, combined with the steps of an image classification method. This embodiment of the invention uses the image CAPTCHA labeling process initiated by a user's internet operation as an example for explanation.
[0068] I. Image Input / Output Module.
[0069] The image input / output module can be used for both image input and output. The image input section, based on the storage and computing module, retrieves a classified image library, a library of images to be classified, and a label library from the storage and computing module. The classified image library can include an initially labeled and classified image library, as well as an image library automatically classified subsequently. The initially labeled and classified image library can be obtained through manual labeling or from other classified image libraries. The library of images to be classified can be obtained through web crawling or by methods provided by users who require labeling and classification.
[0070] For example, a tag library can be obtained in the following ways.
[0071] 1. By analyzing the data of classified images, organize the labels of all classified images, and form a label library after deduplication.
[0072] 2. After selecting a major category, a web crawler collects text data from the web, and then performs lexical and syntactic analysis on the text data to obtain standard labels.
[0073] 3. Supplement and expand the standard tags in the tag library by manually inputting tags.
[0074] The image output section is used to: output all classified images for a specific label as needed; or, based on the user-input image to be classified, label and classify all images of that image before outputting them.
[0075] II. Tag List Generation Module.
[0076] The tag list generation module may include the following steps when generating a tag list:
[0077] 1. Obtain the a labels of the a selected classified images PT.
[0078] 2. There are b image PUTs for the image to be classified. If the number of labels in a PUT is Xi, then obtain all the labels of each image PUT of the image to be classified without deduplication. There are a total of X1+X2+...+Xb=Y labels.
[0079] 3. Select Z tags from the tag library that are different from the already selected a+Y tags.
[0080] 4. Combine the a+Y+Z tags and the single "Other" tag to form a tag list.
[0081] It should be noted that, Figure 3 In the image CAPTCHA, n = Y + Z, and the tag list is {Tag1, Tag2...Taga, Taga+1, Taga+2...Taga+n, others}.
[0082] III. Image verification code generation module and verification module.
[0083] The image verification code generation module, combined with the tag list generated by the tag list generation module, and a (a≥1) images of different categories PT selected from the classified image library, and b (b≥1) images to be classified PUT selected from the unclassified image library, generate an image verification code. The image regions are clearly divided and cannot overlap. The tag list is arranged around the images PT and PUT.
[0084] The user manually drags labels to the image area, thus obtaining the user's labeling result for the image CAPTCHA. The verification module verifies the labeling result. If the user correctly drags the labels of the 'a' images (PT) that have been classified onto the corresponding areas of the 'a' images (PT), the verification passes. If the verification passes, the 'b' images (PUT) to be classified are associated with the labels dragged by the user to their corresponding areas, and the user's labeling result for the images to be classified is recorded.
[0085] For example, taking a=2 and b=1 as an example, the image classification process is described. The classified images PT include {PT1, PT2}, and the images to be classified PUT include {PUT1}. The label list is {Tag1, Tag2, Tag3, Tag4, Other}, where PT1 corresponds to Tag1, PT2 corresponds to Tag2, and PUT1 has been labeled with Tag3 and Tag4. An image verification code is generated based on {PT1, PT2}, {PUT1}, and the label list.
[0086] 1. If the user does not drag Tag1 to the PT1 area, or does not drag Tag2 to the PT2 area, the user's image verification code verification is considered to have failed, and the image verification code tagging process is restarted.
[0087] 2. If the user drags Tag1 to the PT1 area and Tag2 to the PT2 area, the user verification is considered successful. At this point:
[0088] 2.1 If Tag3 is dragged to the PUT1 area, it is considered that the user has tagged PUT1 with Tag3 once. At this time, the total number of tags is N+1, and the number of times tag3 has been tagged is M_tag3+1, where M_tag3 represents the number of times tag3 has been tagged before this tagging. The tagging result is saved to the storage and calculation module.
[0089] If the user drags Tag4 to the PUT1 area, it is considered that the user has tagged Tag4 once in PUT1. At this time, the total number of tags is N+1, and the number of times tag4 has been tagged is M_tag4+1, where M_tag4 represents the number of times tag4 has been tagged before this tagging. The tagging result is saved to the storage and calculation module.
[0090] 2.2 If the user drags the "Other" label to the PUT1 area, that is, the image to be classified is labeled "Other", then the labeling result is considered invalid and PUT1 is not counted. In other words, the storage and calculation module does not record the labeling information.
[0091] V. Storage and Computing Module.
[0092] The storage and calculation module stores the classified image library, the image library to be classified, the label library, and the data information of the labeled images (PUTs) in the image library to be classified. For example, if a PUT is labeled with three tags (Tag1, Tag2, and Tag3), the storage and calculation module will store the following data information for that PUT: PUT: S, Q, N, M_Tag1, M_Tag2, and M_Tag3. Here, S represents the minimum number of times the image PUT needs to be labeled to be considered classifiable, Q represents the minimum probability that the image PUT can be identified with a certain label, and N represents the total number of times the image PUT has been labeled.
[0093] When a tagging process for an image CAPTCHA begins, if the selected image PUT needs to be tagged, the storage and calculation module reads all the tags already tagged under the PUT through the image input / output module. This, combined with the tags corresponding to the categorized images and other tags, generates a tag list. An image CAPTCHA is then generated based on the image to be categorized, the categorized images, and the tag list. After successful user verification and completion of a tagging operation, if the total number of tags N > S, and the MAX{P1,P2,...,Pk} = Pi > Q for a certain image PUT to be categorized, then the PUT is marked as a tag and classified into the categorized image library for subsequent image CAPTCHA generation. In other words, the storage and calculation module stores the tagging information and determines whether the tagging criteria are met. If not, it waits for the next tagging operation; if so, it classifies the image to be categorized into the categorized image library.
[0094] Simultaneously, the storage and computing module can also employ the logic of the image input / output module to output the stored images according to actual needs. For example, it can output the classified image corresponding to the target label based on the target label; or it can output the image to be classified after labeling and classification.
[0095] This invention provides an image classification device based on image CAPTCHAs. The image classification device includes an image input / output module, an image CAPTCHA generation module, a verification module, a label list generation module, and a storage and calculation module. The image input / output module generates image data for the image CAPTCHAs. The image CAPTCHA generation module and the verification module are the main components of the user's interaction with the CAPTCHA labeling results. The label list generation module is closely related to the image CAPTCHA labeling process and is one of the interactive components of the user's CAPTCHA labeling results. The storage and calculation module stores all image-related data information and provides operational logic for the entire image CAPTCHA labeling logic.
[0096] The image classification method provided in this invention uses an image CAPTCHA approach, where all internet users participate in the labeling and classification process. Compared to related technologies that use machine clustering followed by automatic classification, the manual labeling process by internet users improves the accuracy of image classification. Furthermore, unlike methods that use a pre-labeled image library for image recognition and then labeling, which cannot label unclassified images, this invention can effectively label unclassified images.
[0097] To implement the image classification method of this invention, this invention also provides an image classification device, such as... Figure 4 As shown, Figure 4 This is a schematic diagram of an image classification device according to an embodiment of the present invention. The device 40 includes: a storage and calculation module 401 for storing a classified image library, a to-be-classified image library, and a tag library; an image input / output module 402 for obtaining classified images and classification tags of the classified images from the classified image library, obtaining images to be classified and initial tags of the images to be classified from the to-be-classified image library, and obtaining standard tags from the tag library, wherein the standard tags are different from the classification tags and the initial tags; a tag list generation module 403 for generating a tag list based on the classification tags, the initial tags, and the standard tags; an image verification code generation module 404 for generating an image verification code based on the classified images, the to-be-classified images, and the tag list; a verification module 405 for receiving the user's tagging result of the image verification code and verifying the tagging result; and the storage and calculation module 401 for determining the classification tag of the to-be-classified image based on the tagging tag of the to-be-classified image and the initial tag in the tagging result if the verification is successful.
[0098] In some embodiments, the storage calculation module 401 is further configured to determine the total number of times the image to be classified is labeled, and the number of each type of label included in the labeling labels and the initial labels, based on the labeling labels of the image to be classified in the labeling results; wherein, the total number of times of labeling is equal to the sum of the number of labels included in the labeling labels and the initial labels; calculate the ratio of the number of each type of label to the total number of times of labeling to obtain the label probability of each type of label; if the total number of times of labeling is greater than a preset number of times, and there is a label with a probability greater than a preset probability in the label probability, then the label corresponding to the maximum label probability in the label probability is taken as the classification label of the image to be classified.
[0099] In some embodiments, the verification module 405 is further configured to: when the number of images included in the classified image is less than or equal to a preset number, if the labeling labels of the classified images in the labeling result correspond to the classification labels, then the verification is successful; when the number of images included in the classified image is greater than the preset number, determine the number of correct labels based on the labeling labels of the classified images in the labeling result and the classification labels; calculate the ratio of the number of correct labels to the number of images included in the classified image; if the ratio is greater than a preset ratio, then the verification is successful.
[0100] In some embodiments, the image input / output module 402 is further configured to acquire images belonging to different categories if the number of images included in the classified image is at least two, so that the tags included in the classification labels corresponding to the acquired classified images belong to different categories.
[0101] In some embodiments, the storage computing module 401 is further configured to display the image verification code on a display interface, wherein in the image verification code, the image to be classified and the images included in the classified image are located in different regions, and each region does not overlap, and the classification label, the initial label, and the standard label are distributed outside the regions where the images are located; receive the user's drag operation on the display interface, wherein the drag operation includes dragging the classification label, the initial label, or the label included in the standard label to the region where the classified image or the image to be classified is located, and only one label can be dragged into each region; and determine the user's labeling result for the image verification code based on the user's drag operation.
[0102] In some embodiments, the image verification code may also include other tags, and the storage and calculation module 401 is further configured to invalidate the tagging result if the tagging label of the image to be classified in the tagging result is one of the other tags.
[0103] In some embodiments, the storage computing module 401 is further configured to classify the image to be classified into a classified image library according to the classification label, the classified image library including the classified images and the classification labels of the classified images.
[0104] In some embodiments, the image input / output module 402 is further configured to acquire a target label; and select an image corresponding to the target label from the classified image library according to the target label.
[0105] It should be noted that the image classification device provided in the above embodiments is only illustrated by the division of the above-described program modules when performing image classification. In practical applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program modules to complete all or part of the processing described above. Furthermore, the image classification device and image classification method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process and beneficial effects are detailed in the method embodiments, and will not be repeated here. For technical details not disclosed in the device embodiments, please refer to the description of the method embodiments of the present invention for understanding.
[0106] In an embodiment of the present invention, Figure 5 This is a schematic diagram of the composition structure of the image classification device proposed in an embodiment of the present invention, as shown below. Figure 5 As shown, the device 50 proposed in the embodiments of the present invention may further include a processor 501, a memory 502 storing instructions executable by the processor 501, and in some embodiments, the image classification device 50 may further include a communication interface 503 and a bus 504 for connecting the processor 501, the memory 502 and the communication interface 503.
[0107] In this embodiment of the invention, the processor 501 can be at least one of the following: Application-Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field-Programmable Gate Array (FPGA), Central Processing Unit (CPU), Controller, Microcontroller, and Microprocessor. It is understood that for different devices, the electronic device used to implement the above-mentioned processor function can also be other types, and this embodiment of the invention does not specifically limit the specific types. The image classification device 50 may also include a memory 502, which can be connected to the processor 501. The memory 502 is used to store executable program code, which includes computer operation instructions. The memory 502 may include high-speed RAM memory and may also include non-volatile memory, such as at least two disk drives.
[0108] In this embodiment of the invention, bus 504 is used to connect communication interface 503, processor 501 and memory 502 and the mutual communication between these devices.
[0109] In this embodiment of the invention, memory 502 is used to store instructions and data.
[0110] In this embodiment of the invention, the processor 501 is configured to acquire a classified image, a classification label of the classified image, an image to be classified, an initial label of the image to be classified, and a standard label, wherein the standard label is different from both the classification label and the initial label; generate an image verification code based on the classified image, the image to be classified, the classification label, the initial label, and the standard label; receive the user's tagging result for the image verification code and verify the tagging result; if the verification is successful, determine the classification label of the image to be classified based on the tagging label of the image to be classified and the initial label in the tagging result.
[0111] In practical applications, the aforementioned memory 502 can be volatile memory, such as random-access memory (RAM); or non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); or a combination of the above types of memory, and provide instructions and data to the processor 501.
[0112] Furthermore, in this embodiment, the functional modules can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional module.
[0113] If the integrated unit is implemented as a software functional module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the method of this embodiment. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0114] This invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the image classification method as described in any of the above embodiments.
[0115] For example, the program instructions corresponding to an image classification method in this embodiment can be stored on a storage medium such as an optical disc, hard disk, or USB flash drive. When the program instructions corresponding to an image classification method in the storage medium are read or executed by an electronic device, the image classification method as described in any of the above embodiments can be implemented.
[0116] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.
[0117] This invention is described with reference to schematic and / or block diagrams illustrating the implementation of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the schematic and / or block diagrams can be implemented by computer program instructions, as well as combinations of blocks in the schematic and / or block diagrams. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the schematic and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0118] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in the implementation flow diagram. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0119] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0120] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention.
Claims
1. An image classification method, characterized by, The method includes: The process involves obtaining a classified image, its classification label, an image to be classified, an initial label for the image to be classified, and a standard label, wherein the standard label is different from both the classification label and the initial label; when the image to be classified includes at least two images and the initial labels include the same label, no deduplication is performed on the initial label. A tag list is generated based on the classification tags, the initial tags, and the standard tags; the number of tags included in the tag list is greater than or equal to the number of images included in the classified images and the images to be classified; an image verification code is generated based on the classified images, the images to be classified, and the tag list; Receive the user's marking result for the image verification code, and verify the marking result; If the verification passes, the total number of times the image to be classified is determined based on the labeling labels of the image to be classified and the initial labels in the labeling results, as well as the number of each type of label included in the labeling labels and the initial labels; wherein, the total number of times of labeling is equal to the sum of the number of labels included in the labeling labels and the initial labels; the ratio of the number of each type of label to the total number of times of labeling is calculated to obtain the label probability of each type of label; if the total number of times of labeling is greater than a preset number of times, and there is a label with a probability greater than a preset probability in the label probability, then the label corresponding to the maximum label probability in the label probability is taken as the classification label of the image to be classified.
2. The method of claim 1, wherein, The verification of the marking result includes: When the number of images included in the classified images is less than or equal to a preset number, if the labeling labels of the classified images in the labeling results correspond to the classification labels, then the verification is successful. When the number of images included in the classified images is greater than the preset number, the correct number of labels is determined based on the labeling labels of the classified images and the classification labels in the labeling results; Calculate the ratio of the number of correctly labeled images to the number of images included in the classified images; If the ratio is greater than the preset ratio, the verification is successful.
3. The method according to claim 1 or 2, characterized in that, The acquisition of classified images includes: If the number of images included in the classified images is at least two, then images belonging to different categories are obtained, so that the tags included in the classification labels corresponding to the obtained classified images belong to different categories.
4. The method according to claim 1 or 2, characterized in that, Before receiving the user's labeling result for the image verification code, the method further includes: The image verification code is displayed in the display interface, wherein the image to be classified and the images included in the classified image are located in different regions and each region does not overlap, and the classification label, the initial label and the standard label are distributed outside the region where the image is located; The system receives a drag operation from the user on the display interface, wherein the drag operation includes dragging a label included in the category label, the initial label, or the standard label to the area where the image included in the categorized image or the image to be categorized is located, and only one label can be dragged into each area; Based on the user's dragging operation, the user's labeling result for the image verification code is determined.
5. The method according to claim 1 or 2, characterized in that, The image verification code also includes other tags, and the method further includes: If the label of the image to be classified in the labeling result is another label, then the labeling result is invalid.
6. The method of claim 1 or 2, wherein, The method further includes: Based on the classification labels, the image to be classified is categorized into a classified image library, which includes the classified images and their classification labels.
7. The method according to claim 6, characterized in that, After obtaining the classified image library, the method further includes: Obtain the target tag; Based on the target label, select an image corresponding to the target label from the classified image library.
8. An image classification apparatus characterized by comprising: The device includes: The storage and computing module is used to store the classified image library, the image library to be classified, and the label library; The image input / output module is used to obtain classified images and their classification labels from the classified image library, obtain images to be classified and their initial labels from the unclassified image library, and obtain standard labels from the label library, wherein the standard labels are different from the classification labels and the initial labels; when the unclassified image includes at least two images and the initial labels include the same label, no deduplication processing is performed on the initial labels; The tag list generation module is used to generate a tag list based on the classification tags, the initial tags, and the standard tags; the number of tags included in the tag list is greater than or equal to the number of images included in the classified images and the images to be classified; An image verification code generation module is used to generate an image verification code based on the classified image, the image to be classified, and the tag list; The verification module is used to receive the user's marking result for the image verification code and to verify the marking result; The storage and calculation module is further configured to, if the verification is successful, determine the total number of times the image to be classified is tagged, and the number of each type of label included in the tagging labels and the initial labels in the tagging results; wherein, the total number of times tagging is equal to the sum of the number of labels included in the tagging labels and the initial labels; calculate the ratio of the number of each type of label to the total number of times tagging is performed to obtain the label probability of each type of label; if the total number of times tagging is greater than a preset number of times, and there is a label with a probability greater than a preset probability in the label probability, then the label corresponding to the maximum label probability in the label probability is taken as the classification label of the image to be classified.
9. An image classification device characterized by comprising: The device includes a memory and a processor, the memory storing a computer program that can run on the processor, the processor executing the program to implement the method of any one of claims 1-7.
10. A computer-readable storage medium, characterized in that, It stores executable instructions that, when executed by a processor, implement the method described in any one of claims 1-7.
Citation Information
Patent Citations
Method, apparatus and computer-readable medium for labeling image samples
CN109460652A
Method and device for labeling data
CN111461154A
