Classification of test methods, apparatus, equipment, media and procedures products

Abstract

This application discloses a classification testing method, apparatus, device, medium, and program product, belonging to the field of artificial intelligence technology. The method includes: obtaining a leaf label set from a label tree, where each leaf label set includes at least two leaf labels. A leaf label refers to the label corresponding to a leaf node in the label tree, and the label tree is a tree structure constructed based on the hierarchical relationship of label categories; traversing the at least two leaf labels in the leaf label set to generate M-dimensional label combinations, each M-dimensional label combination consisting of M leaf labels, where M is a positive integer greater than 1; and performing classification testing on a multi-label classifier based on the test set corresponding to each M-dimensional label combination. This method can satisfy the testing sufficiency requirement for multi-label classifiers.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI) technology, and in particular to a classification test method, apparatus, device, medium, and program product. Background Technology

[0002] With the advent of the mobile internet era, the amount of image data has exploded exponentially. To facilitate image classification, applications for image classification have been developed.

[0003] Generally, an image contains objects of more than one class, meaning image classification tasks are often multi-label image classification tasks. As applications for multi-label image classification become increasingly widespread in daily life, the accuracy of these applications is becoming increasingly important. For example, in the field of autonomous driving, the classification results of applications on objects in images directly affect the safety of autonomous driving. Therefore, it is necessary to conduct comprehensive system testing on these applications. Summary of the Invention

[0004] This application provides a classification test method, apparatus, device, medium, and program product. The technical solution is as follows:

[0005] According to one aspect of this application, a classification test method is provided, the method comprising:

[0006] Obtain a set of leaf tags from the tag tree, wherein the set of leaf tags includes at least two leaf tags, and the leaf tags refer to the tags corresponding to the leaf nodes on the tag tree, wherein the tag tree is a tree structure constructed based on the hierarchical relationship of tag categories;

[0007] Iterate through at least two leaf labels in the leaf label set to generate M-dimensional label combinations. Each M-dimensional label combination consists of M leaf labels, where M is a positive integer greater than 1.

[0008] Based on the test set corresponding to each M-dimensional label combination, the multi-label classifier is tested for classification.

[0009] According to another aspect of this application, a classification testing apparatus is provided, the apparatus comprising:

[0010] The acquisition module is used to acquire a set of leaf tags from the tag tree. The set of leaf tags includes at least two leaf tags. The leaf tags refer to the tags corresponding to the leaf nodes on the tag tree. The tag tree is a tree structure constructed based on the hierarchical relationship of tag categories.

[0011] The generation module is used to traverse at least two leaf labels in the leaf label set and generate M-dimensional label combinations. Each M-dimensional label combination consists of M leaf labels, where M is a positive integer greater than 1.

[0012] The testing module is used to perform classification tests on the multi-label classifier based on the test set corresponding to each M-dimensional label combination.

[0013] According to another aspect of this application, a computer device is provided, the computer device including a processor and a memory connected to the processor, the memory storing program instructions, wherein the processor executes the program instructions to implement the classification test method provided in various aspects of this application.

[0014] According to another aspect of this application, a computer-readable storage medium is provided, wherein program instructions are stored therein, which, when executed by a processor, implement the classification test method as provided in various aspects of this application.

[0015] According to another aspect of this application, a computer program product (or computer program) is provided, the computer program product (or computer program) including computer instructions stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to perform the method provided in various optional implementations of the above-described classification test method.

[0016] According to another aspect of this application, a chip is provided, the chip including programmable logic circuitry and / or program instructions, which, when the chip is running, are used to implement the classification test methods provided in various aspects of this application.

[0017] The beneficial effects of the technical solutions provided in this application embodiment may include:

[0018] In the above classification testing method, when testing the multi-label classifier, at least two leaf labels are obtained from a pre-set label tree. Then, at least two leaf labels are traversed to generate M-dimensional label combinations. Based on the test set corresponding to each M-dimensional label combination, the multi-label classifier is classified to test whether it can correctly classify test samples with M labels. This ensures that the test of the multi-label classifier can completely cover the M-dimensional leaf label combinations in the label tree, thus fully testing the multi-label classifier in the classification of M-dimensional labels. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This illustration shows a schematic diagram of the tag tree structure provided in an exemplary embodiment of this application;

[0021] Figure 2 A schematic diagram of a communication system provided in an exemplary embodiment of this application is shown;

[0022] Figure 3 A flowchart of a classification testing method provided in an exemplary embodiment of this application is shown;

[0023] Figure 4 A flowchart of a classification testing method provided in another exemplary embodiment of this application is shown;

[0024] Figure 5 A flowchart of a classification testing method provided in another exemplary embodiment of this application is shown;

[0025] Figure 6 A flowchart of a classification testing method provided in another exemplary embodiment of this application is shown;

[0026] Figure 7 A flowchart of a classification testing method provided in another exemplary embodiment of this application is shown;

[0027] Figure 8 A schematic diagram of a test image sample provided in an exemplary embodiment of this application is shown;

[0028] Figure 9 A schematic diagram of a test image sample provided in another exemplary embodiment of this application is shown;

[0029] Figure 10 A schematic diagram of a test image sample provided in another exemplary embodiment of this application is shown;

[0030] Figure 11 A block diagram of a classification testing apparatus provided in an exemplary embodiment of this application is shown;

[0031] Figure 12 A schematic diagram of the structure of a computer device provided in an exemplary embodiment of this application is shown. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0033] In the following description, when referring to the accompanying drawings, the same numbers in different drawings denote the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0034] In the description of this application, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. In the description of this application, it should be noted that, unless otherwise explicitly specified and limited, the terms "connected" and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances. Furthermore, in the description of this application, unless otherwise stated, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0035] First, let me introduce some of the terms used in this application:

[0036] Tag tree: A tree-like structure created by organizing multiple tags based on their hierarchical relationships. For example, such as... Figure 1 As shown, it is a schematic diagram of the tag tree structure provided in an exemplary embodiment of this application. The tag tree includes a root node, child nodes and leaf nodes. Among them, "animal" and "plant" are located on two child nodes of the tag tree respectively; "human", "cat", "dog", "tree" and "grass" are located on five leaf nodes of the tag tree respectively. That is, the leaf node is the last level node of each branch in the tag tree. "human", "cat", "dog", "tree" and "grass" on the tag tree can be called leaf tags.

[0037] For example, multiple leaf labels can include labels on nodes at the same level, such as "cat", "dog", "tree", and "grass", which are all labels corresponding to second-level child nodes in the label tree; multiple leaf labels can also include labels on nodes at different levels, such as "tree" being a label corresponding to a second-level child node in the label tree, and "person" being a label corresponding to a first-level child node in the label tree, and "tree" and "person" being labels on nodes at different levels.

[0038] As applications for multi-label image classification become increasingly prevalent in daily life, the accuracy of these classifications is becoming ever more crucial. For example, in the field of autonomous driving, the classification results of objects in images directly impact the safety of autonomous driving. Therefore, comprehensive system testing of these applications is essential. In actual testing, testers often face black-box scenarios where the code, model, and data are all transparent, raising concerns about the adequacy of the tests. Secondly, test samples (also known as test cases) used in application testing are mostly collected manually, which is time-consuming, labor-intensive, and inefficient. Some tests use adversarial generation to generate test samples, but in black-box testing, the model and training data are unknown, making adversarial generation of test samples difficult to implement. Furthermore, obtaining the expected output of test samples is often difficult, requiring manual annotation in many cases.

[0039] To address the aforementioned technical problems, this application proposes a classification test method. For detailed implementation steps, please refer to the following embodiment.

[0040] The classification test method provided in this application can be applied to computer systems. For example... Figure 2 The diagram shown is a block diagram of a computer system provided in an exemplary embodiment of this application. The computer system includes a computer device 220, a communication network 240, and a server 260.

[0041] Computer device 220 contains an operating system and applications. This operating system provides a runtime environment for testing the multi-label classifier. For example, the operating system could be Android, iOS, HermeticOS, Windows, or Linux.

[0042] For example, a multi-label classifier is used to classify images with multiple labels. For instance, an image containing "cat" and "dog" can be classified using a multi-label classifier, determining that the image's labels include "cat" and "dog." Multi-label classifiers can be applied to various applications to support multi-label classification tasks within those applications. For example, such applications may include instant messaging applications, news and information applications, lifestyle service applications, financial service applications, etc.

[0043] For example, the computer device 220 may include at least one of a smartphone, tablet computer, laptop computer, desktop computer, or notebook computer.

[0044] A communication network 240 exists between computer device 220 and server 260. This communication network 240 can be a wired network or a wireless network. For example, a wired network can be a metropolitan area network, a local area network, a fiber optic network, etc.; a wireless network can be a mobile communication network, a wireless Fidelity network (WiFi), etc.

[0045] Server 260 provides background services for the operation of the operating system and / or applications in computer device 220. For example, server 260 provides background services for testing a multi-label classifier in computer device 220. Server 260 may include at least one of the following: a single server 260, multiple servers 260, a cloud computing platform, and a virtualization center.

[0046] For example, the classification testing method provided in this application can be performed independently by the computer device 220; for instance, the data required for multi-label classifier testing is stored in the local memory of the computer device 220, and the computer device 220 can independently test the multi-label classifier.

[0047] Alternatively, the classification testing method provided in this application can be completed collaboratively by computer device 220 and server 260; for example, when executing the above classification testing method, computer device 220 needs to obtain the data required for multi-label classifier testing from server 260. For example, server 260 includes data storage server 2602 and search engine server 2604; computer device 220 can obtain test samples for the multi-label classifier from search engine server 2604 through communication network 240, and store the test samples in data storage server 2602 for later use; computer device 220 can also, after testing the multi-label classifier based on the test samples, separately store test samples that meet preset conditions and test samples that do not meet preset conditions in data storage server 2602.

[0048] Those skilled in the art will understand that the number of computer devices 220 in the above-described computer system can be more or less. For example, the computer device 220 in the above-described computer system can be only one, or dozens or hundreds, or more. The embodiments of this application do not limit the number and type of computer devices 220 in the computer system.

[0049] Figure 3 A flowchart of a classification testing method provided in an exemplary embodiment of this application is shown, which can be applied to Figure 1 In the computer device shown, the method includes:

[0050] Step 302: Obtain the leaf tag set from the tag tree.

[0051] A leaf tag set includes at least two leaf tags. A leaf tag is a tag corresponding to a leaf node in a tag tree. A tag tree is a tree structure constructed based on the hierarchical relationship of tag categories.

[0052] For example, a label tree is pre-configured in the computer device; leaf labels are obtained from the label tree to generate a leaf label set. For instance, the computer device determines that the initial leaf label set is empty; the computer device traverses all child nodes in the label tree, obtains leaf labels from leaf nodes, obtains at least two leaf labels, and adds these at least two leaf labels to the leaf label set. Figure 1 Taking the tag tree shown as an example, the computer device traverses the 7 child nodes in the tag tree and obtains 5 leaf tags, namely "cat", "dog", "tree", "grass" and "person", and adds the above 5 leaf tags to the leaf tag set.

[0053] The aforementioned hierarchical relationship includes: the label category of the parent node in the label tree contains the label category of the current node, that is, the label category of the current node belongs to the label category of the parent node. For example, the label category of the parent node is "animal", which contains the label category "cat" of the current node.

[0054] For example, the leaf labels mentioned above are used for classifying images containing multiple label elements. An image may contain multiple label elements, and a multi-label classifier can identify these elements, determine the multiple labels the image possesses, and then classify the image based on these labels. For instance, if an image includes images of cats and dogs, the multi-label classifier can identify the images of cats and dogs in the image and determine that the image possesses the labels "cat" and "dog".

[0055] Step 304: Traverse at least two leaf labels in the leaf label set to generate an M-dimensional label combination.

[0056] Here, the dimension M of the label combination refers to the dimension M based on the label partitioning, with each leaf label corresponding to one dimension; each M-dimensional label combination consists of M leaf labels, for example, a two-dimensional label combination includes 2 leaf labels; M is a positive integer greater than 1. For example, the dimension M of the above label combination can be predefined in the program.

[0057] In an M-dimensional label combination, the M leaf labels are different. For example, the computer device traverses at least two leaf labels in the leaf label set, and generates an M-dimensional label combination for every M different leaf labels extracted, until no more M-dimensional label combinations different from the extracted combinations can be extracted from the leaf label set.

[0058] For example, the leaf label set S = {C1, C2, ..., Cd}, and the M-dimensional label combination L extracted from S. M ={{l1,l2,……,l M}|l i ∈S, l j ∈S, l i ≠l j , 1≤i≠j≤M}. For example, the leaf label set S={cat, dog, tree, grass, person}, M takes the value 2; the two-dimensional label combination L2 is {cat, dog; cat, tree; cat, grass; cat, person; dog, tree; dog, grass; dog, person; tree, grass; tree, person; grass, person}, a total of 10 M-dimensional label combinations.

[0059] Optionally, the value of M can be a single numerical value, such as M being 2; or, the value of M can also be a set of numerical values, including at least two numerical values. For example, if the value of M is {2,3}, then the M-dimensional label combination includes two-dimensional label combinations and three-dimensional label combinations. If the leaf label set S = {cat, dog, tree, grass, person}, and M is 2, the two-dimensional label combination L2 is {cat, dog; cat, tree; cat, grass; cat, person; dog, tree; dog, grass; dog, person; tree, grass; tree, person; grass, person}, and the three-dimensional label combination L3 is {cat, dog, tree; cat, dog, grass; cat, dog, person; cat, tree, grass; cat, tree, person; cat, tree, person; cat, grass, person; dog, tree, grass; dog, tree, person; dog, tree, person; dog, grass, person; tree, grass, person}.

[0060] Step 306: Based on the test set corresponding to each M-dimensional label combination, perform classification tests on the multi-label classifier.

[0061] For example, each M-dimensional label combination corresponds to its own test set; the test set may be stored in the local memory of a computer device; or, the test set may be stored on a server. The terminal can retrieve the test set corresponding to each M-dimensional label combination from the local memory or the server.

[0062] When there are at least two M-dimensional label combinations, the computer device can sequentially obtain at least two test sets corresponding to at least two M-dimensional label combinations from the memory or server; then, after obtaining at least two test sets, or during the process of obtaining at least two test sets, the multi-label classifier is sequentially classified using at least two test sets.

[0063] For example, the test set corresponding to the M-dimensional label combination includes at least one test sample, and each test sample includes label elements corresponding to the M leaf labels in the M-dimensional label combination. For instance, the test samples include test image samples, and a test image sample corresponding to the two-dimensional label combination {person, tree} includes images of a person and a tree.

[0064] For example, the multi-label classifier described above is composed of at least one of neural networks, K-nearest neighbors algorithm, decision tree, kernel method of vector output, and boosting algorithm. The types of multi-label classifiers are not limited in the embodiments of this application.

[0065] In summary, the classification testing method provided in this embodiment, when testing a multi-label classifier, obtains at least two leaf labels from a pre-set label tree, then traverses the at least two leaf labels to generate M-dimensional label combinations, and then performs classification tests on the multi-label classifier based on the test set corresponding to each M-dimensional label combination. This tests whether the multi-label classifier can correctly classify test samples with M labels, ensuring that the test of the multi-label classifier can completely cover the M-dimensional leaf label combinations in the label tree. In other words, the multi-label classifier is fully tested in the classification of M-dimensional labels, so that the test of the multi-label classifier satisfies test adequacy, which refers to the degree to which the multi-label classifier is tested.

[0066] In some embodiments, during the testing of a multi-label classifier, classification test results are also provided to indicate the coverage of the multi-label classifier's classification ability with multiple M-dimensional label combinations, such as... Figure 4 As shown. Figure 4 A flowchart of a classification testing method provided in an exemplary embodiment of this application is shown, which can be applied to Figure 1 In the computer device shown, the method includes:

[0067] Step 302: Obtain the leaf tag set from the tag tree.

[0068] A leaf tag set includes at least two leaf tags. A leaf tag is a tag corresponding to a leaf node in a tag tree. A tag tree is a tree structure constructed based on the hierarchical relationship of tag categories.

[0069] Step 304: Traverse at least two leaf labels in the leaf label set to generate an M-dimensional label combination.

[0070] Each M-dimensional label combination consists of M leaf labels, where M is a positive integer greater than 1.

[0071] Step 306a: For each M-dimensional label combination, input the test samples in the test set corresponding to the M-dimensional label combination into the multi-label classifier to obtain the test label set after classifying the test samples.

[0072] For each M-dimensional label combination, the computer device performs independent classification tests on the multi-label classifier in sequence. That is, after the computer device tests the multi-label classifier with the label set corresponding to one M-dimensional label combination, it then tests the multi-label classifier with the label set corresponding to another M-dimensional label combination.

[0073] Alternatively, at least two M-dimensional label combinations correspond to at least two test sets, and the computer device randomly selects test samples from at least two test sets to test the multi-label classifier until the test samples in at least two test sets are exhausted.

[0074] The computer device inputs the test sample into the multi-label classifier, and the multi-label classifier outputs the set of labels that the test sample has, which is the test label set after classifying the test sample.

[0075] Step 306b: Determine that the M-dimensional label combination is a subset of the test label set, and identify the M-dimensional label combination as the covered label combination.

[0076] The computer device compares the tags in the test tag set with the tags in the M-dimensional tag combination. If the M-dimensional tag combination is a subset of the test tag set, then the M-dimensional tag combination is determined to be a covered tag combination. For example, if the tags in the test tag set are the same as the tags in the M-dimensional tag combination, the computer device determines that the M-dimensional tag combination is a subset of the test tag set.

[0077] Optionally, after determining that the M-dimensional tag combination is a covered tag combination, the computer device may also add the M-dimensional tag combination to the covered tag combination set.

[0078] Optionally, for each M-dimensional label combination, the test set includes at least two test samples, and the at least two test samples correspond to at least two test label sets; the computer device determines the M-dimensional label combination as a covered label combination when the M-dimensional label combination is a subset of at least one of the at least two test label sets.

[0079] For example, the test set corresponding to the M-dimensional label combination includes a first test sample and a second test sample. The computer device inputs the first test sample into a multi-label classifier to obtain a first test label set for the first test sample. If the M-dimensional label combination is a subset of the first test label set, the M-dimensional label combination is determined to be a covered label combination. If the M-dimensional label combination does not exist in the covered label combination set H, the M-dimensional label combination is added to the covered label combination set H. The computer device continues to input the second test sample into the multi-label classifier to obtain a second test label set for the second test sample. If the M-dimensional label combination is a subset of the second test label set, the M-dimensional label combination is determined to be a covered label combination. If the M-dimensional label combination exists in the covered label combination set H, the label combination in the current covered label combination set H is maintained, and the M-dimensional label combination is not added back to the covered label combination set H.

[0080] Optionally, for each test sample, if the M-dimensional label combination is a subset of the test label set, the test sample is further augmented to obtain augmented samples; the augmented samples are input into a multi-label classifier to obtain a validation label set for classifying the augmented samples; the M-dimensional label combination is determined to be a subset of the validation label set and is identified as the covered label combination.

[0081] For example, taking the test image sample as the test sample, when the M-dimensional label combination is a subset of the test label set, the computer device can perform augmentation processing on the test image sample in at least one of the following ways: flipping, such as flipping the image vertically and / or horizontally; changing the color, such as changing the brightness, contrast, saturation and hue of the image.

[0082] For example, the computer device inputs the augmented sample into a multi-label classifier, which outputs the set of labels possessed by the augmented sample, thus obtaining the validation label set after classifying the augmented sample. If the labels in the M-dimensional label combination are the same as the labels in the validation label set, the M-dimensional label combination is determined to be a covered label combination. If the M-dimensional label combination does not exist in the covered label combination set H, the M-dimensional label combination is added to the covered label combination set H.

[0083] For example, the aforementioned covered tag set is initially empty; for instance, it can be set to empty when the classification test method is first executed.

[0084] Step 306c: Determine that the M-dimensional label combination is a non-subset of the test label set, and add the test samples corresponding to the test label set to the defect sample set.

[0085] The M-dimensional label combination is a non-subset of the test label set; that is, the M-dimensional label combination is not a subset of the test label set. For example, if the labels in the M-dimensional label combination differ from the labels in the test label set, the computer device adds the test sample corresponding to the test label set to the defect sample set. For example, the defect sample set is initially empty; for instance, it can be set to empty when the classification test method is first executed.

[0086] In some embodiments, for each test sample, if the M-dimensional label combination is a subset of the test label set, the test sample is further augmented to obtain augmented samples. The augmented samples are then input into a multi-label classifier to obtain a validation label set for classifying the augmented samples. If the M-dimensional label combination is determined to be a non-subset of the validation label set, the test samples and / or augmented samples corresponding to the validation label set are added to the defect sample set. For example, if the labels in the M-dimensional label combination are different from the labels in the validation label set, the test samples and / or augmented samples corresponding to the validation label set are added to the defect sample set. Here, "a non-subset of the validation label set" means that the M-dimensional label combination is not a subset of the validation label set.

[0087] Step 306d: After performing a classification test on the samples for the M-dimensional label combination, calculate the ratio of the number of covered label combinations to the number of M-dimensional label combinations to obtain the classification test results.

[0088] For example, the computer device counts the number of covered label combinations in the covered label combination set and counts the total number of M-dimensional label combinations; it calculates the ratio of the number of covered label combinations to the total number of M-dimensional label combinations to obtain the classification test results.

[0089] For example, the classification test results include the coverage of M-dimensional label combinations LLCC(M), which is expressed by the formula LLCC(M):

[0090] LLCC(M) = |{l∈L M |b∈B,PassTestOracle(b)}| / |L M |;

[0091] Where B represents the test set corresponding to the M-dimensional label combination; b represents the test sample in the test set; PassTestOracle(b) represents the M-dimensional label combination that passes the test sample; |{l∈L M |b∈B,PassTestOracle(b)}| represents the number of covered tag combinations; |L M| represents the total number of M-way label combinations; LLCC(M) represents the M-way LLCC probability, which is the M-way Leaf Label Combination Coverage.

[0092] The leaf labels of a label tree can reflect the classification ability of a multi-label classifier. Therefore, in this embodiment, LLCC(M) is used to measure the multi-label classifier's ability to recognize leaf labels. For example, the M-dimensional label combination coverage ratio is used to represent the classification ability of the multi-label classifier. The larger the ratio, the more M-dimensional label combinations it can cover, and the stronger the classification ability of the multi-label classifier; the smaller the ratio, the fewer M-dimensional label combinations it can cover, and the weaker the classification ability of the multi-label classifier.

[0093] In summary, the classification testing method provided in this embodiment, after classifying the test samples, outputs the ratio of the number of covered label combinations to the number of M-dimensional label combinations. This allows users to intuitively see the proportion of covered label combinations to the total number of label combinations, thus understanding the classification capability of the multi-label classifier. Furthermore, this method compares the test results corresponding to the test samples with the label combinations to determine whether the multi-label classifier can correctly classify the test labels. In other words, it directly uses the label combinations as the expected output of the test samples, eliminating the need for manual label annotation to obtain accurate expected outputs. This solves the problem of difficulty in obtaining test oracles, which are mechanisms for determining whether a test passes or fails.

[0094] In the above embodiments, the computer device also collects the covered label combinations, thereby identifying the label combinations that the multi-label classifier cannot cover and knowing the classification defects of the multi-label classifier; and collects the test samples that fail the test, which can be used for subsequent classification defect analysis of the multi-label classifier, and even for retraining and testing of the multi-label classifier.

[0095] In the above embodiments, the computer device further augments the test sample and then re-verifies the test result of the test sample based on the augmented sample, thereby ensuring the robustness of the multi-label classifier on the test sample.

[0096] For example, the test set can be obtained by a computer device searching the network after generating the M-dimensional label combination, as shown in the following implementation: Figure 5 As shown in the Chinese embodiment. Figure 5 A flowchart of a classification testing method provided in an exemplary embodiment of this application is shown, which can be applied to Figure 1 In the computer device shown, the method includes:

[0097] Step 302: Obtain the leaf tag set from the tag tree.

[0098] A leaf tag set includes at least two leaf tags. A leaf tag is a tag corresponding to a leaf node in a tag tree. A tag tree is a tree structure constructed based on the hierarchical relationship of tag categories.

[0099] Step 304: Traverse at least two leaf labels in the leaf label set to generate an M-dimensional label combination.

[0100] Each M-dimensional label combination consists of M leaf labels, where M is a positive integer greater than 1.

[0101] Step 305a: For each M-dimensional label combination, connect the M leaf labels in the M-dimensional label combination using a parallel relationship to obtain the search keywords.

[0102] For example, for an M-dimensional label combination, the computer device concatenates the M leaf labels in the M-dimensional label combination side by side, such as the M leaf labels {C1, C2, ..., C...}. M}, when connected in parallel, can be: C1 and C2 and...and C M , change "C1 and C2 and...and C M "As a search keyword, M is less than or equal to d."

[0103] Step 305b: Based on web search, obtain G candidate samples corresponding to the search keywords.

[0104] For example, a computer device calls a web search engine to search for keywords, obtaining G candidate samples, where G is a positive integer. For instance, if the computer device calls a web search engine to search for images containing the search keywords, it will obtain G candidate image samples.

[0105] Step 305c: Select test samples from the G candidate samples and add them to the test set corresponding to the M-dimensional label combination.

[0106] Optionally, the computer device determines the first K candidate samples out of the G candidate samples as test samples, where K is a positive integer less than or equal to G. For example, the computer device searches for and obtains G candidate samples arranged in order, and extracts the first K candidate samples from the G candidate samples as test samples. For instance, the computer device determines the first K candidate image samples out of the G candidate image samples as test samples.

[0107] For example, the value of K mentioned above can be an empirical value set by the user. For instance, K could be 3; the computer device would then select the top 3 candidate image samples out of 20 candidate image samples as test samples.

[0108] Step 306: Based on the test set corresponding to each M-dimensional label combination, perform classification tests on the multi-label classifier.

[0109] For example, step 306 can be derived from the above. Figure 4 Steps 306a to 306d in the process are implemented.

[0110] In summary, the classification testing method provided in this embodiment, after obtaining an M-dimensional label combination, can use the M leaf labels in the M-dimensional label combination as search keywords and automatically search for the search keywords through a network search, thereby obtaining test samples containing the elements corresponding to the M leaf labels. This method can automatically generate test cases for a multi-label classifier without manual intervention, saving manpower and improving the efficiency of test case generation, thus solving the problem of test case generation. Secondly, the above method can also select the top K candidate samples as test samples after obtaining multiple candidate samples. This is because the multiple candidate samples provided by the search engine are arranged in descending order of similarity to the search keywords, thus ensuring that the top K candidate samples include the elements corresponding to all M leaf labels, thereby ensuring the testing of the multi-label classifier.

[0111] For example, a multi-label classifier includes a multi-label image classifier. The classification testing method provided in the above embodiments can be applied to the field of image classification technology to test multi-label image classifiers, such as... Figure 7 This paper illustrates a method for performing classification tests on a multi-label image classifier, which includes:

[0112] Step 402: Obtain the leaf tag set from the tag tree. The leaf tag set includes at least two leaf tags.

[0113] For example, a computer device traverses a label tree, extracts at least two leaf labels from the tree, and generates a leaf label set. A leaf label refers to the label corresponding to a leaf node in the label tree, which is a tree-like structure constructed based on the hierarchical relationship of label categories. For example, the labels on each node of the label tree can be used to label various elements in an image. For instance, an image containing elements such as roses, daisies, and dogs can be labeled with tags like "rose," "daisy," and "dog" using a multi-label image classifier.

[0114] A dependency relationship can refer to the relationship between two adjacent nodes in a tag tree, where the tag category of the parent node includes the tag category of the child node; a parent node can include multiple child nodes. For example, if the tag of the parent node is "flower", the tags of the child nodes can include "rose", "peony", "crabapple", etc.

[0115] Step 404: Traverse at least two leaf labels in the leaf label set to generate an M-dimensional label combination.

[0116] Here, the dimension M of the label combination refers to the dimension M based on the label partitioning, with each leaf label corresponding to one dimension; each M-dimensional label combination consists of M leaf labels, for example, a five-dimensional label combination includes 5 leaf labels; M is a positive integer greater than 1. For example, the dimension M of the above label combination can be predefined in the test program of the multi-label image classifier.

[0117] In an M-dimensional label combination, the M leaf labels are different. For example, the computer device traverses at least two leaf labels in the leaf label set, and generates an M-dimensional label combination for every M different leaf labels extracted, until no more M-dimensional label combinations different from the extracted combinations can be extracted from the leaf label set.

[0118] Optionally, the value of M can be a single numerical value, such as M being 2; or, the value of M can be a set of numerical values, including at least two numerical values, such as M being {2,3}, in which case the M-dimensional label combination can cover both two-dimensional label combinations and three-dimensional label combinations.

[0119] Step 406: Based on the test image set corresponding to each M-dimensional label combination, perform classification tests on the multi-label image classifier.

[0120] For example, each M-dimensional label combination corresponds to its own set of test images. Optionally, the test image set may be pre-stored in local storage or on a server; or, the test image set may be generated by obtaining it over the network after the M-dimensional label combination has been determined.

[0121] For example, a computer device can obtain a test image set corresponding to each M-dimensional label combination from local storage or a server, and use the obtained test image set to perform classification tests on a multi-label image classifier.

[0122] Alternatively, for each M-dimensional label combination, the computer device connects the M leaf labels in the M-dimensional label combination in a parallel relationship to obtain the search keyword; based on the network search method, it searches for G candidate image samples corresponding to the search keyword; it selects test samples from the G candidate image samples and adds them to the test image set corresponding to the M-dimensional label combination; subsequently, the computer device uses the above test image set to perform classification tests on the multi-label image classifier; where G is a positive integer.

[0123] During the generation of the test image set, the computer device can invoke a web search engine to search for keywords, thereby obtaining G candidate image samples. To ensure that the images used as test samples include all label elements corresponding to the M-dimensional label combinations, the computer device can determine the first K candidate image samples from the G candidate image samples as test image samples, generating a test image set, where K is a positive integer less than or equal to G. For example, the computer device may search and obtain G candidate image samples arranged in order of relevance to M leaf labels, and then extract the first K candidate image samples from these G candidate image samples as test image samples. The value of K mentioned above is an empirical value set by the user.

[0124] For example, the test image set corresponding to the M-dimensional label combination includes at least one test image sample, and each test image sample includes label elements corresponding to the M leaf labels in the M-dimensional label combination. For instance, a test image sample corresponding to the two-dimensional label combination {person, tree} includes images of a person and a tree.

[0125] Alternatively, the classification test of the multi-label image classifier can be performed using the following steps:

[0126] a) For each M-dimensional label combination, input the test image samples from the test image set corresponding to the M-dimensional label combination into the multi-label image classifier to obtain the test label set after classifying the test image samples;

[0127] For example, there are at least two M-dimensional label combinations, and at least two M-dimensional label combinations correspond to at least two test image sets; the computer device uses each test image set to perform independent classification tests on the multi-label image classifier in turn; or, the computer device randomly selects test image samples from at least two test image sets to test the multi-label image classifier until the test image samples in at least two test image sets are exhausted.

[0128] b) Determine the M-dimensional label combination as a subset of the test label set, and identify the M-dimensional label combination as the covered label combination;

[0129] For example, the computer device determines that the labels in the M-dimensional label combination are the same as the labels in the test label set, and identifies the M-dimensional label combination as a covered label combination. If the computer device determines that the M-dimensional label combination is a covered label combination and that the M-dimensional label combination does not exist in the covered label combination set, it adds the M-dimensional label combination to the covered label combination set.

[0130] Optionally, for each M-dimensional label combination, the test image set includes at least two test image samples, and the at least two test image samples correspond to at least two test label sets; the computer device determines the M-dimensional label combination as a covered label combination when the M-dimensional label combination is a subset of at least one of the at least two test label sets.

[0131] For example, the test image set corresponding to the M-dimensional label combination includes a first test image sample and a second test image sample. The computer device inputs the first test image sample into a multi-label image classifier to obtain a first test label set for the first test image sample. If the M-dimensional label combination is a subset of the first test label set, the M-dimensional label combination is determined to be a covered label combination. If the M-dimensional label combination does not exist in the covered label combination set H, the M-dimensional label combination is added to the covered label combination set H. The computer device continues to input the second test image sample into the multi-label image classifier to obtain a second test label set for the second test image sample. If the M-dimensional label combination is a subset of the second test label set, the M-dimensional label combination is determined to be a covered label combination. If the M-dimensional label combination exists in the covered label combination set H, the label combination in the current covered label combination set H is maintained, and the M-dimensional label combination is not added back to the covered label combination set H.

[0132] Optionally, for each test image sample, if the M-dimensional label combination is a subset of the test label set, the test image sample is further augmented to obtain an augmented image sample; the augmented image sample is input into a multi-label image classifier to obtain a validation label set for classifying the augmented image sample; the M-dimensional label combination is determined to be a subset of the validation label set and is identified as the covered label combination.

[0133] For example, a computer device may augment a test image sample in at least one of the following ways: flipping, such as flipping the image vertically and / or horizontally; changing the color, such as changing the brightness, contrast, saturation, and hue of the image.

[0134] For example, the computer device inputs an augmented image sample into a multi-label image classifier, which outputs a set of labels possessed by the augmented image sample, thus obtaining a verification label set after classifying the augmented image sample. If the labels in the M-dimensional label combination are the same as the labels in the verification label set, the M-dimensional label combination is determined to be a covered label combination. If the M-dimensional label combination does not exist in the covered label combination set H, the M-dimensional label combination is added to the covered label combination set H.

[0135] c) Determine the M-dimensional label combination as a non-subset of the test label set, and add the test samples corresponding to the test label set to the defect sample set;

[0136] For example, if the labels in the M-dimensional label combination are different from the labels in the test label set, the computer device adds the test image sample corresponding to the test label set to the defect sample set.

[0137] In some embodiments, for each test image sample, if the M-dimensional label combination is a subset of the test label set, the test image sample is further augmented to obtain an augmented image sample. The augmented image sample is then input into a multi-label image classifier to obtain a verification label set for classifying the augmented image sample. If the M-dimensional label combination is determined to be a non-subset of the verification label set, the test image sample and / or augmented image sample corresponding to the verification label set is added to the defect sample set. For example, if the labels in the M-dimensional label combination are different from the labels in the verification label set, the test image sample and / or augmented image sample corresponding to the verification label set is added to the defect sample set.

[0138] d) After classifying image samples for M-dimensional label combinations, calculate the ratio of the number of covered label combinations to the number of M-dimensional label combinations to obtain the classification test results.

[0139] For example, the computer device counts the number of covered label combinations in the covered label combination set and the total number of M-dimensional label combinations; it then calculates the ratio of the number of covered label combinations to the total number of M-dimensional label combinations to obtain the classification test result. This classification test result may include the M-dimensional label combination coverage rate, which reflects the multi-label image classifier's image recognition capability for M-dimensional label combinations.

[0140] For example, the above-mentioned multi-label image classifier is composed of at least one of neural network, K-nearest neighbor algorithm, decision tree, kernel method of vector output, and boosting algorithm. The types of multi-label image classifiers are not limited in the embodiments of this application.

[0141] In summary, the classification testing method provided in this embodiment, when testing a multi-label image classifier, obtains at least two leaf labels from the label tree, then traverses at least two leaf labels to generate M-dimensional label combinations, and then performs classification tests on the multi-label image classifier based on the test image set corresponding to each M-dimensional label combination. This tests whether the multi-label image classifier can correctly classify test image samples with M labels, ensuring that the test of the multi-label image classifier can completely cover the M-dimensional leaf label combinations in the label tree, so that the multi-label image classifier is fully tested in the classification of M-dimensional labels, and the test of the multi-label image classifier satisfies the test sufficiency requirement.

[0142] After classifying the test image samples, the method can output the ratio of the number of covered label combinations to the number of M-dimensional label combinations, allowing users to intuitively see the proportion of covered label combinations to the total number of label combinations and understand the classification ability of the multi-label image classifier. Furthermore, this method compares the test results corresponding to the test image samples with the label combinations to determine whether the multi-label image classifier can correctly classify the test image samples. It uses the label combinations as the expected output of the test image samples, eliminating the need for manual label annotation to obtain accurate expected outputs and solving the problem of difficulty in obtaining test predictions.

[0143] Furthermore, the M leaf labels in the M-dimensional label combination can be used as search keywords. The search keywords are automatically searched through the network to obtain test image samples containing the elements corresponding to the M leaf labels. This automatically generates test cases for the multi-label classifier without manual intervention, saving human resources and improving the efficiency of test case generation. Therefore, the problem of test case generation is solved.

[0144] In some embodiments, the value of M can be preset by the user. For example, the value of M can be preset by the user before generating the M-dimensional label combination; for instance, it can be preset before retrieving the leaf label set from the label tree, or after retrieving the leaf label set from the label tree. For example, in response to a dimension setting operation for the label combination, the computer device sets the dimension of the label combination to M.

[0145] For example, before starting the classification test method, the computer device displays a settings interface, which includes an M-value setting control. The user can input the value of M through the M-value setting control, or select the value of M through the M-value setting control. The settings interface also includes a confirmation control. In response to the confirmation operation of the test data on the confirmation control, the computer device sets the dimension of the label combination to the current value of M.

[0146] For example, users can also construct a label tree before executing the classification test method; store the constructed label tree in local storage or on a server; the settings interface can also include a label tree selection control; users can obtain the label tree required for the classification test from local storage or on a server through the label tree selection control, and the computer device responds to the selection operation on the label tree selection control to determine the label tree used for this classification test.

[0147] In summary, the classification testing method provided in this embodiment allows the user to define the value of M to determine the coverage criterion based on the label tree used in the test. For example, two-dimensional leaf node coverage can be used to determine the coverage of the multi-label classifier for the two-dimensional label combination. The dimension of the label combination can be adjusted according to the actual needs of the classification test, thereby better covering the label test range. Secondly, the constructed label tree can be reused, improving the utilization rate of the label tree. In fact, the existing label tree can be directly used in this classification test, which can improve the efficiency of the classification test and avoid the need to construct a label tree for each classification test, resulting in a waste of human resources and low efficiency of classification testing.

[0148] It should also be noted that the steps in the embodiments of this application can be performed as a whole.

[0149] Alternatively, this embodiment can be divided into two parts: the first part is preparation: test case generation; the second part is testing: classification testing of the multi-label classifier. First, leaf label sets are obtained from the label tree. At least two leaf labels in the leaf label sets are traversed to generate M-dimensional label combinations. Then, the test set corresponding to the M-dimensional label combinations is obtained through a network search and stored, completing the preparation. Next, the multi-label classifier is tested using the test set, completing the subsequent testing. For example... Figure 5 Steps 302 to 305c in the illustrated embodiment can be used as preparatory steps, and step 306 can be used as a testing step.

[0150] Alternatively, this embodiment can be divided into two parts: the first part, preparation: generating M-dimensional label combinations; and the second part, testing: classification testing of the multi-label classifier. First, leaf label sets are obtained from the label tree. At least two leaf labels in the leaf label set are traversed to generate M-dimensional label combinations, which are then stored, completing the preparation. Next, the M-dimensional label combinations are obtained, and a test set corresponding to each M-dimensional label combination is obtained through a network search. The multi-label classifier is then used to perform classification testing on the test set, completing the subsequent testing. For example... Figure 5Steps 302 to 304 in the illustrated embodiment can be used as preparatory steps, and steps 305a to 306 can be used as testing steps.

[0151] For example, in the second and third scenarios described above, the preparation and testing work can be performed by the same computer device or by different computer devices.

[0152] Figure 7 A flowchart of a classification testing method provided in an exemplary embodiment of this application is shown, which can be applied to Figure 1 In the computer device shown, taking an image as an example when the test sample is an image, the method includes:

[0153] Step 501: Obtain the multi-label classifier, the label combination set, and set the covered label combination set to empty and the defect sample set to empty.

[0154] An empty set of covered label combinations can be represented as H=, and an empty set of defect samples can be represented as D=.

[0155] The tag combination set L includes multiple M-dimensional tag combinations. That is, in this embodiment, the multiple M-dimensional tag combinations are pre-generated and can be directly obtained by the computer device.

[0156] Step 502: Obtain a tag combination from the tag combination set.

[0157] The computer device retrieves a tag combination a from the tag combination set L.

[0158] Step 503: Use the leaf tags in the tag combination as search keywords to retrieve the test image set.

[0159] The computer device uses a search engine to retrieve test image set B by using the leaf tags in tag combination a as search keywords.

[0160] For example, the computer device connects the M leaf tags in tag combination a in parallel to obtain search keywords, automatically inputs the search keywords into the search engine, obtains multiple candidate images, and selects the top K from the multiple candidate images to generate a test image set B.

[0161] Step 504: Select a test image sample from the test image set, input it into the multi-label classifier, and obtain the classification test result.

[0162] The computer device selects a test image sample b from the test image set B, inputs it into the multi-label classifier f, and outputs the test label after classifying the test image sample b, thus obtaining the classification test result f(b).

[0163] Step 505: Determine whether the classification test results contain the label combination.

[0164] The computer device determines whether the classification test result f(b) contains the label combination a; that is, whether the label set represented by f(b) contains the label set represented by a. If the classification test result f(b) contains the label combination a, proceed to step 506; if the classification test result f(b) does not contain the label combination a, proceed to step 511.

[0165] Step 506: Determine whether the test image sample passes the metamorphosis test.

[0166] The computer device augments the test image sample b, for example, by flipping it, to obtain augmented image sample b'. The computer device inputs the augmented image sample b' into a multi-label classifier f to obtain a validation label set f(b'). It then determines whether the validation label set f(b') contains the label combination a. This process is called metamorphosis test. If the validation label set f(b') contains the label combination a, the metamorphosis test is passed, and step 507 is executed. If the validation label set f(b') does not contain the label combination a, the metamorphosis test is failed, and step 511 is executed.

[0167] Step 507: Add the tag combination to the set of covered tag combinations.

[0168] The computer device adds tag combination 'a' to the overridden tag combination set H, and then performs step 508.

[0169] Step 508: Determine if the test image set is empty.

[0170] If the test image set B is not empty, the computer device returns to step 504; if the test image set B is empty, it executes step 509.

[0171] Step 509: Determine if the tag combination set is empty.

[0172] If the label combination set L is not empty, the computer device returns to step 502; if the label combination set L is empty, it executes step 510.

[0173] Step 510: Calculate coverage.

[0174] The computer device calculates the ratio between the number of covered tag combinations and the number of tag combinations in the tag combination set to obtain the coverage rate. It should be noted that the coverage rate can be expressed as LLCC(M).

[0175] Step 511: Add the test image sample and / or augmented image sample to the defect sample set.

[0176] The computer device adds the test image sample b and the augmented image sample b' to the defect sample set D.

[0177] In summary, the classification testing method provided in this embodiment offers a series of coverage criteria based on label trees. When a coverage criterion is selected, a set of label combinations to be covered is obtained. Each label combination in this set is used as a keyword for image search. The top K images found in the search are selected as test image samples to generate a test image set. The correctness and robustness of the multi-label classifier in image classification can be assessed by comparing the classification test results with the label combinations. If the classification test result for an image contains the corresponding label combination, the image classification is considered correct. Then, a metamorphic test is used to examine the robustness of the multi-label classifier on the image. If one of the K images in a set passes the test, the label combination corresponding to that set is considered covered; if no image passes the test, the label combination corresponding to that set is considered uncovered. After the test, the coverage rate and defect sample set can be obtained as optimization references for the multi-label classifier.

[0178] For example, with Figure 1 Taking the label tree as an example, the above embodiments will be illustrated in detail. Figure 1 We know that the tag tree has 5 leaf nodes: cat, dog, tree, grass, and person. Assuming the selected coverage criterion is two-dimensional leaf node coverage (i.e., M is 2), the tag combination set L to be covered is {cat, dog; cat, tree; cat, grass; cat, person; dog, tree; dog, grass; dog, person; tree, grass; tree, person; grass, person}, a total of 10 two-dimensional tag combinations. A copy L' is also created for L. The already covered tag combination set is initialized. Defect sample set Assuming the transformation relationship chosen for subsequent transformation tests is horizontally flipping an image to obtain a new image, the classification test results of the multi-label classifier on these two images should remain consistent. For each two-dimensional label combination, we perform the following operation: extract the label combination "cat, dog" from L and perform the test. First, use "cat and dog" as keywords to perform an image search on a search engine, and take the first 3 returned images as the test image set B: image b1, image b2, and image b3, as follows. Figure 8 As shown.

[0179] First, image b1 is input into a multi-label classifier. Assume the classification test result is f(b1) = {cat}, which is the multi-label classifier's prediction output for the image. Clearly, f(b1) does not contain the label combination "cat, dog," indicating an error in the multi-label classifier's prediction. Therefore, image b1 is added to the defective image set D.

[0180] Next, image b2 is input into the multi-label classifier, assuming the classification test result is f(b2) = {cat, dog}. At this point, f(b2) contains the label combination "cat, dog", and the multi-label classifier correctly classifies image b2. Continuing to consider the robustness of the multi-label classifier on image b2, image b2 is horizontally flipped to obtain image b2', as shown below. Figure 9 As shown.

[0181] Image b2' is input into a multi-label classifier. Assuming the classification test result is f(b2') = {cat, dog}, it is clear that f(b2) = f(b2'). The multi-label classifier passes the metamorphosis test and adds the label combination "cat, dog" to the already covered label combination set H.

[0182] Next, input image b3 into the multi-label classifier, assuming the classification test result is f(b3) = {cat, dog}. At this point, f(b3) contains the label combination "cat, dog", and the multi-label classifier correctly classifies image b3. Continuing to consider the robustness of the multi-label classifier on image b3, we horizontally flip image b3 to obtain image b3', as shown below. Figure 10 As shown.

[0183] Image b3' is input into a multi-label classifier. Assuming the classification test result is f(b3') = {cat}, it is clear that f(b3) ≠ f(b3'). The multi-label classifier violates the metamorphosis test, that is, the multi-label classifier is not robust enough on image b3'. At this time, b3 and b3' are added to the defective image set D.

[0184] The remaining label combinations in L are tested in the same way as described above until L is an empty set, at which point the testing process stops. At this point, the coverage rate of the two-dimensional leaf labels |H| / 10 can be calculated based on the set of covered label combinations H, and the defect image set D can also be obtained.

[0185] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0186] Figure 11 This illustration shows a structural block diagram of a classification testing apparatus provided in an exemplary embodiment of this application. The classification testing apparatus can be implemented as all or part of a computer device through software, hardware, or a combination of both. The apparatus includes:

[0187] The acquisition module 602 is used to acquire a set of leaf tags from the tag tree. The set of leaf tags includes at least two leaf tags. The leaf tags refer to the tags corresponding to the leaf nodes on the tag tree. The tag tree is a tree structure constructed based on the hierarchical relationship of tag categories.

[0188] The generation module 604 is used to traverse at least two leaf labels in the leaf label set and generate M-dimensional label combinations. Each M-dimensional label combination consists of M leaf labels, where M is a positive integer greater than 1.

[0189] The testing module 606 is used to perform classification tests on the multi-label classifier based on the test set corresponding to each M-dimensional label combination.

[0190] In some embodiments, the test module 606 is configured to:

[0191] For each M-dimensional label combination, the test samples in the test set corresponding to the M-dimensional label combination are input into the multi-label classifier to obtain the test label set after classifying the test samples;

[0192] The M-dimensional label combination is determined to be a subset of the test label set, and the M-dimensional label combination is determined to be a covered label combination;

[0193] After performing a classification test on the M-dimensional label combination, the ratio of the number of covered label combinations to the number of M-dimensional label combinations is calculated to obtain the classification test results.

[0194] In some embodiments, the test module 606 is configured to:

[0195] When the M-dimensional label combination is a subset of the test label set, the test sample is augmented to obtain augmented samples;

[0196] The augmented sample is input into the multi-label classifier to obtain a set of validation labels for classifying the augmented sample;

[0197] The M-dimensional label combination is determined to be a subset of the verification label set, and the M-dimensional label combination is determined to be the covered label combination.

[0198] In some embodiments, the test samples include at least two, and the at least two test samples correspond to at least two test label sets;

[0199] The test module 606 is configured to determine the M-dimensional label combination as the covered label combination when the M-dimensional label combination is a subset of at least one of the at least two test label sets.

[0200] In some embodiments, the test module 606 is further configured to determine that the M-dimensional label combination is a non-subset of the test label set, and add the test sample corresponding to the test label set to the defect sample set.

[0201] In some embodiments, the acquisition module 602 is further configured to:

[0202] For each M-dimensional label combination, the M leaf labels in the M-dimensional label combination are connected in a parallel relationship to obtain the search keywords;

[0203] Based on the web search method, G candidate samples corresponding to the search keywords are obtained, where G is a positive integer;

[0204] The test sample is selected from the G candidate samples and added to the test set corresponding to the M-dimensional label combination.

[0205] In some embodiments, the acquisition module 602 is used to determine the first K candidate samples among the G candidate samples as the test samples, where K is a positive integer less than or equal to G.

[0206] In some embodiments, the device further includes a setting module 608;

[0207] Setting module 608 is used to respond to the setting operation of the dimension of the tag combination, and to set the dimension of the tag combination to M.

[0208] In summary, the classification testing device provided in this embodiment, when testing a multi-label classifier, obtains at least two leaf labels from a pre-set label tree, then traverses the at least two leaf labels to generate M-dimensional label combinations, and then performs classification tests on the multi-label classifier based on the test set corresponding to each M-dimensional label combination. This tests whether the multi-label classifier can correctly classify test samples with M labels, ensuring that the test of the multi-label classifier can completely cover the M-dimensional leaf label combinations in the label tree, thus ensuring that the multi-label classifier is fully tested in label classification with dimension M.

[0209] It should be noted that the classification testing device provided in the above embodiments is only illustrated by the division of the functional modules described above when executing the classification testing method. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the classification testing device and the classification testing method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0210] Figure 12 A schematic diagram of a computer device provided in an exemplary embodiment of this application is shown. This computer device may be a terminal or a server that executes the classification test method provided in this application. Specifically:

[0211] Computer device 700 includes a central processing unit (CPU) 701, a system memory 704 including random access memory (RAM) 702 and read-only memory (ROM) 703, and a system bus 705 connecting the system memory 704 and the CPU 701. Computer device 700 also includes a basic input / output system (I / O system) 706 that facilitates information transfer between various devices within the computer, and a mass storage device 707 for storing the operating system 713, application programs 714, and other program modules 715.

[0212] The basic input / output system 706 includes a display 708 for displaying information and an input device 709 for user input, such as a mouse or keyboard. Both the display 708 and the input device 709 are connected to the central processing unit 701 via an input / output controller 710 connected to the system bus 705. The basic input / output system 706 may also include the input / output controller 710 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 710 also provides output to a display screen, printer, or other types of output devices.

[0213] Mass storage device 707 is connected to central processing unit 701 via a mass storage controller (not shown) connected to system bus 705. Mass storage device 707 and its associated computer-readable media provide non-volatile storage for computer device 700. That is, mass storage device 707 may include computer-readable media (not shown) such as hard disk or compact disc read-only memory (CD-ROM) drive.

[0214] Computer-readable media can include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid-state storage technologies, CD-ROM, digital versatile optical disc (DVD), or solid-state drives (SSD), other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Random access memory can include resistive random access memory (ReRAM) and dynamic random access memory (DRAM). Of course, those skilled in the art will recognize that computer storage media are not limited to the above-mentioned types. The system memory 704 and the mass storage device 707 mentioned above can be collectively referred to as memory.

[0215] According to various embodiments of this application, the computer device 700 can also be connected to a remote computer on a network, such as the Internet, for operation. That is, the computer device 700 can be connected to a network 712 via a network interface unit 711 connected to the system bus 705, or the network interface unit 711 can be used to connect to other types of networks or remote computer systems (not shown).

[0216] The aforementioned memory also includes one or more programs, which are stored in the memory and configured to be executed by the CPU to implement the classification test method described above.

[0217] This application also provides a computer-readable storage medium storing at least one instruction, which is loaded and executed by a processor to implement the classification test method described in the above embodiments.

[0218] Optionally, the computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), solid-state drives (SSDs), or optical discs, etc. The random access memory may include resistive random access memory (ReRAM) and dynamic random access memory (DRAM).

[0219] This application also provides a computer program product (or computer program) including computer instructions stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the classification test method provided in the above embodiments.

[0220] According to another aspect of this application, a chip is provided, the chip including programmable logic circuits and / or program instructions, which, when the chip is running, are used to implement the classification test methods described in the above embodiments.

[0221] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0222] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0223] The above description is merely an exemplary embodiment that can be implemented in this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A classification test method, characterized in that, The method includes: Obtain a set of leaf tags from the tag tree. The set of leaf tags includes at least two leaf tags. The leaf tags refer to the tags corresponding to the leaf nodes on the tag tree. The leaf nodes include the last-level nodes of each branch in the tag tree. The tag tree is a tree structure constructed based on the hierarchical relationship of tag categories. Iterate through at least two leaf labels in the leaf label set to generate M-dimensional label combinations. Each M-dimensional label combination consists of M leaf labels, where M is a positive integer greater than 1. Based on the test image set corresponding to each M-dimensional label combination, a classification test is performed on the multi-label image classifier, which is used to classify images with multiple labels.

2. The method according to claim 1, characterized in that, The classification test of the multi-label image classifier based on the test image set corresponding to each M-dimensional label combination includes: For each M-dimensional label combination, the test samples in the test image set corresponding to the M-dimensional label combination are input into the multi-label image classifier to obtain a test label set after classifying the test samples. The M-dimensional label combination is determined to be a subset of the test label set, and the M-dimensional label combination is determined to be a covered label combination; After performing a classification test on the M-dimensional label combination, the ratio of the number of covered label combinations to the number of M-dimensional label combinations is calculated to obtain the classification test results.

3. The method according to claim 2, characterized in that, The step of determining that the M-dimensional label combination is a subset of the test label set and determining that the M-dimensional label combination is a covered label combination includes: When the M-dimensional label combination is a subset of the test label set, the test sample is augmented to obtain augmented samples; The augmented sample is input into the multi-label image classifier to obtain a set of verification labels for classifying the augmented sample; The M-dimensional label combination is determined to be a subset of the verification label set, and the M-dimensional label combination is determined to be the covered label combination.

4. The method according to claim 2, characterized in that, The test samples include at least two, and the at least two test samples correspond to at least two test label sets; The step of determining that the M-dimensional label combination is a subset of the test label set and determining that the M-dimensional label combination is a covered label combination includes: If the M-dimensional label combination is a subset of at least one of the at least two test label sets, the M-dimensional label combination is determined to be the covered label combination.

5. The method according to claim 2, characterized in that, The method further includes: If the M-dimensional label combination is determined to be a non-subset of the test label set, the test sample corresponding to the test label set is added to the defect sample set.

6. The method according to any one of claims 1 to 5, characterized in that, Before performing classification tests on the multi-label image classifier based on the test image set corresponding to each M-dimensional label combination, the method further includes: For each M-dimensional label combination, the M leaf labels in the M-dimensional label combination are connected in a parallel relationship to obtain the search keywords; Based on the web search method, G candidate samples corresponding to the search keywords are obtained, where G is a positive integer; The test sample is selected from the G candidate samples and added to the test image set corresponding to the M-dimensional label combination.

7. The method according to claim 6, characterized in that, The step of selecting the test sample from the G candidate samples includes: The first K candidate samples out of the G candidate samples are determined as the test samples, where K is a positive integer less than or equal to G.

8. The method according to any one of claims 1 to 5, characterized in that, The method further includes: In response to the operation of setting the dimension of the tag combination, the dimension of the tag combination is set to M.

9. A classification testing device, characterized in that, The device includes: The acquisition module is used to acquire a set of leaf tags from the tag tree. The set of leaf tags includes at least two leaf tags. The leaf tags refer to the tags corresponding to the leaf nodes on the tag tree. The leaf nodes include the last-level nodes of each branch in the tag tree. The tag tree is a tree structure constructed based on the hierarchical relationship of tag categories. The generation module is used to traverse at least two leaf labels in the leaf label set and generate M-dimensional label combinations. Each M-dimensional label combination consists of M leaf labels, where M is a positive integer greater than 1. The testing module is used to perform classification tests on the multi-label image classifier based on the test image set corresponding to each M-dimensional label combination. The multi-label image classifier is used to classify images with multiple labels.

10. A computer device, characterized in that, The computer device includes a processor and a memory connected to the processor. The memory stores program instructions, and when the processor executes the program instructions, it implements the classification test method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program instructions that, when executed by a processor, implement the classification test method as described in any one of claims 1 to 8.

12. A computer program product, characterized in that, The computer program product includes computer instructions stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the classification test method as described in any one of claims 1 to 8.

Images