Efficient ground truth annotation

Abstract

A computer-implemented method for determining a set of target items to be annotated to train a machine learning application. The method includes providing a training dataset having a set of data samples and an autoencoder having a classifier. The autoencoder includes an embedding model that maps the set of data samples to a set of compressed feature vectors. The set of compressed feature vectors defines a compressed feature matrix. The method further includes: defining a graph associated with the compressed feature matrix; applying a clustering algorithm to identify node clusters in the graph and applying a centrality algorithm to identify the central nodes of the node clusters; retrieving node labels from annotator nodes for the central nodes; propagating the annotated node labels to other nodes in the graph; and performing training of the embedding model and the classifier using the annotated node labels and the propagated node labels.

Description

Technical Field

[0001] This invention relates to a computer-implemented method for determining a set of target items to be annotated in order to train a machine learning application.

[0002] The present invention also relates to corresponding systems and corresponding computer program products. Background Technology

[0003] Current supervised machine learning algorithms can be data-intensive, meaning they may require large amounts of annotated data to be successfully trained. This can be problematic for applying such algorithms in industrial environments because most industries do not maintain large annotated datasets, and annotating large datasets is typically quite costly due to the need for qualified subject matter experts. This annotated training data can also be referred to as ground truth.

[0004] Therefore, there is a need for methods that facilitate the efficient generation of such basic facts (especially from large datasets). Summary of the Invention

[0005] According to an embodiment, the present invention is implemented as a computer-implemented method for determining a set of target items to be annotated for training a machine learning application. The method includes providing a training dataset comprising a set of data samples. The set of data samples includes multiple data items, such as pixels. The method also includes providing an autoencoder. The autoencoder includes a classifier. The method includes performing initial training of an embedding model of the autoencoder, wherein the embedding model is configured to map the set of data samples to a set of compressed feature vectors. The feature vectors include feature elements. The set of compressed feature vectors defines a compressed feature matrix of the set of data samples. The method includes providing a definition of a graph associated with the compressed feature matrix, applying a clustering algorithm to identify one or more node clusters of the graph, and applying a centrality algorithm to identify one or more centroid nodes of the one or more node clusters. The method includes retrieving one or more node labels from an annotator for the one or more centroid nodes of the one or more node clusters. This creates annotated node labels.

[0006] This method involves propagating annotated node labels from one or more central nodes to other nodes in the graph, thereby creating propagated node labels and performing further training of the autoencoder. Further training of the autoencoder involves training an embedding model of the autoencoder and a classifier of the autoencoder with the annotated and propagated node labels. The classifier is configured to predict one or more node labels for elements of the compressed feature vector.

[0007] This specific approach allows for the generation of fundamental facts in an advanced manner.

[0008] Specifically, the method according to embodiments of the invention can identify one or more target items in a data sample set that should be annotated to effectively train machine learning applications, particularly with acceptable accuracy, recall, and / or precision. According to embodiments, one or more target items establish a restricted set of items for the data sample set. According to embodiments, restricted means that only a few or very few data items need to be annotated compared to the entire data item set. Therefore, the method according to embodiments of the invention can facilitate the automatic identification of the most relevant data items in the training dataset that should be annotated.

[0009] According to an embodiment, the method further includes repeating the application of clustering algorithms, centrality algorithms, propagation of annotation labels, and further training of the autoencoder in one or more iterative loops until convergence.

[0010] Repeating the above iterations until convergence can be used to verify that the cluster center nodes and their annotations have been correctly selected. According to an example, convergence will mean that the trained embedding model no longer changes according to one or more predefined criteria.

[0011] According to an embodiment, the method further includes outputting one or more node labels from the previous iteration before convergence as a set of target items to be annotated. The output labels establish a set of target items in the training dataset, which should be annotated by a human annotator to generate a favorable training dataset. In this regard, a favorable training dataset should be specifically understood as a training dataset that provides good training results for the corresponding machine learning application.

[0012] According to an embodiment, the method further includes repeatedly retrieving node labels from one or more annotators for one or more centroid nodes of one or more clusters. This repeated retrieval can, on the one hand, be used to confirm earlier annotations. Furthermore, obtaining additional feedback from annotators can be particularly useful if the determined node clusters have changed to a predefined extent during iterative cycles.

[0013] The definition of the graph associated with the compressed feature matrix facilitates the use of graph-based analysis to analyze the compressed feature matrix. According to an embodiment, this graph is defined by an adjacency matrix. The adjacency matrix is ​​defined as the product of the compressed feature matrix and its transpose.

[0014] According to embodiments, the application of clustering algorithms includes the application of graph-based clustering algorithms. Such graph-based clustering algorithms can be used to efficiently identify clusters of graphs. Several graph-based clustering algorithms known to those skilled in the art exist, and these algorithms can be selected based on the corresponding training dataset according to embodiments. Particularly suitable graph-based clustering algorithms may include k-spanning tree or minimum spanning tree algorithms, shared nearest neighbor algorithms, betweenness centrality-based algorithms, or spectral clustering algorithms.

[0015] According to embodiments, the application of a centrality algorithm includes applying a graph-based centrality algorithm. Such a graph-based centrality algorithm can be used to efficiently identify the central nodes of a graph cluster. Several graph-based centrality algorithms known to those skilled in the art exist, which can be selected according to the corresponding training dataset and the graph clustering, depending on the embodiment. A particularly suitable graph-based centrality algorithm includes approximating the product of the matrix exponent of the adjacency matrix and a random probe vector, calculating the diagonal of the adjacency matrix based on the product of the matrix exponent and the random probe vector, and calculating node centrality based on the calculated diagonal until a predetermined number of central nodes from one or more central nodes have been detected.

[0016] According to another embodiment, a system is provided for performing a computer-implemented method for executing a machine learning application. The system includes a processor and a computer-readable storage device. The system is configured to provide a training dataset comprising a set of data samples, provide an autoencoder containing a classifier, and perform initial training of an embedding model of the autoencoder. The embedding model is configured to map the set of data samples to a set of compressed feature vectors comprising feature elements. The set of compressed feature vectors defines a compressed feature matrix of the set of data samples. The system is also configured to provide a definition of a graph associated with the compressed feature matrix, apply a clustering algorithm to identify one or more node clusters of the graph, apply a centrality algorithm to identify one or more centroids of the one or more node clusters, and retrieve one or more node labels from annotators for the one or more centroids of the one or more node clusters. This creates annotated node labels. The system is also configured to propagate the annotated node labels of the one or more centroids to other nodes of the graph, thereby creating propagated node labels. The system is further configured to perform further training of the autoencoder. Further training includes training an embedding model of the autoencoder and a classifier of the autoencoder using annotated and propagated node labels, wherein the classifier is configured to predict one or more node labels of elements of the compressed feature vector.

[0017] According to another embodiment, a computer program product is provided for a system including a processor and a computer-readable storage medium to determine a set of target items to be annotated for training a machine learning application. The computer program product includes a computer-readable storage medium having program instructions embodied therein, which are executable by the system to cause the system to perform a method including providing a training dataset comprising a set of data samples. The set of data samples includes multiple data items, such as pixels. The method also includes providing an autoencoder. The autoencoder includes a classifier. The method includes performing initial training of an embedding model of the autoencoder, wherein the embedding model is configured to map the set of data samples to a set of compressed feature vectors. The feature vectors include feature elements. The set of compressed feature vectors defines a compressed feature matrix of the set of data samples. The method includes providing a definition of a graph associated with the compressed feature matrix, applying a clustering algorithm to identify one or more node clusters of the graph, and applying a centrality algorithm to identify one or more centroid nodes of the one or more node clusters. The method includes retrieving one or more node labels from an annotator for the one or more centroid nodes of the one or more node clusters. This creates annotated node labels. This method involves propagating annotated node labels from one or more central nodes to other nodes in the graph, thereby creating propagated node labels and performing further training of the autoencoder. Further training of the autoencoder involves training an embedding model of the autoencoder and a classifier of the autoencoder with the annotated and propagated node labels. The classifier is configured to predict one or more node labels for elements of the compressed feature vector.

[0018] According to another embodiment, a computer-implemented method for training a machine learning application is provided. The method includes providing a training dataset comprising a set of data samples and a set of target items to be annotated by an annotator; receiving the training dataset with the annotated target items; and training a cognitive model of the machine learning application using the training data including the annotated target items. Further embodiments relate to a training dataset and a cognitive model that can be obtained through the method for training a machine learning application.

[0019] Embodiments of the invention will now be described in more detail with reference to the accompanying drawings and by way of illustrative and non-limiting examples. Attached Figure Description

[0020] Figure 1a , Figure 1b and Figure 1c A computer-implemented method according to an embodiment of the present invention is shown;

[0021] Figure 2a It shows Figure 1a , 1bThe corresponding flowchart of the method shown in 1c;

[0022] Figure 2b A computer-based method for training machine learning applications using a training dataset is shown.

[0023] Figure 2c A computer-based method for performing machine learning applications is shown;

[0024] Figure 3 It shows what can be used to perform, such as Figure 1a , 1b 1c and Figure 2a , 2b A schematic block diagram of the computing system for the computer-implemented method shown in Figure 2c;

[0025] Figure 4 The mapping performed by the autoencoder during training is shown;

[0026] Figure 5 An exemplary block diagram of an automatic encoder according to an embodiment of the present invention is shown;

[0027] Figure 6 A diagram illustrating a computer-implemented method according to an embodiment of the present invention will be applied thereto; and

[0028] Figure 7 A more detailed block diagram of a server according to an embodiment of the present invention is shown. Detailed Implementation

[0029] Referring to Figures 1-7, some general terms from embodiments of the present invention are described.

[0030] The term "basic facts" can generally refer to information provided through direct observation (i.e., empirical evidence) rather than information provided through inference.

[0031] Embodiments of the present invention provide a computer-implemented method for efficiently generating foundational facts and training data for machine learning algorithms and applications.

[0032] Graph theory is the study of graphs, which are mathematical structures used to model pairwise relationships between objects. In this context, a graph consists of vertices or nodes and lines connecting them, called edges. Graphs are widely used in applications that model the dynamics of many types of relationships and processes in physical, biological, social, and information systems. Therefore, many practical problems in modern technology, science, and business applications are often represented by graphs.

[0033] Node centrality is a widely used metric for determining the relative importance of a node within a complete network or graph. Node centrality can be used to determine which nodes are important in complex networks, for example, to understand influencers or identify hotspot links. For instance, node centrality is often used to determine how influential someone is within a social network, or in the theory of spatial grammar, how important a room is within a building or how well a road is used within a city network.

[0034] Figure 1a , 1b Figures 1 and 1c illustrate a computer-implemented method according to an embodiment of the present invention.

[0035] Figure 2a It shows Figure 1a , 1b The corresponding flowchart of the method shown in 1c. Figure 3 It shows what can be used to perform, such as Figure 1a , 1b Schematic block diagrams of the computing system for the computer-implemented method shown in 1c and 2a.

[0036] First refer to Figure 3 It illustrates a computing system 300, which includes a server 310 configured to run a machine learning application (MLAP) 311, a machine learning training program (MLTP) 312 for training the application, and a target item determination program (TIDP) 313 for determining target items of a large dataset that should be annotated to efficiently generate a training dataset for the machine learning application. The machine learning application 311 may in particular be a deep learning application.

[0037] Server 310 is coupled to database 320. Database 320 may include storage 321 for storing one or more datasets, particularly large datasets. The datasets stored in storage 321 may, in particular, be datasets that have not yet been annotated. Therefore, the datasets stored in storage 321 may also be represented as initial datasets, raw datasets, or initial training datasets. Each dataset may include multiple data samples. The multiple data samples may include, for example, various types of images (e.g., {cat, dog}, {line graph, scatter plot, histogram, geological map, or text (sentence, paragraph, full text)}). Each of the multiple data samples may include multiple data items, such as multiple pixels.

[0038] Database 320 also includes storage 322 for storing the training dataset. The training dataset can be derived from the initial or original dataset by annotations of one or more target items from multiple data samples of the initial or original dataset.

[0039] System 300 also includes a scanning machine 330. The scanning machine 330 can provide scanning and printing functions. Therefore, the scanning machine 330 can also be referred to as a scanner / printer. System 300 also includes user equipment 340. User equipment 340 may include a display and user input devices, such as a keyboard, to provide a user interface to users of system 300. User equipment 340 may specifically include an application programming interface (API) 341, which is adapted to provide an interface to programs 311, 312, and 313 running on server 310.

[0040] Server 310, scanning machine 330, and user equipment 340 are coupled to network 350 and can communicate with each other via network 350. Network 350 can be implemented as a local area network (LAN), a general wide area network (WAN), and / or a public network (e.g., the Internet).

[0041] Now for reference Figure 1a and 2a The method begins at block 201. In block 201, an initial training dataset is provided. The initial training dataset can typically be any dataset that should be annotated to provide an annotated training dataset as the basis for the machine learning application. The initial training dataset can be stored in memory 321 of database 320, and it can be received, for example, via network 350.

[0042] Furthermore, in block 202, an autoencoder 100 is provided, and in block 203, a definition of a graph of the compressed feature matrix to be provided by the autoencoder 100 is provided.

[0043] The autoencoder 100 includes an embedding model and a classifier. The autoencoder 100 can be implemented as a neural network configured to learn efficient encodings, or in other words, efficient representations, of data samples from the training dataset. The autoencoder 100 may include several hidden layers.

[0044] In block 204, the autoencoder 100 performs initial training of its embedding model. Initial training is performed using an initial dataset comprising multiple data samples, such as a dataset stored in memory 321 of database 320. The embedding model of the autoencoder 100 is configured to map this set of data samples to a set of compressed feature vectors comprising feature elements. The embedding model of the autoencoder 100 in block 204 is an unsupervised embedding model.

[0045] More specifically, now refer to Figure 1b The autoencoder 100 maps the input vector 110 of the data samples to a compressed feature vector 111. This mapping can also be represented as encoding. The compressed feature vector establishes the code for the input vector 110.

[0046] Input vector 110 comprises multiple vector elements, such as multiple pixels of an image. Together with the encoding side, the autoencoder is configured to learn the decoding side to decode the encoding back to an output vector 112 corresponding to the data sample*. More specifically, the purpose of autoencoder 100 is to decode the encoded / compressed feature vector such that the output vector 112 and the corresponding data sample* are as close as possible to the input vector 110 and the input data sample. The size of the compressed feature vector 111, such as the number of elements in the compressed feature vector 111, defines the encoding dimension of autoencoder 100.

[0047] The autoencoder 100 is configured to map each data sample and its corresponding input vector into a compressed feature vector. This set of compressed feature vectors defines the compressed feature matrix M of the data sample set. C Compressed feature matrix M C The size of M is "encoding dimension * number of data samples", or in other words, it is constructed by concatenating all the compressed feature vectors of the data sample set. As a simple example, if the dataset includes 5 data samples mapped to compressed feature vectors V1, V2, V3, V4, and V5, then the compressed feature matrix M... C as follows:

[0048] M C =(V1V2V3V4V5)

[0049] Or more generally, if the dataset includes N data samples, then the compressed feature matrix M C Includes N feature vectors V1, ... V N The graph defined in block 203 can be defined by adjacency matrix A according to an embodiment. Adjacency matrix A can be defined by compressed feature matrix M. C and the transpose M of the compressed feature matrix C T The product is defined as follows:

[0050] A = M C *M C T

[0051] Example diagram Figure 1c As shown in Figure 120. After the initial training of the embedding model, in block 205, the method applies a clustering algorithm to identify one or more clusters of the graph.

[0052] The clustering algorithm according to embodiments of the present invention aims to identify one or more clusters in a graph defined by an adjacency matrix A. Typically, a clustering algorithm performs the task of grouping a set of objects in such a way that objects in the same group (cluster) are more similar to each other than objects in other groups (other clusters) according to some predetermined criteria.

[0053] Multiple clustering algorithms are known to those skilled in the art, and can be selected according to the corresponding dataset based on embodiments. According to embodiments, clustering algorithms particularly suitable for clustering graphs can be used. Such graph-based clustering algorithms may include K-spanning tree or minimum spanning tree algorithms, shared nearest neighbor algorithms, betweenness centrality-based algorithms, or spectral clustering algorithms.

[0054] refer to Figure 1c The diagram shows the first cluster 130 indicated by black nodes and the second cluster 140 indicated by gray nodes.

[0055] Next, in box 206, a computer-implemented centrality algorithm is used to identify clusters of one or more nodes (e.g., Figure 1c One or more central nodes of clusters 130 and 140. (Refer to...) Figure 1c The center nodes 130c of cluster 130 and 140c of cluster 140 have been identified.

[0056] In graph analysis, node centrality is a widely used metric to determine the relative importance of a node within a complete network or graph. Node centrality can be used to identify which nodes are important in complex networks to understand influencers or pinpoint hotspot links. For example, node centrality is often used to determine how influential someone is within a social network, or in the theory of spatial grammar, how important a room is within a building or how well a road is used within a city network.

[0057] Multiple centrality algorithms are known to those skilled in the art, and can be selected according to the corresponding dataset based on embodiments. According to embodiments, centrality algorithms particularly suitable for graphs can be used.

[0058] According to an embodiment, a graph-based clustering algorithm can be used, which includes approximating the product of a random probe vector and a matrix exponent of an adjacency matrix defining the graph. The graph-based clustering algorithm also includes calculating the diagonal of the adjacency matrix based on the product of the matrix exponent and the random probe vector, and calculating node centrality based on the calculated diagonal until a predefined number of central nodes in one or more central nodes have been detected. This method is described in detail in US 2014 / 0351307A1, the entire contents of which are incorporated herein by reference.

[0059] An alternative method for calculating node centrality according to a further embodiment may sample the nodes of the graph in an attempt to reduce the number of nodes and thus reduce computational complexity.

[0060] According to embodiments, multiple metrics can be used to define centrality, and multiple corresponding centrality algorithms can be used. Such metrics include, for example, graph degree centrality, betweenness centrality, compact centrality, and eigenvector centrality.

[0061] Then, in block 207, a human annotator is asked to annotate center nodes 130C and 140C. For this annotation, the corresponding portion of the original data sample of the original image is also shown to the human annotator. The human annotator assigns labels as annotations to center nodes 130C and 140C. Typically, one or more labels can be associated with any given node of the graph, where each such label indicates a certain characteristic of the item corresponding to that node. For example, the value of a particular label can indicate the type, category, attribute, feature, etc., of an item in a system represented by the graph. The label can be, for example, the content of the corresponding portion of an image. As an example, the annotator can label center node 130C as "dog" and center node 140C as "cat". Computer-implemented programs, for example, via... Figure 3 The user equipment 340 shown receives these tags as annotations. As a result, annotation node tags have been created. Figure 1c In the example, the annotation node label of node 130c, which may represent, for example, the label "cat", is shown by a solid ring, while the annotation node label of node 140c, which may represent, for example, the label "dog", is shown by a dashed ring.

[0062] In box 208, the program performs label propagation of the annotated node labels to other nodes in the graph (more specifically, to other nodes in the identified clusters 130 and 140). This has created, as shown in the diagram. Figure 1c The propagation node labels are indicated by solid or dashed loops.

[0063] Label propagation plays a crucial role in many applications of network science, including link prediction, recommender systems, community detection, seed set expansion, and semi-supervised learning. According to embodiments, multiple label propagation algorithms can be used. According to embodiments, label propagation can be performed by applying a propagation function to the labeled node labels. The propagation function can typically be a function defining a matrix X of the graph, particularly a function of the graph's adjacency matrix A, or a matrix derived from the adjacency matrix, such as the corresponding Laplacian matrix or a function thereof.

[0064] Many different propagation functions are known. Specific examples of known propagation functions that can be used according to an embodiment include the exponential kernel (ep). αA ) and von Neumann nucleus (I-αA) -1A variant of , where I is the identity matrix. This known propagation function is based on the assumption of homogeneity, which is the concept that elements in a graph are connected to other elements that share the same properties, and by extension, similar elements are linked by (multiple) short paths.

[0065] According to the embodiment, the propagation function can be defined as Chebyshev extension f(X) = Σ j＝0 c j P j (X), where P j It is the j-th Chebyshev polynomial.

[0066] Then, in block 209, the computer-implemented method performs further training of the autoencoder 100. This further training utilizes annotated and propagated node labels. More specifically, the autoencoder 100 also includes a classifier 102 configured to predict the node labels of the elements of the compressed feature vector. The elements of the compressed feature vector correspond to the nodes of Figure 120. Further training involves training the embedding model of the autoencoder 100 and the classifier 102 of the autoencoder 100 using the annotated and propagated node labels. This will be explained in more detail below.

[0067] Then, in one or more iteration blocks S210, the computer-implemented iteration of the aforementioned blocks, more specifically, iterating over blocks 205 for applying the clustering algorithm, 206 for applying the centrality algorithm, 207 for retrieving annotations for the central nodes, 208 for propagating the annotation labels, and 209 for performing further training of the autoencoder 100, until convergence. In this respect, convergence will mean that clusters 130 and 140 have been sufficiently separated from each other according to a predetermined convergence criterion. More specifically, before each iteration, block 210 checks whether the predetermined convergence criterion has been met.

[0068] It should be noted that, according to the embodiment, the iterative loop may not include block 207 for retrieving annotations for the centroid nodes. This may be optionally performed or repeated as needed during the iterative loop. In particular, if the centroid nodes of clusters already identified in the first or subsequent steps of applying the clustering and centrality algorithms have not changed, further manual annotation of these centroid nodes / clusters may not be necessary.

[0069] If the predefined convergence criteria have been met, the computer-implemented method outputs, in block 211, a set of node labels that have already been annotated in the previous iteration loop. Figure 1c In the example shown, these will simply be the two node labels corresponding to the dashed and solid loops of the labels "cat" and "dog".

[0070] The node labels output establishes the set of target items that should be annotated in the training dataset for successful training of the machine learning algorithm. The training dataset can then be submitted to one or more human annotators to perform a (limited) annotation task to label only the set of target items identified in the training dataset. In the example above, this would mean that the human annotators only need to label "cat" and "dog" in the data samples of the training dataset.

[0071] The generated training dataset, which includes annotated target items, was subsequently used to build, as follows: Figure 2b The following are the fundamental facts used to train machine learning applications.

[0072] At box 221, a training dataset is received, for example, by machine learning training program 312. The training dataset includes annotations of the set of target items as described above. At box 222, machine learning training program 312 uses the training dataset to train the cognitive model of machine learning application 311.

[0073] In block 223, machine learning training program 312 provides a trained cognitive model as output.

[0074] Now for reference Figure 2c The document provides a computer-implemented method 230 for performing machine learning applications. This method can be, for example, by... Figure 3 The machine learning application 311 shown is used to perform this.

[0075] At box 231, machine learning application 311 receives, for example, a trained cognitive model from machine learning training program 312.

[0076] At block 232, machine learning application 311 uses a cognitive model to perform machine learning applications.

[0077] In block 233, machine learning application 311 provides the output of the machine learning application.

[0078] like Figure 2c The computer-implemented method 230 shown establishes the inference phase of machine learning applications and their underlying algorithms.

[0079] Figure 4 and 5 The function and structure of the automatic encoder 100 according to an embodiment of the present invention are shown in more detail. In this respect, Figure 4 The mapping performed by the autoencoder 100 during training step 209 is shown in more detail. Figure 5 An exemplary block diagram of an automatic encoder 100 is shown.

[0080] refer to Figure 4The autoencoder 100 performs a mapping from the input vector 110 of the data sample to the corresponding compressed feature vector 111. As previously described, the autoencoder is configured to learn the decoding side to decode the encoding back to the corresponding output vector 112. Furthermore, the autoencoder 100 is also configured to consider the labels already assigned during the current iteration loop according to annotations or propagation. More specifically, the autoencoder 100 also performs a label mapping from the compressed feature vector to the labels already assigned (according to annotations and / or propagation) in the current iteration loop in block 209. In other words, the autoencoder 100 predicts the node labels for each element of the compressed feature vector 111. The prediction of node labels is performed using classifier 102. Since the autoencoder 100 also considers class labels in block 209, it can be represented as a semantic autoencoder. According to an embodiment, the semantic autoencoder can be viewed as an autoencoder "supervised" by labels predicted from the graph.

[0081] Therefore, now refer to Figure 5 According to the embodiments, the autoencoder 100 can be described as a conventional autoencoder 101 that encodes an input vector into a compressed feature vector and decodes it back into an output vector. Furthermore, the autoencoder 100 includes a classifier 102, which also predicts the label of the compressed feature vector through its embedding model. As a result of training block 209, an updated embedding model has been computed, which will be used in the next iteration loop.

[0082] Figure 6 Figure 600 illustrates a computer-implemented method according to an embodiment of the present invention that will be applied thereto. Figure 600 can also be defined by an adjacency matrix, which is composed of a compressed feature matrix M. C and the transpose M of the compressed feature matrix C T Defined as the product of, as shown in the reference above. Figure 1c The method according to an embodiment of the invention first applies a clustering algorithm as described in reference box 205 to identify one or more clusters of a graph. In this example, a first cluster 610 indicated by black nodes and a second cluster 620 indicated by gray nodes are shown. Then, as described in reference box 206, a computer-implemented centrality algorithm is used to identify one or more centroid nodes of the one or more node clusters (i.e., clusters 610 and 620). Figure 6 In the process, the center node 610c of cluster 610 and the center node 620c of cluster 620 have been identified.

[0083] The method according to this embodiment now includes identifying one or more boundary nodes. A boundary node can be defined as a node located at the boundary between two clusters, i.e., a node located at the boundary between clusters 610 and 620. Boundary nodes can be calculated or determined using various methods and algorithms known to those skilled in the art, according to predetermined criteria.

[0084] exist Figure 6 In this example, it is assumed that the algorithm has calculated the node 610b of cluster 610 as the boundary node, and the two nodes 620b of cluster 620 as the boundary nodes.

[0085] According to embodiments, the method may further include retrieving node labels for one or more boundary nodes from one or more annotators. Figure 6 In the example, the annotation node labels for a boundary node 610b, such as the label "cat", can be shown by solid rings, and the annotation node labels for a boundary node 620b, such as the label "dog", can be shown by dashed rings. In this example, these correspond to the labels of the corresponding center nodes 610c and 620c of clusters 610 and 620, respectively.

[0086] According to a further embodiment, the method may include identifying one or more farthest nodes from one or more clusters. The farthest node may be defined as the node in a given cluster that is farthest from the center node of that cluster. The farthest node can be calculated or determined by predetermined criteria using various methods and algorithms known to those skilled in the art.

[0087] exist Figure 6 In this example, it is assumed that the algorithm has calculated node 610f of cluster 610 as the farthest node and node 620f as the farthest node of cluster 620.

[0088] According to embodiments, the method may further include retrieving node labels for one or more farthest nodes from one or more annotators. Figure 6 In the example, the annotation node label representing, for example, the label "cat" at the furthest node 610f is shown by a solid ring, and the annotation node label representing, for example, the label "dog" at the furthest node 620f is shown by a dashed ring. In this example, these correspond to the labels of the corresponding center nodes 610c and 620c of clusters 610 and 620, respectively.

[0089] Now for reference Figure 7A more detailed embodiment of server 310 is shown. Server 310 can operate with many other computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations suitable for use with server 310 include, but are not limited to, personal computer systems, server computer systems, thin clients, fat clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the aforementioned systems or devices. In particular, server 310 operates with scanning machine 330 of system 300 and user equipment 340 of system 300, such as... Figure 3 As shown.

[0090] Server 310 can be described in the general context of executable instructions in a computer system, such as program modules executed by the computer system. Typically, program modules may include routines, programs, objects, components, logic, data structures, etc., that perform a specific task or implement a specific abstract data type. Server 310 is shown as a general-purpose computing device. Components of server 310 may include, but are not limited to, one or more processors or processing units 716, system memory 728, and a bus 718 that couples various system components, including system memory 728, to processor 716.

[0091] Bus 718 represents one or more of several types of bus architectures, including memory buses or memory controllers, peripheral buses, accelerated graphics ports, and processor or local buses using any of the various bus architectures. By way of example and not limitation, these architectures include the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MCA) bus, the Enhanced ISA (EISA) bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0092] Server 310 typically includes various computer system readable media. Such media can be any available media accessible by server 310, and it includes volatile and non-volatile media, removable and non-removable media.

[0093] System memory 728 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 730 and / or cache memory 732. Server 310 may also include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 734 may be provided for reading from and writing to non-removable, non-volatile magnetic media (not shown, and generally referred to as "hard disk drives"). Although not shown, disk drives for reading from and writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks such as CD-ROMs, DVD-ROMs, or other optical media may be provided. In such instances, each may be connected to bus 718 via one or more data media interfaces. As will be further described and illustrated below, memory 728 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of embodiments of the invention.

[0094] As an example and not a limitation, a program / utility 740 having a set (at least one) of program modules 742, along with an operating system, one or more applications, other program modules, and program data, may be stored in memory 728. Each of the operating system, one or more applications, other program modules, and program data, or some combination thereof, may include an implementation of a networking environment. Program modules 742 generally perform the functions and / or methods of embodiments of the invention described herein. Program module 742 may specifically perform one or more steps of a computer implementation of a method for determining a set of target items to be annotated for training a machine learning application according to embodiments of the invention, for example, one or more steps of the method described with reference to FIG. S, 2a, 2b, and / or 2c. Program module 742 may specifically include as described with reference to FIG. S, 2a, 2b, and / or 2c. Figure 3 as well as Figure 2a , 2b The corresponding flowcharts in 2c describe procedures 311, 312, and 313.

[0095] Server 310 can also communicate with one or more external devices 715, such as a keyboard, indicating device, display 724, etc.; one or more devices that enable a user to interact with server 310; and / or any device that enables server 310 to communicate with one or more other computing devices (e.g., network interface card, modem, etc.). This communication can occur via input / output (I / O) interface 722. Furthermore, server 310 can communicate with one or more networks, such as a local area network (LAN), a general area network (WAN), and / or a public network (e.g., the Internet), via network adapter 720. As depicted, network adapter 720 communicates with other components of server 310 via bus 718. It should be understood that, although not shown, other hardware and / or software components can be used in conjunction with server 310. Examples include, but are not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archive storage systems.

[0096] The present invention may be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor / processing unit of server 310 and / or user equipment 340 to execute embodiments of the present invention.

[0097] Computer-readable storage media can be tangible devices capable of retaining and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable optical disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices such as punch cards or recessed structures with instructions recorded thereon, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0098] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device, or via a network, such as the Internet, a local area network (LAN), a wide area network (WAN), and / or a wireless network, to an external computer or external storage device. The network may include copper cables, optical fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the respective computing / processing device.

[0099] Computer-readable program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​(e.g., Smalltalk, C++, etc.) and conventional procedural programming languages ​​(e.g., the "C" programming language or similar programming languages). The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, to perform embodiments of the invention, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) may execute computer-readable program instructions to personalize the electronic circuitry by utilizing state information from the computer-readable program instructions.

[0100] Embodiments of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0101] These computer-readable program instructions may be provided to a processor of a programmable data processing apparatus to produce machine, such that the instructions, which execute via a processor of a computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and / or other devices to operate in a particular manner, such that the computer-readable storage medium in which the instructions are stored includes an article of writing comprising instructions for implementing embodiments of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0102] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer-implemented process, such that the instructions, which execute on the computer, other programmable apparatus or other device, perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0103] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions comprising one or more executable instructions for implementing a specified logical function. In some alternative embodiments, the functions mentioned in the blocks may occur in a non-linear order as shown in the figures. For example, two blocks shown consecutively may actually be executed substantially simultaneously, or these blocks may sometimes be executed in reverse order, depending on the functions involved. It will also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0104] Various embodiments of the invention have been described for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terminology used herein has been chosen to best explain the principles of the embodiments, their practical application, or improvements to existing technologies in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A computer-implemented method, comprising: A training dataset is provided, comprising a collection of data samples, wherein the training dataset includes multiple data samples, the data samples including images; An autoencoder is provided, the autoencoder including a classifier; The initial training of the embedding model of the autoencoder is performed. The embedding model is configured to map a set of data samples to a set of compressed feature vectors that include feature elements. The set of compressed feature vectors defines the compressed feature matrix of the set of data samples. Provide a definition of the graph associated with the compressed feature matrix; Clustering algorithms are applied to identify clusters of one or more nodes in the graph; A centrality algorithm is applied to identify one or more central nodes of the clusters of the one or more nodes; Annotated node labels are created by retrieving one or more node labels from the annotator for the clustering of the one or more central nodes of the one or more nodes; Propagate the annotation node labels of the one or more central nodes to other nodes in the graph, thereby creating propagated node labels; and Perform further training on the autoencoder, the further training including The embedding model of the autoencoder and the classifier of the autoencoder are trained using annotated node labels and propagated node labels, wherein the classifier is configured to predict one or more node labels of the elements of the compressed feature vector.

2. The method according to claim 1, further comprising: In one or more iterative loops, the clustering algorithm is applied repeatedly, the centrality algorithm is applied, annotation labels are propagated, and further training of the autoencoder is performed until convergence.

3. The method according to claim 2 further includes outputting one or more node labels from the previous iteration before convergence as a set of target items to be annotated.

4. The method according to claim 1, further comprising: Repeatedly retrieve node labels for the one or more central nodes used for the clustering of the one or more nodes from one or more commenters.

5. The method of claim 1, wherein the graph is defined by an adjacency matrix, the adjacency matrix being the product of the compressed feature matrix and the transpose of the compressed feature matrix.

6. The method of claim 1, wherein applying the clustering algorithm includes applying a graph-based clustering algorithm.

7. The method of claim 1, wherein applying the centrality algorithm includes applying a graph-based centrality algorithm.

8. The method of claim 6, wherein the graph-based clustering algorithm is selected from the group consisting of: K-spanning tree or minimum spanning tree algorithm; Shared nearest neighbor algorithm; Algorithms based on betweenness centrality; and Spectral clustering algorithm.

9. The method of claim 7, wherein the graph-based centrality algorithm comprises: The graph is approximated by the product of the matrix exponent of the adjacency matrix and the random probe vector, wherein the graph is defined by the adjacency matrix, which is the product of the compressed feature matrix and the transpose of the compressed feature matrix; The diagonal of the adjacency matrix is ​​calculated based on the product of the matrix exponent and the random probe vector; as well as The node centrality is calculated based on the diagonal, until a predefined number of central nodes are detected among the one or more central nodes.

10. The method according to claim 1, further comprising: Identify one or more boundary nodes, which are defined as nodes located at the boundary between two clusters in the node cluster.

11. The method according to claim 10, further comprising: Retrieve node labels for the one or more boundary nodes from one or more commenters.

12. The method according to claim 1, further comprising: Identify one or more farthest nodes in the cluster of the one or more nodes.

13. The method according to claim 12, further comprising: Retrieve node labels from one or more annotators for one or more farthest nodes used in the clustering of the one or more nodes.

14. The method of claim 5, wherein the step of propagating the annotation node labels includes applying a propagation function to the annotation node labels, wherein the propagation function is a function of the adjacency matrix.

15. The method of claim 14, wherein the propagation function is defined as Chebyshev extension f(X) = Σ j=0 c j P j (X), where P j It is the j-th Chebyshev polynomial.

16. The method according to claim 1, further comprising: Use the training dataset to train a cognitive model for a machine learning application.

17. A system comprising one or more processors for executing computer-readable instructions, the computer-readable instructions controlling the one or more processors to perform operations, the operations including: A training dataset is provided, comprising a collection of data samples, wherein the training dataset includes multiple data samples, the data samples including images; An autoencoder is provided, the autoencoder including a classifier; The initial training of the embedding model of the autoencoder is performed. The embedding model is configured to map a set of data samples to a set of compressed feature vectors that include feature elements. The set of compressed feature vectors defines the compressed feature matrix of the set of data samples. Provide a definition of the graph associated with the compressed feature matrix; Clustering algorithms are applied to identify clusters of one or more nodes in the graph; A centrality algorithm is applied to identify one or more central nodes of the clusters of the one or more nodes; Annotated node labels are created by retrieving one or more node labels from the annotator for the clustering of the one or more central nodes of the one or more nodes; Propagate the annotation node labels of the one or more central nodes to other nodes in the graph, thereby creating propagated node labels; and Perform further training on the autoencoder, the further training including: The embedding model of the autoencoder and the classifier of the autoencoder are trained using annotated node labels and propagated node labels, wherein the classifier is configured to predict one or more node labels of the elements of the compressed feature vector.

18. The system of claim 17, further comprising: In one or more iterative loops, the application of the clustering algorithm, the application of the centrality algorithm, the propagation of annotation labels, and the execution of further training of the autoencoder are repeated until convergence.

19. The system of claim 17, further comprising: Output one or more node labels from the previous iteration before convergence as the set of target items to be annotated.

20. The system of claim 17, further configured to Repeat the step of retrieving node labels from one or more annotators for the clustering of the one or more central nodes.

21. A computer program product having program instructions embodied therein, the program instructions being executable by a processor to cause the processor to perform operations, the operations including: A training dataset is provided, comprising a collection of data samples, wherein the training dataset includes multiple data samples, the data samples including images; An autoencoder is provided, the autoencoder including a classifier; The initial training of the embedding model of the autoencoder is performed. The embedding model is configured to map a set of data samples to a set of compressed feature vectors that include feature elements. The set of compressed feature vectors defines the compressed feature matrix of the set of data samples. Provide a definition of the graph associated with the compressed feature matrix; Clustering algorithms are applied to identify clusters of one or more nodes in the graph; A centrality algorithm is applied to identify one or more central nodes of the clusters of the one or more nodes; Annotated node labels are created by retrieving one or more node labels from the annotator for the clustering of the one or more central nodes of the one or more nodes; Propagate the annotation node labels of the one or more central nodes to other nodes in the graph, thereby creating propagated node labels; and Perform further training on the autoencoder, the further training including: The embedding model of the autoencoder and the classifier of the autoencoder are trained using annotated node labels and propagated node labels, wherein the classifier is configured to predict one or more node labels of the elements of the compressed feature vector.

22. The computer program product according to claim 21, further comprising: In one or more iterative loops, the application of the clustering algorithm, the application of the centrality algorithm, the propagation of annotation labels, and the execution of further training of the autoencoder are repeated until convergence.

Images