Apparatus and computer-implemented method for machine learning

Abstract

Apparatus and computer-implemented method for machine learning, where the method includes providing an input, where the input includes a first node, a first name, a second node, and a second name, where the input includes an edge between the first node and the second node; determining an expression that associates the edge with the first node and the second node in accordance with the input using a first model; determining an expression associating the first node with the first name and an expression associating the second node with the second name from the input using a second model, wherein the method includes providing a question including the first name and the second name; an expression including the first name and the second name is determined according to the question using a third model, wherein the method includes determining an answer to the question according to the expression.

Description

Technical Field

[0001] The present invention relates to apparatus and computer-implemented methods for machine learning. Background Art

[0002] Visual or symbolic representations of graph-based structures are a popular form of presenting information and are ubiquitous in real life and on the Internet. Summary of the invention

[0003] Apparatus and computer-implemented methods for machine learning provide visual graph question answering, where the input is a digital image or symbolic description of a graph along with a natural language question, and the task is to answer the question correctly given the input.

[0004] A computer-implemented method for machine learning comprises providing an input, in particular, a digital image or a symbolic description of a graph, wherein the input comprises a first node, a first name, a second node, and a second name, wherein the input comprises an edge between the first node and the second node, wherein the method comprises: determining, using a first model according to the input, an expression associating the edge with the first node and the second node, wherein the first model is configured to recognize that the edge is between the first node and the second node, wherein the first model is configured to determine, according to the input, an expression associating the edge with the first node and the second node; determining, using a second model according to the input, an expression associating the first node with the first name and an expression associating the second node with the second name, wherein the second model The method comprises providing a question including the first name and the second name; determining an expression including the first name and the second name according to the question using a third model, wherein the third model is configured to identify the first name and the second name in the question and determine an expression including the first name and the second name according to the question, and wherein the method comprises determining an answer to the question according to the expression. The nodes and the edges represent portions of a graph. The graph is determined by a computer-implemented method for machine learning. The answer to the question is determined by a computer-implemented method for machine learning based on the content of the graph.

[0005] According to an example, the method comprises receiving the input from a sensor or at an interface, wherein the input represents a circuit, wherein the edge represents an electrical connection between a first electrical component represented by the first node and a second electrical component represented by the second node, wherein the question is related to a state of the circuit, the first electrical component, the second electrical component, or the electrical connection, wherein the answer comprises the state, and wherein the method comprises outputting the state or automatically approving or tidying up the circuit according to the state. The computer-implemented method for machine learning may be part of an inspection for manufacturing electrical connections.

[0006] According to an example, the method comprises receiving the input from a sensor or at an interface, wherein the input represents a map, wherein the edge represents a connection between a first waypoint represented by the first node and a second waypoint represented by the second node, wherein the question is related to the connection or a connection in the map including the following items: the first waypoint, the second waypoint and / or the connection between the first waypoint and the second waypoint, wherein the method comprises selecting a connection for moving a technical system according to the answer and moving the technical system via the connection. The map can be a subway map or a map of facilities. The technical system can be a physical system, such as a robot, in particular a vehicle or a manufacturing machine. A computer-implemented method for machine learning can help automatically move the technical system from one waypoint to another waypoint in the map or automatically move part of the waypoint from one waypoint to another waypoint in the map.

[0007] According to an example, the first model is configured for semantic segmentation of the digital image and is configured to identify that the edge is between the first node and the second node according to the result of the semantic segmentation. The semantic segmentation assigns classes to pixels of the digital image, such as nodes, edges or other. The portion of the graph can be determined by the computer-implemented method based on the coordinates and classes of the pixels of the digital image.

[0008] According to an example, the first model is configured to determine a matrix comprising rows for the first node and columns for the second node and indicating the existence of an edge between the first node and the second node, the determination being performed by entries in elements of the matrix that are in the rows and the columns, wherein the method comprises determining the matrix and determining an expression associating the edge with the first node and the second node according to the matrix. The entries in the matrix are determined by the computer-implemented method, in particular according to the coordinates and the class of the pixels of the digital image.

[0009] According to an example, the second model is configured for optical character recognition in the digital image, and is configured to determine the first name and the second name according to the result of the optical character recognition. The first name and the second name are determined by the computer-implemented method based on the digital image.

[0010] According to an example, the first model is configured to determine the position of the first node in the digital image, in particular, pixel coordinates, wherein the second model is configured to determine the position of the first name in the digital image, in particular, pixel coordinates, wherein the method includes determining the position and determining an expression associating the first node with the first name according to the position.

[0011] According to an example, the third model is configured to determine a first name in the question according to a position of a first variable in a template for the question and to determine a second name in the question according to a position of a second variable in the template for the question, wherein the method includes determining an expression including the first name and the second name according to the template. The question is an instance of a template in which names are substituted for variables.

[0012] According to an example, the method includes determining answer set programming facts according to the expression and determining the answer according to the answer set programming facts. The expression presents a combinatorial problem. The computer-implemented method can use answer set programming to reason about the problem and determine the answer that solves the combinatorial problem.

[0013] The device for machine learning comprises at least one processor and at least one memory, wherein the at least one processor is configured to execute instructions that, when executed by the at least one processor, cause the device to perform the method, wherein the at least one memory is configured to store the instructions.

[0014] A computer program comprising computer executable instructions which, when executed by a computer, causes the computer to perform the method has the advantages provided by the method. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Further embodiments are derived from the following description and the accompanying drawings. In the drawings: Figure 1 Schematically depicting a device used for machine learning, Figure 2 Depicted is an exemplary input, Figure 3 A functional representation of an exemplary problem is depicted, Figure 4 Describes the architecture used for machine learning, Figure 5A flowchart having steps for a computer-implemented method for machine learning is depicted. DETAILED DESCRIPTION

[0016] Figure 1 Schematically depicted is a device 100 for machine learning. The device 100 comprises at least one processor 102 and at least one memory 104.

[0017] The device 100 may include an interface 106. The interface 106 may be configured to receive an input 108. The input 108 may be a digital image or a symbolic depiction of a diagram.

[0018] The device 100 is configured to answer questions. The questions may be natural language questions.

[0019] Input 106 may be configured to receive input 108 from sensor 110. Sensor 110 may be configured to capture input 108. Sensor 110 may be a camera.

[0020] The device 100 may be configured to actuate a technical system 112. The technical system 112 may be a robot, in particular a vehicle or a manufacturing machine. The device 100 may be configured to output an output 114 for actuating the technical system 112. The interface 106 may be configured to output the output 114.

[0021] Input 108 may represent a circuit.

[0022] A graph may represent a circuit. A graph may include nodes representing electrical components. A graph may include edges representing electrical connections between electrical components. An edge of a graph may represent an electrical connection between an electrical component represented by a first node of the graph and a second electrical component represented by a second node of the graph.

[0023] The problem may be related to the state of the circuit. The problem may be related to the state of the first electrical component. The problem may be related to the state of the second electrical component. The problem may be related to the state of the electrical connection.

[0024] The answer may include the status.

[0025] Device 100 may be configured to output a status. Output 114 may include a status.

[0026] The device 100 or technical system 114 may be configured to automatically approve or trim the circuit according to the status. The device 100 may be configured to actuate the manufacturing machine to trim the circuit if the status indicates a faulty circuit, electrical connection, or electrical component. The device 100 may be configured to actuate the manufacturing machine to approve the circuit if the status indicates that the circuit, electrical connection, and / or electrical component is approved.

[0027] Input 108 may represent a map.

[0028] The graph may represent a map. The graph may include nodes representing waypoints. The graph may include edges representing connections between waypoints. The edge of the graph may represent a connection between a first waypoint represented by a first node of the graph and a second waypoint represented by a second node of the graph.

[0029] The question may be related to a connection. The question may be related to another connection in the map. The other connection may include a first waypoint. The other connection may include a second waypoint. The other connection may include a connection between the first waypoint and the second waypoint.

[0030] Device 100 may be configured to: select a connection for moving technical system 112 in accordance with the answer; and move technical system 112 via the connection. Output 114 may include: the selected connection or an instruction that results in moving technical system 112 via the connection.

[0031] Figure 2 An exemplary input 108 is depicted. The exemplary input includes a digital image 200.

[0032] Digital image 200 depicts a first element 202 next to character N1, a second element 204 next to character N2, a third element 206 next to character N3, a fourth element 208 next to character N4, and a fifth element 210 next to character N5. Digital image 200 may depict more or fewer elements. Elements may be depicted next to more or fewer characters.

[0033] Digital image 200 depicts a first connection 212 between first element 202 and second element 204. Digital image 200 depicts a second connection 214 between second element 204 and third element 206. Digital image 200 depicts a third connection 216 between third element 206 and fourth element 208. Digital image 200 depicts a fourth connection 218 between fourth element 208 and fifth element 210.

[0034] Digital image 200 may include different types of connections and elements.

[0035] According to the example, the digital image 200 includes two different types of connections and three different types of elements. The first element group includes the first element 202, the second element 204, and the third element 206. The second element group includes the third element 206, the fourth element 208, and the fifth element 210. According to the example, the first element 202 and the second element 204 are of the first type, the fourth element and the fifth element 210 are of the second type, and the third element 206 is of the third type.

[0036] Different groups may include different types of connections. According to an example, the first connection 212 and the second connection 214 are first type of connections. According to an example, the third connection 216 and the fourth connection 218 are second type of connections.

[0037] According to an example, a connection between two elements of a first type is of a first type. According to an example, a connection between two elements of a second type is of a second type. According to an example, a connection between an element of a first type and an element of a third type is of a first type. According to an example, a connection between an element of a second type and an element of a third type is of a second type.

[0038] The graph may include nodes representing elements and edges representing connections. According to an example, the graph includes a first node representing a first element 202, a second node representing a second element 204, and a third node representing a third element 206, a fourth node representing a fourth element 208, and a fifth node representing a fifth element 210. According to an example, the graph includes a first edge representing a first connection 212, a second edge representing a second connection 214, and a third edge representing a third connection 216, and a fourth edge representing a fourth connection 218.

[0039] An element can be a waypoint. An element can be an electrical component.

[0040] Figure 3 The functional representation of the exemplary problem "How many elements are between N1 and N5" is depicted. In the functional representation, Subtract() is a function used to determine the difference between the inputs to the function Subtract(), Count() is a function used to determine the number of elements between the inputs to the Count() function, Int(2) is a function that provides the value 2, Element(N1) is a function that outputs value N1. Element(N1) is a function that outputs value N2.

[0041] N1 and N5 are names of elements in the digital image 200 .

[0042] Example questions based on the template: “How many elements are between [Element 1] and [Element 2]”.

[0043] The template includes a first variable [Element 1] and a second variable [Element 2]. The first variable is a placeholder for the first name in the problem. The second variable is a placeholder for the second name in the problem. The order in which the names appear in the problem can correspond to the order in which the variables are inserted in the function representation.

[0044] The function representation can be based on the following template: the template includes the functions Subtract(), Count(), Int(), ShortestPath() as described above, the first variable name Element 1 replaces the name N1 at the first occurrence Element (Element 1) of the function, and the second variable name Element 2 replaces the name N5 at the second occurrence Element (Element 2) of the function.

[0045] Figure 4 An architecture 400 for machine learning is depicted. The architecture 400 is configured to determine answers 402 from a first input 404 (eg, a digital image or symbolic description of a graph) and a second input 406 (eg, a question). The answers 402 are, for example, an answer set.

[0046] The architecture 400 includes a first model 408. The first model 408 is configured to: identify edges between nodes; and determine, from the first input 404, an expression 410 that associates the edge with the node.

[0047] The architecture 400 includes a second model 412. The second model 412 is configured to: from the first input 404, identify names and associations between names and nodes.

[0048] The second model 412 is configured to determine an expression 414 that associates a node with a name according to the first input 404 .

[0049] The architecture 400 includes a third model 416. The third model 416 is configured to: recognize a name in the second input 406; and determine, from the second input 406, an expression 418 that includes the name.

[0050] The architecture 400 includes a module 420. The module 420 is configured to determine the answer 402 according to the expressions output by the first model 408, the second model 412, and the third model 416.

[0051] The device 100 is configured according to the architecture 400 , for example.

[0052] Figure 5 A flow chart with steps of a computer-implemented method for machine learning is depicted. The method is explained with reference to architecture 400.

[0053] The method includes step 502 .

[0054] In step 502, input 404 is provided.

[0055] The method may include receiving input 404 from the sensor 110 or at the interface 106 .

[0056] The input 404 is, in this example, a digital image of a graph. The input 404 may be a symbolic description of the graph.

[0057] The input 404 includes a first node, a first name, a second node, and a second name.

[0058] The input 404 may include more nodes and names. An example for the input 404 is the digital image 200 .

[0059] The input 404 includes an edge between a first node and a second node. The input 404 includes a first name and a second name.

[0060] An example for an edge is the connection 212 between the first element 202 and the second element 204. An example for a first node is the first element 202. An example for a second node is the second element 204. An example for a first name is N1. An example for a second name is N2.

[0061] The method includes step 504 .

[0062] In step 504, a question 406 is provided that includes a first name and a second name.

[0063] The method includes step 506 .

[0064] In step 506 , the first model 408 is used to determine the expression 410 that associates an edge with the first node and the second node according to the input 404 .

[0065] The first model 408 may be configured for semantic segmentation of the first input 404 (particularly, a digital image), and configured to identify that an edge is between a first node and a second node according to a result of the semantic segmentation.

[0066] The first model 408 may be configured to determine a matrix including a row for the first node and a column for the second node and indicating the existence of an edge between the first node and the second node by entries in the elements of the matrix that are in the row and the column.

[0067] The method may include: determining a matrix; and determining 410 an expression associating an edge with a first node and a second node in accordance with the matrix.

[0068] The first model 408 may be configured to determine a location, in particular, pixel coordinates, of the first node in the input 404 (eg, a digital image).

[0069] The first model 408 may be configured to determine a location, in particular, pixel coordinates, of the second node in the input 404 (eg, a digital image).

[0070] The first model 408 comprises, for example, an artificial neural network configured to map the input 404 to an expression 410 and / or a position.

[0071] The method includes step 508 .

[0072] At step 508 , an expression 414 associating the first node with the first name and an expression 414 associating the second node with the second name are determined from the input 404 using the second model 412 .

[0073] The second model 412 may be configured for optical character recognition in a digital image, and configured to determine the first name and the second name according to a result of the optical character recognition.

[0074] The method may include determining the first name and the second name using optical character recognition.

[0075] The method may include determining 414 an expression associating the first node with the first name and the second node with the second name according to respective positions of the first name and the second name in the input 404 .

[0076] The second model 412 may be configured to determine a location, in particular, pixel coordinates, of the first name in the input 404 (eg, a digital image).

[0077] Step 508 may include: determining a position; and determining an expression associating the first node with the first name according to the position.

[0078] The method includes step 510 .

[0079] Step 510 includes determining, using third model 416 , according to question 406 , an expression 418 that includes the first name and the second name.

[0080] The third model 416 may be configured to determine a first name in the question 406 according to a position of the first variable in the template for the question 406 .

[0081] The third model 416 may be configured to determine the second name in the question 406 according to the position of the second variable in the template for the question 406 .

[0082] Step 510 may include determining 418 an expression including the first name and the second name according to the template.

[0083] The method includes step 512 .

[0084] Step 512 includes determining answer 402 to question 406 according to expression 410 output by first model 408 for input 404 , expression 414 output by second model 410 for input 404 , and expression 418 output by third model 416 for question 406 .

[0085] Step 512 may include: determining answer set programming facts according to expressions provided by first model 408, second model 412, and third model 416; and determining answer 402 according to the answer set programming facts.

[0086] The method may include step 514 .

[0087] Input 404 may represent an electrical circuit. An edge may represent an electrical connection between a first electrical component represented by a first node and a second electrical component represented by a second node.

[0088] Question 406 may be related to a state of the electrical circuit, a state of the first electrical component, a second electrical component, or an electrical connection.

[0089] The 402 response may include a status.

[0090] Step 514 may include outputting the status or automatically approving or sorting the circuit according to the status, in particular, using the technical system 112. This means that the technical system 112 may operate according to the output 114 to approve or sort the circuit according to the status. The output 114 may be the status or a signal determined according to the status in order to operate the technical system 112 according to the status.

[0091] The input 404 may represent a map. An edge may represent a connection between a first waypoint represented by a first node and a second waypoint represented by a second node. A question may be related to the connection or connections in the map including the first waypoint, the second waypoint, and / or a connection between the first waypoint and the second waypoint.

[0092] Step 514 may include: selecting a connection for moving the technology system in accordance with answer 402; and moving technology system 112 via the connection. Technology system 112 may be operable using output 114. Output 114 may be determined in accordance with answer 402, for example, to provide a trajectory or directly direct an input to technology system 112 to move from a first waypoint to a second waypoint.

[0093] The method is not limited to adjacent nodes. The method also applies to nodes connected by paths in the graph. The method is not limited to two nodes. The method also applies to paths that include more than two nodes.

[0094] The task solved by the method is, for example, answering natural language questions. A natural language question may ask for the shortest path between two stations in a subway map while avoiding stations with certain properties.

[0095] For natural language questions, the method may consider only those natural language questions that can be answered using information that can be found in the digital image representing the subway map. The method may ignore all other symbolic information from the digital image or the natural language question. The method parses the graph to identify nodes and edges, reads and understands the names that label the stations and associate them with the nodes of the graph, and understands the natural language questions and infers the information extracted from the digital image to answer the natural language questions.

[0096] The first model 408 may be configured for optical graph recognition, for example as described in C. Auer, C. Bachmaier, FJ Brandenburg, A. Gleiβner, J. Reislhuber, Optical graph recognition, in: Graph Drawing, Springer Berlin Heidelberg, Berlin, Heidelberg, 2013, pp. 529–540.

[0097] The second model 412 can be configured for optical character recognition, as described in AM Sabu, ASDas, A survey on various optical character recognition techniques, in: 2018 Conference on Emerging Devices and Smart Systems (ICEDSS), 2018, pp. 152–155.

[0098] The third model 416 can be configured to determine a function procedure for representing a natural language question using a CLEVR template, for example, as described in D. Mack, A. Jefferson, CLEVR graph: A dataset for graph question answering, 2018. URL: https: / / github.com / Octavian-ai / clevr-graph Described in .

[0099] The method may use answer set programming, for example as described in G. Brewka, T. Eiter, M. Truszczynski, Answer set programming at a glance, Commun. ACM 54 (2011) 92–103.

[0100] Module 420 may include an answer set programming solver.

Claims

1. A computer-implemented method for machine learning, characterized in that The method comprises providing (502) an input (404), in particular a digital image or symbolic description of a graph, wherein the input (404) comprises a first node, a first name, a second node and a second name, wherein the input (404) comprises an edge between the first node and the second node, wherein the method comprises: determining (506) an expression (410) associating the edge with the first node and the second node according to the input (404) using a first model (408), wherein the first model (408) is configured to recognize that the edge is between the first node and the second node, wherein the first model (408) is configured to determine the expression (410) associating the edge with the first node and the second node according to the input (404); determining (508) an expression (414) associating the first node with the first name and an expression (414) associating the second node with the second name according to the input (404) using a second model (412), wherein the second model (412) The method comprises providing (504) a question (406) including the first name and the second name; determining (510) an expression (418) including the first name and the second name in accordance with the question (406) using a third model (416); wherein the third model (416) is configured to identify the first name and the second name in the question (406) and to determine an expression (418) including the first name and the second name in accordance with the question (406); and wherein the method comprises determining (512) an answer (402) to the question (406) in accordance with the expression.

2. The method according to claim 1, characterized in that The method comprises receiving (502) the input (404) from a sensor (110) or at an interface (106), wherein the input (404) represents an electrical circuit, wherein the edge represents an electrical connection between a first electrical component represented by the first node and a second electrical component represented by the second node, wherein the question (406) relates to a state of the circuit, the first electrical component, the second electrical component, or the electrical connection, wherein the answer comprises the state, and wherein the method comprises (514) outputting the state or automatically approving or reorganizing the circuit according to the state.

3. The method according to claim 1, characterized in that The method comprises receiving (502) the input (404) from a sensor (110) or at an interface (106), wherein the input (404) represents a map, wherein the edge represents a connection between a first waypoint represented by the first node and a second waypoint represented by the second node, wherein the question relates to the connection or connections in the map including: the first waypoint, the second waypoint and / or a connection between the first waypoint and the second waypoint, wherein the method comprises (514) selecting a connection for moving the technical system in accordance with the answer (402) and moving the technical system (112) via the connection.

4. The method according to any one of the preceding claims, characterized in that The first model (408) is configured for semantic segmentation of the digital image, and is configured to identify that the edge is between the first node and the second node according to a result of the semantic segmentation.

5. The method according to any one of the preceding claims, characterized in that The first model (408) is configured to determine a matrix comprising rows for the first node and columns for the second node and indicating the existence of an edge between the first node and the second node, the determination being made by entries in the elements of the matrix that are in the rows and the columns, wherein the method comprises (506) determining the matrix and determining an expression (410) associating the edge with the first node and the second node according to the matrix.

6. The method according to any one of the preceding claims, characterized in that The second model (412) is configured for optical character recognition in the digital image, and is configured to determine the first name and the second name according to a result of the optical character recognition.

7. The method according to any one of the preceding claims, characterized in that The first model (408) is configured to determine the position of the first node in the digital image, in particular, pixel coordinates, wherein the second model is configured to determine the position of the first name in the digital image, in particular, pixel coordinates, wherein the method includes (506) determining the position and determining an expression associating the first node with the first name according to the position.

8. The method according to any one of the preceding claims, characterized in that The third model (416) is configured to determine a first name in the question according to a position of a first variable in a template for the question and to determine a second name in the question according to a position of a second variable in the template for the question, wherein the method includes determining (510) an expression including the first name and the second name according to the template.

9. The method according to any one of the preceding claims, characterized in that The method includes (512) determining an answer set programming fact according to the expression and determining the answer according to the answer set programming fact.

10. A device (100) for machine learning, characterized in that: The device (100) comprises at least one processor (102) and at least one memory (104), wherein the at least one processor (102) is configured to execute instructions which, when executed by the at least one processor (102), cause the device (100) to perform a method according to one of claims 1 to 8, wherein the at least one memory (104) is configured to store the instructions.

11. A computer program, characterized in that The computer program comprises computer executable instructions which, when executed by a computer, cause the computer to perform the method according to one of claims 1 to 8 .

Images