Device and computer-implemented method for machine learning

Abstract

Apparatus and computer-implemented method for machine learning, the method comprising: providing (202) a first class and a second class of a classification, providing (208) a first set of scene graphs, wherein the first set of scene graphs comprises scene graphs of digital images that are incorrectly classified in the first class or the second class, providing (208) a second set of scene graphs, wherein the second set of scene graphs comprises scene graphs of digital images that are correctly classified with respect to the first class or the second class, determining (212) a rule depending on the first set of scene graphs and the second set of scene graphs, which specifiesthat the presence of a first object and / or a second object in a digital image and / or a relation between the first object and the second object in the scene graph of the digital image leads to the classification of the digital image including a misclassification of the digital image into the second class instead of the first class.

Description

State of the art

[0001] The invention relates to a device and a computer-implemented method for machine learning.

[0002] In machine learning, classifiers can be used to classify digital images. Disclosure of the invention

[0003] The device and the computer-implemented method according to the independent claims provide a rule for identifying whether a classifier makes systematic errors. The rule can help to provide means for improving the classifier.

[0004] The procedure comprises providing a first class and a second class of a classification, providing a first set of scene graphs, wherein the first set of scene graphs comprises scene graphs of digital images that are incorrectly classified in the first class or the second class, providing a second set of scene graphs, wherein the second set of scene graphs comprises scene graphs of digital images that are correctly classified with respect to the first class or the second class, and determining a rule, dependent on the first set of scene graphs and the second set of scene graphs, which specifies that the presence of a first object and / or a second object in a digital image and / or a relation between the first object and the second object in the scene graph of the digital image results inthat the classification of the digital image includes a misclassification of the digital image into the second class instead of the first class.

[0005] The digital image can be captured by a sensor, in particular a camera, radar sensor, lidar sensor, infrared sensor, ultrasonic sensor or motion sensor.

[0006] The procedure can involve detecting a rule-dependent anomaly or determining the cause of a rule-dependent anomaly. For example, the rule-dependent anomaly is detected if the rule specifies the misclassification, and otherwise no rule-dependent anomaly is detected.

[0007] The procedure may include operating a technical system, in particular a system for transmitting information, especially a monitoring system or a medical imaging system, depending on the classification, and operating the technical system regardless of the classification if an anomaly is detected depending on the rule.

[0008] Operating the technical system may include outputting an output, in particular a control signal for controlling the technical system or a display of the classification to be shown by the technical system, which is determined depending on the classification, if no anomaly is detected, and not outputting the output or outputting the output with an indication that the anomaly is detected, if the anomaly is detected.

[0009] The procedure can include determining a logic program depending on the first set of scene graphs and the second set of scene graphs, and determining the rule depending on the logic program.

[0010] The method may include the following: providing a label for at least one digital image, determining a classification for the at least one digital image depending on at least one part of the at least one digital image, in particular pixel values ​​of the at least one digital image, determining a scene graph for the at least one digital image depending on at least one part of the at least one digital image, in particular pixel values ​​of the at least one digital image, determining whether the classification of the at least one digital image is correct or not, depending on the label and the classification of the at least one digital image, and adding the scene graph for the at least one digital image to the first set of scene graphs if the classification is incorrect, or adding the scene graph for the at least one digital image to the second set of scene graphs.if the classification is correct. That is, the scene graphs are divided into two sets.

[0011] The device comprises at least one processor and at least one memory, wherein the at least one processor is designed to execute instructions which, when executed by the at least one processor, cause the device to perform the method, and wherein the at least one memory is designed to store the instructions.

[0012] A computer program may contain computer-readable instructions which, when executed by the computer, cause the computer to perform the procedure.

[0013] Further advantageous embodiments are derived from the following description and the drawing. They show: Fig. 1 schematically a device for machine learning, Fig. 2 a flowchart of a computer-implemented machine learning process, Fig. 3 schematically an example of misclassification for images 302, Fig. 4. Schematically, a scene graph for a first digital image, Fig. 5 schematically shows a scene graph for a second digital image.

[0014] Fig. Figure 1 schematically shows a device 100 for machine learning. The device 100 comprises at least one processor 102 and at least one memory 104.

[0015] The at least one processor 102 is designed to execute instructions which, when executed by the at least one processor 102, cause the device 100 to execute a machine learning method, in particular a computer-implemented method. The at least one memory 104 is designed to store the instructions. The instructions can be computer-readable.

[0016] The device 100 optionally includes a sensor 106. The sensor 106 is, for example, a camera, a radar sensor, a lidar sensor, an infrared sensor, an ultrasonic sensor or a motion sensor.

[0017] Device 100 is designed to determine a classification based on the digital image. Device 100 may also be designed to control a technical system 108 based on the classification.

[0018] Technical system 108 can be a system for transmitting information, in particular a monitoring system or a medical imaging system. Technical system 108 can include sensor 106 instead of device 100. Technical system 108 can include device 100.

[0019] Device 100 is optionally designed to provide the classification for an output. Device 100 is designed to determine a rule depending on the digital image. Device 100 is optionally designed to detect an anomaly and / or to determine the cause of an anomaly depending on the rule.

[0020] A computer program can include computer-readable instructions.

[0021] The device 100 comprises a classifier designed to output a classification of a digital image depending on at least one part of the digital image, in particular pixel values ​​of the digital image.

[0022] Device 100 comprises a model designed to construct scene graphs that include objects and relations between objects in the digital image, which are detected by the model based on the digital image. Device 100 comprises a logic thinking device designed to determine rules based on a logic program. Device 100 is designed to execute a machine learning method for determining the logic program and the rules based on digital images.

[0023] The logic program comprises a set of rules. A rule is an if-then expression whose if part can contain multiple conditions, some possibly with negation. The then part has a single atom that must always be true if the if part is true.

[0024] A rule r is an expression of the form h(X)←b1(Y1),…,bk(Yk),not bk+1(Yk+1),…,not bn(Yn) where h(x), b1(Y1),...,bn(Yn) are first-order atoms. The right-hand side of rule r is a conjunction of atoms. Furthermore, X, Y1,...,Yn are tuples of either variables or constants, each with a length equal to the parity of the predicates h, b1,...,bn. The left-hand side of rule r is called its head, denoted head(r), while the right-hand side is its body, denoted body(r). The positive and negative parts of the body are each denoted body + (r) and body - (r) is denoted.

[0025] The rule r is positive if body - (r)=ø, and otherwise negative.

[0026] The following rule r states that the classifier tends to classify a digital image into a first class c1 rather than a second class c2: r:misclassified(id1,c1,c2)←contains(id1,id2),person(id2),dog(id3),contains(id1,id3),execute(id2,id3) where id2,id3 are identifiers of specific objects in a scene id1 that is depicted in the digital image, and where contains, person and execute are relations.

[0027] The rule r states that the presence of a first object id2 and / or a second object id3 in a digital image, i.e., the scene id1, and / or a relation, e.g., execute, between the first object id2 and the second object id3 in the scene graph of the digital image results in the classification of the digital image including a misclassification of the digital image into the second class c2 instead of the first class c1.

[0028] The rule r is determined using inductive logic programming. Inductive logic programming is described, for example, in Zeng et al. 2014: Qiang Zeng, Jignesh M. Patel, and David Page. Quickfoil: Scalable inductive logic programming. Proc. VLDB Endow., 8(3):197-208, 2014.

[0029] The goal of inductive logic programming is to generalize individual instances or observations, given background knowledge, by building hypotheses about previously unseen instances. Inductive logic programming is used to learn the rule r that distinguishes misclassified scenes from correctly classified ones.

[0030] Fig. Figure 2 shows a flowchart of the machine learning process.

[0031] One goal of the method is to detect systematic errors in a pretrained classifier f:(Ω-ℝ3)→C. The input for the method is the classifier f, a labeled data set D={(i k ,c k )}k=1n with n samples of a scene image i∈I from a set of scene images I and a label c∈C from a set of classes C and the first class c1 and the second class c2 with c1#c2, c1∈C, c2∈C.

[0032] The classification can be a scene classification. Assuming a set of object classes C for a given scene image I:Ω-ℝ 3 Where Ω represents pixel space, scene classification concerns the construction of a classifier. The classifier is, for example, a convolutional neural network with L.

[0033] Layers according to a function f:(Ω→ℝ 3 )→C, namely f(I)=argmax k∈C (softmax(f l ·...·f2·f1(I)) k ), where f l defines the l-th layer of the network.

[0034] The procedure includes step 202.

[0035] Step 202 involves providing the first class c1 and the second class c2 of the classification.

[0036] Step 202 involves providing pairs (i k ,c k ) of a digital image i k and a marking c k, which specifies at least one correct class of classification for the digital image ik.

[0037] According to an example, the pairs (i k ,c k ) provided from a storage that contains the pairs (i k ,c k ) includes.

[0038] Step 202 may include capturing the respective digital image. Step 202 may also include identifying characteristics of the respective digital image.

[0039] The respective digital image is captured, for example, by sensor 106.

[0040] The procedure includes step 204.

[0041] Step 204 involves providing classifications of the digital images. As shown in the example, the classifications of the digital images are provided using the classifier: f:(Ω→ℝ3)→C

[0042] The classification f(i k ) of a respective digital image i kwill depend on the specific digital image i k provided, in particular pixel values ​​of the respective digital image i k .

[0043] The procedure includes step 206.

[0044] Step 206 involves determining a scene graph G i for the digital images i. According to an example, the scene graphs Gi of the digital images i are provided with the model. The scene graph of a respective digital image i k will depend on the specific digital image i k provided, in particular pixel values ​​of the respective digital image i k .

[0045] The procedure includes step 208.

[0046] Step 208 involves providing a first set of scene graphs EG+ and a second set of scene graphs EG-. The first set of scene graphs EG+ and the second set of scene graphs EG- can be read from a memory containing a predetermined first set of scene graphs EG+ and a predetermined second set of scene graphs EG. If the predetermined first set of scene graphs EG+ and the predetermined second set of scene graphs EG- are available, steps 202 through 206 can be omitted; that is, the procedure can begin with step 208.

[0047] The first set of scene graphs EG+ comprises scene graphs Gi of digital images i that are incorrectly classified by the classifier. Incorrectly classified can mean that the classification f(ik) of a given digital image i k not with the marking c k matches the one for the respective digital image i kis provided.

[0048] The second set of scene graphs EG- comprises scene graphs Gi of digital images i that are correctly classified by the classifier. Correctly classified can mean that the classification f(i k ) of a respective digital image i k with the designation c k matches the one for the respective digital image i k is provided.

[0049] According to an example, depending on the designation c k , which are for a respective digital image i k is provided, and the classification f(i k ) of the respective digital image i k determines whether the classification f(i k ) of the respective digital image whether it is incorrect or correct.

[0050] If the classification f(i k ) of the respective digital image i k is incorrect, i.e., not marked with c kmatches the one for the respective digital image i k , is provided, the scene graph for the respective digital image i k to the first set of scene graphs EG+ added.

[0051] If the classification f(i k ) of the respective digital image i k is correct, i.e., with the designation c k matches the one for the respective digital image i k , is provided, the scene graph for the respective digital image i k to the second set of scene graphs E G added.

[0052] According to one example, a classifier-based partition involves collecting incorrectly and correctly classified digital images i, which are labelled as first-class images c1, into the following two sets: E+={i∈l|f(i)=c1,(i,c2)∈D}E−={i∈l|f(i)=c1,(i,c1)∈D} According to one example, scene graph construction includes constructing the first set of scene graphs. EG+ for the digital images from set E + and constructing the second set of scene graphs EC− for the digital images from set E - .

[0053] The procedure includes step 210.

[0054] Step 210 involves determining the logic program depending on the first set of scene graphs and the second set of scene graphs.

[0055] In this example, a problem instance of inductive logic programming is formulated as follows by translating the scene graphs G. iinto their respective representations as binary basic predicates. Formulating the inductive logic programming can consider any available background knowledge B, which describes further semantic information regarding object classes appearing in the scene. The background knowledge B can include unary and binary basic predicates derived from the scene graphs G. i be constructed.

[0056] The procedure includes step 212.

[0057] Step 212 involves determining the rule r depending on the first set of scene graphs. EG+ and the second set of scene graphs EC−.

[0058] The rule r is determined depending on the logic program.

[0059] According to an example, the rule r is determined by the logical thinking device that thinks logically about the logic program. In the example, a logical thinking device of inductive logic programming thinks about the program of inductive logic programming. Zeng et al. (2014) describe an exemplary logical thinking device of inductive logic programming that can be used to determine the rule r.

[0060] The procedure may include a step 214.

[0061] Step 214 involves detecting a rule-dependent anomaly or determining a cause of a rule-dependent anomaly.

[0062] The procedure may include a step 216.

[0063] Step 216 includes operating the technical system 108.

[0064] According to one example, technical system 108 is operated depending on the classification if no anomaly is detected.

[0065] According to an example, the technical system 108 is operated independently of the classification if an anomaly is detected depending on the rule.

[0066] Operating the technical system 108 includes, for example, outputting a message if no anomaly is detected, depending on the rule.

[0067] Operating the technical system 108 includes, for example, not outputting the output if an anomaly is detected, depending on the rule.

[0068] The procedure may include outputting the result with a statement that the anomaly is detected, if the anomaly is detected.

[0069] The output is, for example, a control signal for controlling technical system 108. The output is, for example, a display of the classification for display by technical system 108.

[0070] The output is determined, for example, depending on the classification.

[0071] The procedure is described as an example of a rule r. Using the logical thinking device, several rules can be determined as a result of logical reasoning on the logic program.

[0072] The rule r identifies a pattern in the data D that the classifier f has incorrectly classified into the second class c2.

[0073] The following is an example of an application of the method for identifying the pattern in an input consisting of five digital images.

[0074] Fig. Figure 3 schematically shows an example of a misclassification for images 302 in an example first class 304 and an example second class 306. A predicted class 308 for a first digital image 302-1 is the first class 304. A label 310 for the first digital image 302-1 is the second class 306. That is, the first digital image 302-1 is misclassified. The predicted class 308 for a second digital image 302-2 is the first class 304. The label 310 for the second digital image 302-2 is the second class 306. That is, the second digital image 302-2 is misclassified. The predicted class 308 of a third digital image 302-3 is the first class 304. The marking 310 for the third digital image 302-3 is the first class 304. That is, the third digital image 302-3 is correctly classified. The predicted class 308 of a fourth digital image 302-4 is the second class 306.The designation 310 for the fourth digital image 302-4 corresponds to class 306. This means that the fourth digital image 302-4 is correctly classified. The predicted class 308 for the fifth digital image 302-5 corresponds to class 306. The designation 310 for the fifth digital image 302-5 corresponds to class 306. This means that the fifth digital image 302-5 is correctly classified.

[0075] For example, the first image 302-1 and the second image 302-2 were misclassified. The procedure systematically analyzes the misclassification errors using rule induction.

[0076] The scene graph is a structured representation of a scene that expresses objects, attributes, and relations between objects within the scene. The scene graph can be determined by the model according to a scene graph construction procedure, such as that described in Zhu et al. 2022. Guangming Zhu, Liang Zhang, Youliang Jiang, Yixuan Dang, Haoran Hou, Peiyi Shen, Mingtao Feng, Xia Zhao, Qiguang Miao, Syed Afaq Ali Shah, and Mohammed Bennamoun. Scene graph generation: A comprehensive survey. CoRR, abs / 2201.00443, 2022.

[0077] Fig. Figure 4 schematically shows an example scene graph 402 for a scene represented in the first digital image 302-1. The first digital image 302-1 comprises a scene im1, which is assigned an attribute "Scene" in the scene graph 402. The first digital image 302-1 comprises a first outline box id2, which contains a person and is assigned an attribute "Person" in the scene graph 402. The first digital image 302-1 comprises a second outline box id3, which contains a dog and is assigned an attribute "Dog" in the scene graph 402. The first digital image 302-1 comprises a third outline box id4, which contains a pedestrian area and is assigned an attribute "PedestrianArea" in the scene graph 402. Scene graph 402 includes a relation “contains” between the attribute “scene” or the scene im1 and the attribute “person” or the first border box id2.Scene graph 402 includes a "contains" relation between the attribute "Scene" or the scene im1 and the attribute "Dog" or the second border box id3. Scene graph 402 includes a "locatedOn" relation between the attribute "Person" or the first border box id2 and the attribute "PedestrianArea" or the third border box id4. Scene graph 402 includes a "locatedOn" relation between the attribute "Dog" or the second border box id3 and the attribute "PedestrianArea" or the third border box id4. Scene graph 402 includes an "Execute" relation between the attribute "Person" or the first border box id2 and the attribute "Dog" or the second border box id3. The attribute "Person" represents a first object in scene graph 402. The attribute "Dog" represents a second object in scene graph 402.The relation “Execute” represents a relation between the first object and the second object in the scene graph 402.

[0078] Fig. Figure 5 schematically shows an example scene graph 502 for the second digital image 302-2. The second digital image 302-2 comprises a scene im2, which is assigned an attribute "Scene" in the scene graph 502. The scene graph 502 contains the same entries as the scene graph 402. The attribute "Dog" represents a second object in the scene graph 502. The relation "Execute" represents a relation between the first object and the second object in the scene graph 502.

[0079] Based on this input, the first set of scene graphs comprises E + the first digital image 302-1, and the second set of scene graphs E - The third digital image comprises 302-3.

[0080] The instance of inductive logic programming in this example is E+={misclassified(im1,c1,c2),scene(im2)}E−={misclassified(im3,c1,c2),misclassified(im4,c1,c2)} where im1 represents the first digital image 302-1, im2 represents the second digital image 302-2, im3 represents the third digital image 302-3, im4 represents the fourth digital image 302-4, c1 represents the first class 304, and c2 represents the second class 306.

[0081] Background knowledge B can include the unary and binary basic predicates constructed from the scene graphs of the five digital images.

[0082] Inductive logic programming is used to extract rules such that the following holds true: rU⊨EG+rU⊭EG+

[0083] In this example, this leads to the rule r:misclassified(X,c1,c2)←contains(X,Y),person(Y),dog(Z), contains(X,Z),execute(Y,Z) where Y and Z are identifiers of specific objects in a scene X, represented in an input digital image, and where contains, person, dog, and execute are relations. For example, the anomaly depends on the rule r is detected if the rule r specifies the misclassification for the input digital image, and otherwise no anomaly is detected depending on the rule r.

[0084] Misclassification or anomaly can be detected in scene classifications, for example, in autonomous driving or robotics, depending on the rule r. More generally, the method applies to detecting misclassification or anomaly when classifying images of scenes into a set of predefined classes. For example, in the context of autonomous driving, scenarios in which a target vehicle is expected to stop are distinguished from scenarios in which the target vehicle is allowed to continue driving. In robotics, for example, images depicting combinations of objects that form a stable structure can be classified against combinations that are likely to fall apart, such as a glass on a table versus a glass on a sphere.A digital image for which a misclassification or anomaly is detected, or the classification determined depending on such a digital image, cannot be used for decision-making in autonomous driving or robotics. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited non-patent literature

[0000] Qiang Zeng, Jignesh M. Patel, and David Page. Quickfoil: Scalable inductive logic programming. Proc. VLDB Endow., 8(3):197-208, 2014

[0028] Zeng et al. 2014

[0059] Guangming Zhu, Liang Zhang, Youliang Jiang, Yixuan Dang, Haoran Hou, Peiyi Shen, Mingtao Feng, Xia Zhao, Qiguang Miao, Syed Afaq Ali Shah und Mohammed Bennamoun. Scene graph generation: A comprehensive survey. CoRR, abs / 2201.00443, 2022

[0076]

Claims

[1] Computer-implemented method for machine learning, characterized bythat the method comprises the following: providing (202) a first class and a second class of a classification, providing (208) a first set of scene graphs, wherein the first set of scene graphs comprises scene graphs of digital images that are incorrectly classified in the first class or the second class, providing (208) a second set of scene graphs, wherein the second set of scene graphs comprises scene graphs of digital images that are correctly classified with respect to the first class or the second class, determining (212) a rule depending on the first set of scene graphs and the second set of scene graphs, which rule indicates that a presence of a first object and / or a second object in a digital image and / or a relation between the first object and the second object in the scene graph of the digital image leads to,that the classification of the digital image involves a misclassification of the digital image into the second class instead of the first class., [2] Method according to claim 1, characterized by that the digital image is captured (202) by a sensor (108), in particular a camera, a radar sensor, a lidar sensor, an infrared sensor, an ultrasonic sensor or a motion sensor. [3] Method according to one of the preceding claims, characterized by that the method comprises detecting (214) an anomaly depending on the rule or determining a cause of an anomaly depending on the rule. [4] Method according to claim 3, characterized bythat the method comprises operating (216) a technical system (108), in particular a system for transmitting information, in particular a monitoring system or a medical imaging system, depending on the classification and operating the technical system (108) independently of the classification if an anomaly is detected depending on the rule. [5] Method according to claim 4, characterized by that operating the technical system (108) comprises outputting an output, in particular a control signal for controlling the technical system (108) or an indication of the classification for displaying the technical system (108), which is determined depending on the classification if no anomaly is detected, and not outputting the output or outputting the output with an indication that the anomaly is detected if the anomaly is detected. [6] Method according to one of the preceding claims, characterized bythat the method comprises determining (210) a logic program depending on the first set of scene graphs and the second set of scene graphs and determining (212) the rule depending on the logic program. [7] Method according to one of the preceding claims, characterized bythat the method comprises the following: providing an identifier for at least one digital image, determining (204) a classification for the at least one digital image depending on at least a part of the at least one digital image, in particular pixel values ​​of the at least one digital image, determining (206) a scene graph for the at least one digital image depending on at least a part of the at least one digital image, in particular pixel values ​​of the at least one digital image, determining (208) whether the classification of the at least one digital image is correct or not depending on the identifier and the classification of the at least one digital image and adding the scene graph for the at least one digital image to the first set of scene graphs if the classification is incorrect, or adding the scene graph for the at least one digital image to the second set of scene graphs,if the classification is correct., [8] Device (100) for machine learning, characterized by that the device (100) comprises at least one processor (102) and at least one memory (104), wherein the at least one processor (102) is designed to execute instructions which, when executed by the at least one processor (102), cause the device (100) to carry out the method according to one of claims 1 to 7, and wherein the at least one memory (104) is designed to store the instructions. [9] Computer program, characterized by that the computer program comprises computer-readable instructions which, when executed by the computer, cause the computer to carry out the method according to any one of claims 1 to 7.

Images