Device and computer-implemented method for machine learning
By using scene diagrams in machine learning classifiers to analyze the objects and relationships of digital images, rules are formulated to correct systemic errors, and the shortcomings of classifiers in the prior art are solved, and the accuracy and system performance of the classifier are improved.
Patent Information
- Application Number
- JP2024152951
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-09-06
- Filing Date
- 2024-09-05
- Publication Date
- 2025-03-18
AI Technical Summary
Existing machine learning classifiers have shortcomings in identifying and correcting systemic errors, resulting in degradation of classification errors and system performance.
By providing classification rules for the first and second categories, using scene maps to analyze the objects and relationships of digital images, identifying the key factors that lead to misclassification, and thus formulating rules to correct systemic errors.
Effectively identify and correct systemic errors, improve the accuracy of the classifier and the overall performance of the system, and ensure reliable classification results are provided in the information transmission and monitoring system.
Smart Images
Figure 2025037849000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to devices and computer-implemented methods for machine learning. [Background technology]
[0002] In machine learning, a classifier can be used to classify a digital image. Summary of the Invention [Problem to be solved by the invention]
[0003] The device and computer-implemented method according to the independent claims provide rules for identifying whether the classifier is making systematic mistakes, which may serve to provide a means for improving the classifier. [Means for solving the problem]
[0004] The method comprises the steps of providing a first class and a second class of classification; providing a first set of scene graphs, the first set of scene graphs including scene graphs of digital images that are misclassified into the first class or the second class; providing a second set of scene graphs, the second set of scene graphs including scene graphs of digital images that are correctly classified with respect to the first class or the second class; and determining, in response to the first set of scene graphs and the second set of scene graphs, a rule indicating that a presence of a first object and / or a second object in the digital image and / or a relationship between the first object and the second object in the scene graphs of the digital image results in a classification of the digital image including a misclassification of the digital image into the second class rather than the first class.
[0005] The digital image may be captured by a sensor, in particular a camera, a radar sensor, a lidar sensor, an infrared sensor, an ultrasonic sensor or a motion sensor.
[0006] The method may include detecting an anomaly in response to the rule or identifying a cause of the anomaly in response to the rule, for example, if the rule indicates a misclassification, then an anomaly is detected in response to the rule, and otherwise, no anomaly is detected in response to the rule.
[0007] The method may comprise the steps of operating a technical system, in particular a system for transmitting information, in particular a surveillance system or a medical imaging system, depending on the classification, and operating the technical system independently of the classification if an anomaly is detected depending on the rules.
[0008] The step of operating the technical system includes a step of outputting an output, in particular a control signal for controlling the technical system or an indication of the classification for display by the technical system, determined depending on the classification if no anomaly is detected, and a step of not outputting an output or outputting an output having an indication that an anomaly has been detected if an anomaly is detected.
[0009] The method may include determining a logic program as a function of the first set of scene graphs and the second set of scene graphs, and determining a rule as a function of the logic program.
[0010] The method may comprise the steps of providing a label to the at least one digital image, determining a classification of the at least one digital image depending on at least a part of the at least one digital image, in particular pixel values of the at least one digital image, determining a scene graph of the at least one digital image depending on at least a part of the at least one digital image, in particular pixel values of the at least one digital image, determining whether the classification of the at least one digital image is correct or not depending on the label and the classification of the at least one digital image, adding the scene graph of the at least one digital image to a first set of scene graphs if the classification is incorrect, and adding the scene graph of the at least one digital image to a second set of scene graphs if the classification is correct, which means that the scene graphs are split into two sets.
[0011] The device comprises at least one processor and at least one memory, the at least one processor configured to execute instructions that, when executed by the at least one processor, cause the device to perform the method, and the at least one memory configured to store the instructions.
[0012] The computer program may include computer readable instructions which, when executed by a computer, cause the computer to perform the method.
[0013] Further advantageous embodiments become apparent from the following description and the drawings. [Brief description of the drawings]
[0014] [Figure 1] 1 illustrates diagrammatically a device for machine learning. [Diagram 2] 1 shows a flowchart of a computer-implemented method for machine learning. [Diagram 3] 3 illustrates a schematic of an exemplary misclassification of an image 302. [Figure 4]2 illustrates a schematic of a scene graph of a first digital image; [Diagram 5] 2 illustrates a schematic of a scene graph for a second digital image. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0015] 1 shows a schematic diagram of a device for machine learning 100. The device 100 comprises at least one processor 102 and at least one memory 104.
[0016] The at least one processor 102 is configured to execute instructions that, when executed by the at least one processor 102, among other things, cause the device 100 to perform a computer-implemented method for machine learning. The at least one memory 104 is configured to store instructions. The instructions may be computer-readable instructions.
[0017] The device 100 optionally comprises a sensor 106. The sensor 106 is, for example, a camera, a radar sensor, a lidar sensor, an infrared sensor, an ultrasonic sensor, or a motion sensor.
[0018] The device 100 is configured to determine a classification in response to the digital image. The device 100 is optionally configured to control a technical system 108 in response to the classification.
[0019] The technical system 108 may be a system for transmitting information, in particular a monitoring system or a medical imaging system. The technical system 108 may comprise a sensor 106 instead of the device 100. The technical system 108 may comprise the device 100.
[0020] Device 100 is optionally configured to provide the classification in an output. Device 100 is configured to determine a rule responsive to the digital image. Device 100 is optionally configured to detect anomalies and / or identify a cause of the anomaly responsive to the rule.
[0021] A computer program may include computer readable instructions.
[0022] The device 100 comprises a classifier configured to output a classification of the digital image depending on at least a part of the digital image, in particular the pixel values of the digital image.
[0023] The device 100 comprises a model configured to construct a scene graph including objects and relationships between objects in the digital image detected by the model in response to the digital image. The device 100 comprises a reasoner configured to determine rules in response to a logic program. The device 100 is configured to implement a method for machine learning to determine the logic program and the rules in response to the digital image.
[0024] A logic program contains a set of rules. A rule is an if-then expression whose if-part may contain several conditions, possibly including negation, and whose then-part has a single atom that must be true whenever the if-part is true.
[0025] Rule r is h(X) ← b1(Y1), ..., b k (Y k ), not b k+1 (Y k+ 1), ..., not b n (Y n ) It is expressed in the form of h(X), b1(Y1), ..., b n (Y n) is a primary atom. The right-hand side of rule r is a conjunction of atoms. Furthermore, X, Y1, ..., Y n are predicates h, b1, ..., b n The left side of a rule r is called its head and is denoted head(r), and the right side is its body, denoted body(r). The positive and negative parts of the body are denoted body(r), respectively. + (r) and body - It is shown as (r).
[0026] Rule r is the body - If (r)=φ, it is positive, otherwise it is negative.
[0027] The following rule r indicates that the classifier tends to classify digital images into a first class c1 rather than a second class c2, i.e. r:missclassifies(id1,c 1、 c2)←includes(id1, id2), person(id2), dog(id3), includes(id1, id3), walking(id2, id3) where id2, id3 are identifiers of concrete objects within the scene id1 depicted in the digital image, and includes, person and walking are relations.
[0028] Rule r indicates that the presence of a first object id2 and / or a second object id3 in a digital image, i.e., scene id1, and / or a relationship between the first object id2 and the second object id3 in the scene graph of the digital image, e.g., walking, results in classification of the digital image including a misclassification of the digital image into the second class c2 instead of the first class c1.
[0029] The rule r is determined by inductive logic programming. Inductive logic programming is described, for example, in the book by Zeng et al., “Qiang Zeng, Jignesh M. Patel, David Page, “Quickfoil: Scalable inductive logic programming”. Proc. VLDB Endow., 8(3):197-208, 2014”.
[0030] The goal of inductive logic programming is to generalize from individual cases or observations in the presence of background knowledge by constructing hypotheses about unseen cases. Inductive logic programming is used to learn rules r that distinguish correctly from incorrectly classified scenes.
[0031] FIG. 2 is a flow chart of the machine learning method.
[0032] The goal of the method is to
number
number
[0033] The classification may be a scene classification.
number
number
number
[0034] The method includes step 202.
[0035] Step 202 includes providing a first class c1 and a second class c2 of the classification.
[0036] Step 202 is to generate a digital image i k and digital image i k A label c indicating at least one correct class of the classification k Pair with (i k , c k ).
[0037] According to one example, the pair (i k、 c k ) is the pair (i k、 c k ) is provided from storage containing
[0038] Step 202 optionally includes capturing each digital image.Step 202 optionally includes labeling each digital image.
[0039] Each digital image is captured, for example, by sensor 106 .
[0040] The method includes step 204 .
[0041] Step 204 includes providing a classification of the digital image. According to one example, the classification of the digital image is provided by a classifier.
number
[0042] Each digital image k Classification of f(i k ) for each digital image i k , in particular each digital image i k , depending on the pixel values.
[0043] The method includes step 206 .
[0044] Step 206 computes the scene graph G of the digital image i. i According to one example, the scene graph G of the digital image i is determined as i is provided by the model. Each digital image i k Scene Graph
number
[0045] The method includes step 208.
[0046] Step 208 selects a first scene graph set.
number
number
number
number
number
number
number
number
[0047] First scene graph set
number
[0048] A second set of scene graphs
number
[0049] According to one example, each digital image i k Label provided to c k and each digital image i k Classification of f(i k ) and according to each digital image i k Classification of f(i k ) is determined to be false or true.
[0050] Each digital image k Classification of f(i k ) is incorrect, i.e., for each digital image i k Label provided to c k If it does not match, then for each digital image i k The scene graph of the first set of scene graphs
number
[0051] Each digital image k Classification of f(i k ) is correct, i.e., for each digital image i k Label provided to c k If it matches, then for each digital image i k The scene graph of the first set is a second set of scene graphs.
number
[0052] According to one example, classifier-based scene segmentation includes collecting incorrectly classified digital images i and correctly classified digital images i, which are labeled as images of a first class c1, into two sets, respectively: E + = {i∈I|f(i)=c 1、 (i, c2)∈D} E- = {i∈I|f(i)=c 1、 (i, c1)∈D}
[0053] According to one example, the scene graph construction is performed by dividing the set E + The first set of scene graphs for digital images from
number
number
[0054] The method includes step 210.
[0055] Step 210 includes determining a logic program as a function of the first set of scene graphs and the second set of scene graphs.
[0056] In this example, the inductive logic programming problem instance is a scene graph G i By converting each of the expressions as binary ground predicates, it is formulated as follows: Optionally, the inductive logic programming formulation can take into account available background knowledge B, describing further semantic information about object classes appearing in the scene. The background knowledge B is a representation of the scene graph G i It may include unary and binary ground predicates constructed from
[0057] The method includes step 212.
[0058] Step 212 selects a first set of scene graphs.
number
number
[0059] The rule r is determined according to a logic program.
[0060] According to one example, the rule r is determined by a reasoner that reasons on a logic program. In this example, an inductive logic programming reasoner reasons about an inductive logic programming program. Zeng et al. 2014 describes an exemplary inductive logic programming reasoner that can be used to determine the rule r.
[0061] The method may include step 214.
[0062] Step 214 includes detecting an anomaly in accordance with the rules or determining the cause of the anomaly in accordance with the rules.
[0063] The method may include step 216.
[0064] Step 216 includes operating the technology system 108 .
[0065] According to one example, the technical system 108 acts in response to the classification if no anomaly is detected.
[0066] According to one example, the technical system 108 operates independently of the classification when an anomaly is detected according to a rule.
[0067] Operating the technical system 108 may include, for example, producing an output if no anomaly is detected according to a rule.
[0068] Operating the technical system 108 may include, for example, not providing an output if an anomaly is detected according to a rule.
[0069] The method may include, if an anomaly is detected, providing an output having an indication that an anomaly has been detected.
[0070] The output may be, for example, a control signal for controlling the technical system 108. The output may be, for example, a representation of the classification for display by the technical system 108.
[0071] The output is determined, for example, according to the classification.
[0072] The method is illustrated with the example of rule r: As a result of reasoning on a logic program, multiple rules may be determined by the reasoner.
[0073] The rules r identify patterns in the data D that the classifier f incorrectly classifies into a second class c2.
[0074] An example application of the method to identify patterns in an input containing five digital images is given below.
[0075] FIG. 3 shows a schematic diagram of an exemplary misclassification of an image 302 in an exemplary first class 304 and an exemplary second class 306. The predicted class 308 of the first digital image 302-1 is the first class 304. The label 310 of the first digital image 302-1 is the second class 306. This means that the first digital image 302-1 is misclassified. The predicted class 308 of the second digital image 302-2 is the first class 304. The label 310 of the second digital image 302-2 is the second class 306. This means that the second digital image 302-2 is misclassified. The predicted class 308 of the third digital image 302-3 is the first class 304. The label 310 of the third digital image 302-3 is the first class 304. 3. The predicted class 308 of the fourth digital image 302-4 is the second class 306. The label 310 of the fourth digital image 302-4 is the second class 306. This means that the fourth digital image 302-4 is correctly classified. The predicted class 308 of the fifth digital image 302-5 is the second class 306. The label 310 of the fifth digital image 302-5 is the second class 306. This means that the fifth digital image 302-5 is correctly classified.
[0076] For example, the first image 302-1 and the second image 302-2 were misclassified. The method uses rule induction to systematically analyze the misclassification errors.
[0077] A scene graph is a structured representation of a scene that represents the objects, attributes, and relationships between objects in the scene. The scene graph can be determined by a model according to a method for constructing a scene graph, such as that described in Zhu et al., “Guangming Zhu, Liang Zhang, Youliang Jiang, Yixuan Dang, Haoran Hou, Peiyi Shen, Mingtao Feng, Xia Zhao, Qiguang Miao, Syed Afaq Ali Shah, Mohammed Bennamoun, “Scene graph generation: A comprehensive survey”. CoRR, abs / 2201.00443, 2022”.
[0078] 4 shows a schematic diagram of an exemplary scene graph 402 of a scene represented in a first digital image 302-1. The first digital image 302-1 includes a scene im1 associated with the attribute "Scene" in the scene graph 402. The first digital image 302-1 includes a person and includes a first bounding box id2 associated with the attribute "Person" in the scene graph 402. The first digital image 302-1 includes a dog and includes a second bounding box id3 associated with the attribute "Dog" in the scene graph 402. The first digital image 302-1 includes a pedestrian area and includes a third bounding box id4 associated with the attribute "PedestrianArea" in the scene graph 402. The scene graph 402 includes a relationship "includes" between the attribute "Scene" or scene im1 and the attribute "Person" or first bounding box id2. The scene graph 402 includes an attribute “Scene” or scene im1 and a relationship “includes” with an attribute “Dog” or a second bounding box id3. The scene graph 402 includes an attribute “Person” or a first bounding box id2 and a relationship “locatedOn” with an attribute “PedestrianArea” or a third bounding box id4. The scene graph 402 includes an attribute “Dog” or a second bounding box id3 and a relationship “locatedOn” with an attribute “PedestrianArea” or a third bounding box id4. The scene graph 402 includes an attribute “Person” or a first bounding box id2 and a relationship “walking” with an attribute “Dog” or a second bounding box id3. The attribute “Person” represents a first object in the scene graph 402. The attribute “Dog” represents a second object in the scene graph 402. The relationship “walking” represents a relationship between a first object and a second object in the scene graph 402 .
[0079] 5 illustrates a schematic of an exemplary scene graph 502 for the second digital image 302-2. The second digital image 302-2 includes a scene im2 associated with an attribute "Scene" in the scene graph 502. The scene graph 502 includes similar entries as the scene graph 402. The attribute "Dog" represents a second object in the scene graph 502. The relationship "walking" represents a relationship between the first object and the second object in the scene graph 502.
[0080] Based on this input, we create a first set of scene graphs, E + contains a first digital image 302-1 and a second scene graph set E - includes a third digital image 302-3.
[0081] In this example, an example of inductive logic programming is E + ={misclassifies(im1, c1, c2), scene(im2)} E - ={misclassifies(im3, c1, c2), misclassifies(im 4、 c 1、 c2)} Here, im1 represents the first digital image 302-1, im2 represents the second digital image 302-2, im3 represents the third digital image 302-3, im4 represents the fourth digital image 302-4, c1 represents the first class 304, and c2 represents the second class 306.
[0082] Background knowledge B may include unary and binary evidence predicates constructed from scene graphs of the five digital images.
[0083] The inductive logic programming used to extract the rule r is as follows:
number
[0084] In this example, this results in the following rule: r:missclassifies(X, c 1、 c2)←includes(X, Y), person(Y), dog(Z), includes(X, Z), walking(Y, Z) where Y, Z are identifiers of tangible objects in scene X depicted in the input digital image, and includes, person, dog and walking are relations.
[0085] For example, if the rule r indicates a wrong classification for the input digital image, then an anomaly is detected according to the rule r; otherwise, no anomaly is detected according to the rule r.
[0086] For example, during scene classification for autonomous driving or robotics, misclassifications or anomalies may be detected according to the rule r. More generally, the method applies to the detection of misclassifications or anomalies during the task of classifying images of a scene into a set of predefined classes. For example, in the context of autonomous driving, a distinction is made between scenarios in which the target vehicle is expected to stop and scenarios in which the target vehicle is allowed to continue driving. In robotics, for example, images showing combinations of objects that form a stable structure can be classified from combinations that are likely to collapse, such as glass on a table and glass on a bowl. Digital images in which misclassifications or anomalies are detected, or classifications determined according to such digital images, may not be usable for decision-making in autonomous driving or robotics.
Claims
1. 1. A computer-implemented method for machine learning, comprising: Providing (202) a first class and a second class of a classification; providing (208) a first set of scene graphs, the first set of scene graphs including scene graphs of digital images misclassified in the first class or the second class; providing (208) a second set of scene graphs, the second set of scene graphs including scene graphs of digital images that are correctly classified with respect to the first class or the second class; determining (212) a rule in response to the first set of scene graphs and the second set of scene graphs that indicates that a presence of a first object and / or a second object in a digital image and / or a relationship between the first object and the second object in the scene graph of the digital image results in the classification of the digital image including a misclassification of the digital image into the second class rather than the first class; The method according to claim 1, further comprising:
2. The digital image is captured (202) by a sensor (106), in particular a camera, a radar sensor, a lidar sensor, an infrared sensor, an ultrasonic sensor or a motion sensor; 2. The method according to claim 1 .
3. Detecting anomalies according to said rules (214); or identifying the cause of the anomaly in accordance with said rules; 3. The method according to claim 1, further comprising:
4. - operating (216) a technical system (108), in particular a system transmitting information, in particular a surveillance system or a medical imaging system, depending on said classification; operating said technical system (108) when an anomaly is detected according to said rules and independent of said classification; The method of claim 3, comprising:
5. The step of operating the technical system (108) comprises: - outputting an output, in particular a control signal for controlling said technical system (108) or an indication of said classification for display by said technical system (108) determined as a function of said classification if no anomaly is detected; not providing the output or, if the anomaly is detected, providing the output with an indication that the anomaly has been detected; The method of claim 4, comprising:
6. determining (210) a logic program as a function of the first set of scene graphs and the second set of scene graphs; determining (212) said rules in response to said logic program; 6. The method according to claim 1, further comprising:
7. Providing a label for at least one digital image; - determining (204) a classification of said at least one digital image as a function of at least a part of said at least one digital image, in particular the pixel values of said at least one digital image; - determining (206) a scene graph of said at least one digital image as a function of at least a portion of said at least one digital image, in particular pixel values of said at least one digital image; determining (208) whether the classification of the at least one digital image is correct or not as a function of the label and the classification of the at least one digital image; adding the scene graph of the at least one digital image to the first set of scene graphs if the classification is incorrect or adding the scene graph of the at least one digital image to the second set of scene graphs if the classification is correct; 7. The method according to claim 1, further comprising:
8. A device (100) for machine learning, comprising: The device (100) comprises: At least one processor (102); At least one memory (104); Equipped with The at least one processor (102) is configured to execute instructions that, when executed by the at least one processor (102), cause the device (100) to perform a method according to any one of claims 1 to 7; The at least one memory (104) is configured to store the instructions. A device comprising:
9. A computer program comprising computer readable instructions which, when executed by a computer, cause the computer to carry out the method according to any one of claims 1 to 7.