Apparatus and computer-implemented method for processing a digital image for abnormality or nality detection
By combining object detection models, knowledge graphs and rules defined by experts, the problems of anomalies or normality in digital image detection are solved, and efficient detection in the fields of autonomous driving and manufacturing are achieved.
Patent Information
- Application Number
- CN202480008579.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-01-23
- Filing Date
- 2024-01-11
- Publication Date
- 2025-08-29
AI Technical Summary
The prior art is difficult to effectively detect abnormalities or normality when processing digital images, especially in applications such as autonomous driving and robot assembly. The lack of sufficient training data and exogenous information leads to insufficient detector quality.
By combining object detection models, knowledge graphs and expert-defined rules, geometric relationships and semantic similarities between objects are determined, and classifiers are used to detect abnormalities or normality.
Improve the detection accuracy of abnormality or normality of digital images, especially in the fields of autonomous driving and manufacturing, which can reliably distinguish the geometric relationship between normal and abnormality, and reduce the dependence on a large amount of training data.
Smart Images
Figure CN120569722A_ABST
Abstract
Description
Background Art
[0001] The present invention relates to a method and apparatus for processing digital images for abnormality or normality detection.
[0002] Biase, GD, Blum, H., Siegwart, R., Cadena, C.: “Pixel-wise anomaly detection in complex driving scenes,” in IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, Virtual Conference, June 19-25, 2021, pp. 16918-16927, Computer Vision Foundation / IEEE (2021). The disclosure of this invention discloses: determining whether a given image is anomaly at the pixel level.
[0003] Eiter, T., Kaminski, T.: “Exploiting contextual knowledge for hybrid classification of visual objects,” in JELIA, Lecture Notes in Computer Science, vol. 10021, pp. 223-239 (2016), disclosing: classifying images based on manually specified constraints such that no constraints are violated.
[0004] Invention Disclosure
[0005] A computer-implemented method for processing a digital image for abnormality or normality detection comprises: providing a digital image; determining, based on the digital image, geometric relationships between objects depicted in the digital image; providing knowledge about normal and / or abnormal geometric relationships between the objects; determining, based on the geometric relationships between the objects and the knowledge, a likelihood indicating normal or abnormal geometric relationships between the objects in the digital image; and detecting abnormality or normality based on the likelihood. The geometric relationships of the objects provide a good basis for detecting abnormalities. This information can be derived from the knowledge for improving abnormality or normality detection.
[0006] Determining the likelihood may include determining a likelihood value for a plurality of pairs of objects depicted in the digital image, pair by pair, and determining the likelihood based on the likelihood values. This provides a likelihood value for the image based on the pair by pair values of the objects depicted in the image. This further improves detection.
[0007] Determining the likelihood according to the likelihood values may include: determining the likelihood according to a weighted sum of the likelihood values; determining the likelihood according to a minimum value among the likelihood values; and determining the likelihood according to a maximum value among the likelihood values.
[0008] The method may comprise determining a geometric relationship between a first object and a second object, in particular based on a scene graph, wherein the knowledge comprises a knowledge graph defining allowed and / or disallowed relationships, wherein determining a likelihood value for the first object and the second object comprises determining a likelihood value for indicating normality when it is found that the geometric relationship satisfies the knowledge about allowed relationships from the knowledge graph or violates the knowledge about disallowed relationships from the knowledge graph, or determining a likelihood value for indicating abnormality when it is found that the geometric relationship violates the knowledge about allowed relationships from the knowledge graph or satisfies the knowledge about disallowed relationships from the knowledge graph. This integrates the knowledge graph to further improve detection. The scene graph presents the geometric relationships of all objects, and the knowledge graph defines which of these geometric relationships are normal and which are abnormal.
[0009] The method may include determining a geometric relationship between a first object and a second object, wherein the knowledge includes a rule determining whether the first object and the second object are in a normal geometric relationship or an abnormal geometric relationship, wherein determining a likelihood value for the first object and the second object includes determining a likelihood value indicating normality when the first object and the second object are found to be in a normal geometric relationship according to the rule, or determining a likelihood value indicating abnormality when the first object and the second object are found to be in an abnormal geometric relationship according to the rule. This integrates the rules to further improve detection.
[0010] The method may include classifying the likelihood using a classifier that indicates abnormality or normality based on the likelihood.
[0011] The method may include determining semantic similarity between objects depicted in a digital image, wherein the method includes classifying likelihood and semantic similarity using a classifier, the classifier indicating abnormality or normality based on likelihood and semantic similarity. This aggregates semantic similarity and likelihood. The additional information provided by semantic similarity further improves detection.
[0012] Determining the geometric relationship may include determining positions of the objects, determining a scene graph based on the positions, and determining the geometric relationship based on the scene graph. The positions reliably indicate the geometric relationship between the objects.
[0013] An apparatus for processing digital images for abnormality or normality detection comprises at least one processor and at least one memory, wherein the at least one processor is configured to execute instructions which, when executed by the at least one processor, cause the apparatus to perform the method, and wherein the at least one memory is configured to store the instructions.
[0014] A computer program comprising instructions which, when executed by a computer, cause the computer to perform the method.
[0015] Further advantageous embodiments emerge from the following description and the accompanying drawings. In the drawings:
[0016] Figure 1 schematically depicts an apparatus for processing digital images,
[0017] Figure 2 The normal situation is schematically depicted.
[0018] Figure 3 The abnormal situation is schematically depicted.
[0019] Figure 4 schematically depicts a printed circuit board comprising different objects,
[0020] Figure 5 depicts a flow chart of a first embodiment of a method for processing a digital image,
[0021] Figure 6 A flow chart of a second embodiment of a method for processing a digital image is depicted.
[0022] Figure 1 A device 100 for processing digital images is schematically depicted.
[0023] The device 100 includes at least one processor 102 and at least one memory 104 .
[0024] The device 100 includes an interface 106 for a sensor 108 and / or the sensor 108. Figure 1 In the example depicted in , device 100 includes sensor 108 .
[0025] For example, the sensor 108 is a camera, a radar sensor, a lidar sensor, a motion sensor, an infrared sensor, or an ultrasonic sensor.
[0026] The sensor 108 is configured to capture a digital image or reconstruct a digital image using geometric information captured by the sensor 108. For example, lidar or radar points are processed to reconstruct a digital image of the driving scene. In one example, the sensor 108 is configured to capture sensor data from the sensor 108 or multiple sensors 108 to determine a digital image (e.g., of the environment of the device 100). The digital image can be determined by the device 100 based on the sensor data.
[0027] The digital image represents visual data. The digital image may represent visual data captured or reconstructed by the sensor 108 (eg, a camera, radar, lidar, or ultrasonic sensor).
[0028] The device 100 may be configured to detect an object in a digital image. In one example, the device 100 is configured to detect an object from a set of digital images captured by the sensor 108 .
[0029] The device 100 can be configured to detect objects using an object detection model (e.g., R-CNN, YOLO, CenterNet, DETR model). The object detection model applied to the test data can be trained using training data from the same distribution. The object detection model applied to the test data can be trained on other training data and directly applied to the test data (zero-shot transfer).
[0030] The device 100 can be configured to detect categories of objects in digital images. In one example, the device 100 is configured to detect categories from a set of digital images captured by the sensor 108.
[0031] The output from the object detection model may include the identified objects, such as cars, pedestrians, and traffic lights. The output from the object detection model may include the object location, such as the xmin, xmax, ymin, and ymax of a two-dimensional bounding box, or the location (x, y, z) of a three-dimensional center box with individual box dimensions x, y, and z. The output from the object detection model may include a classification score for the object.
[0032] The output from the object detection model can be further used to generate a scene graph.
[0033] For example, the scene graph is determined as disclosed in "Scene Graph Generation: A Comprehensive Survey" (https: / / arxiv.org / pdf / 2201.00443.pdf).
[0034] The present disclosure relates to the problem of semantic anomaly detection, which is used to determine whether a scene or situation under test is normal or abnormal: given some input data (e.g., a set of images), the device 100 is configured to determine whether they depict real (i.e., normal) geometric relationships of objects or unreal (i.e., abnormal) geometric relationships.
[0035] As an example, a geometric relationship of objects like a car driving in front of another car corresponds to a normal situation.
[0036] Figure 2 A normal situation is schematically depicted, where a first car 202 is in front of a second car 204 .
[0037] As an example, in this disclosure, a geometric relationship in which one car is at least partially obscured by another car is considered an abnormal situation.
[0038] Figure 3 An abnormal situation is schematically depicted, wherein a first car 302 is partially obscured by a second car 304 .
[0039] The considered problem is important and relevant in many applications, such as autonomous driving or visual inspection of products assembled by robots.
[0040] For example, in the field of autonomous driving, the device 100 is an autonomous driving vehicle or a part thereof, which is configured to reliably distinguish between normal geometric relationships of objects and abnormal geometric relationships of objects.
[0041] In an adversarial attack, an attacker may place a sticker on the back of a car in an attempt to cause device 100 to misclassify the sticker as a stop sign ( <stop>sign).
[0042] One constraint used to mitigate this misclassification is:
[0043] <-locatedIn(X,Y),type(X,"StopSign"),type(Y,"Car")
[0044] This means that stickers that are misclassified as stop signs will violate the constraint LocatedIn(StopSign,Car).
[0045] Device 100 is configured to base its decision making on the normal geometric relationships of objects. In one example, device 100 is configured to detect a scene with unusual geometric relationships of objects as a normal scene. For example, device 100 is configured to detect a scene including a car transporter with cars as objects as a normal scene. In one example, device 100 is configured to detect a scene with unusual geometric relationships of objects as an abnormal scene. In one example, device 100 is configured to distinguish whether unusual geometric relationships of objects represent a normal scene or an abnormal scene.
[0046] For example, in the field of manufacturing, the device 100 is an assembly or a part thereof, which is configured to detect the geometric relationship of the components of a product (particularly a product manufactured automatically). The device 100 is configured to distinguish whether the detected geometric relationship of the components represents a normal geometric relationship of the components or an abnormal geometric relationship of the components. For example, the device 100 is configured to detect that the product includes a plastic top and a metal bottom. For example, the device 100 is configured to detect that the geometric relationship is abnormal (for example, for a specific electronic control unit architecture). The device 100 is configured to identify a problem with the product when an abnormal geometric relationship is detected.
[0047] Figure 4 A printed circuit board (PCB board) 400 comprising different objects is schematically depicted. Exemplary objects are light emitting devices LEDs (L1, ..., L8), resistors (R1, ..., R17) and capacitors (C1, ..., C6) at specific locations.
[0048] The device 100 is configured to detect and classify different objects.
[0049] The device 100 is configured to evaluate a knowledge graph and / or expert-defined rules that specify which objects are expected to be located in which positions relative to each other.
[0050] exist Figure 4 In the example depicted in , for example, the knowledge graph and / or expert-defined rules define that the resistor is to the right of the LED. If the expected placement of the object is not detected, this may be an outlier (i.e., a part with a production defect) that the classifier can identify.
[0051] In one example, the device 100 includes a classifier configured to detect abnormality or normality.
[0052] against Figure 4 In the example depicted in , device 100 can be configured to detect outliers (ie, parts with production defects).
[0053] The device 100 may include an output terminal 110. For example, the output terminal 110 is configured to output a result of detecting normality or abnormality. The output terminal 110 may be configured to control the operation of the device 100, for example, the action of the device 100.
[0054] Detecting abnormalities in images can often be challenging due to the very small number of abnormal images used to meaningfully train detectors. Models trained with sufficient data (e.g., open-world data) such as ChatGPT are not adaptable to embedded devices or devices in cars. Auxiliary methods based on logical rules can compensate for this. In addition, without auxiliary exogenous information, the quality of the detector depends heavily on the training data.
[0055] Methods for abnormality or normality detection may use output from an object detection model, including identified objects and locations of the objects.
[0056] Detection is improved by injecting exogenous information through rules or knowledge graphs.
[0057] Figure 5 A flow chart of a corresponding first embodiment of a method for processing a digital image is depicted.
[0058] The method includes step 502 .
[0059] In step 502, a digital image is provided.
[0060] For example, the image is provided via the interface 106. For example, the image is captured by the sensor 108.
[0061] Afterwards, execute step 504.
[0062] In step 504, objects depicted in the digital image are detected.
[0063] In this example, multiple objects are detected.
[0064] Thereafter, objects among the plurality of objects are identified in step 506 and their positions and scene graphs are determined in step 508 .
[0065] Step 508 includes determining geometric relationships between objects depicted in the digital image based on the scene graph.
[0066] In one example, determining the geometric relationship includes determining positions of the objects and determining the geometric relationship based on their positions.
[0067] For example, a geometric relationship between a first object i and a second object j is determined.
[0068] Afterwards, step 510 is executed.
[0069] Step 510 includes providing knowledge about normal and / or abnormal geometric relationships between objects.
[0070] In a first embodiment, providing knowledge includes providing a knowledge graph or a rule set.
[0071] For example, the rules are configured to determine whether a first object i and a second object j are in an allowed geometric relationship or an unallowed geometric relationship. The rules can be defined by an expert or learned via ILP (Inductive Logic Programming) by training the model with positive and negative examples from the scene.
[0072] The knowledge graph and rule set define the allowed or not allowed geometric relationships of objects.
[0073] A knowledge graph consists of nodes representing objects and edges representing relationships between objects.
[0074] According to one example, a knowledge graph includes an edge connecting a first node representing a first object and a second node representing a second object, wherein the edge corresponds to a geometric relationship between the first object and the second object.
[0075] According to one example, the knowledge graph is missing an edge connecting a first node and a second node. The missing edge may correspond to a missing information about a geometric relationship between the first object and the second object.
[0076] This missing edge can correspond to a geometric relationship between the first object and the second object. For example, in a knowledge graph that uses edges to capture normal geometric relationships, the missing edge can indicate an abnormal geometric relationship. For example, in a knowledge graph that uses edges to capture abnormal geometric relationships, the missing edge can indicate a normal geometric relationship.
[0077] For example, the knowledge graph includes a first node representing a first object i and a second node representing a second object j.
[0078] Optionally, the method includes step 512 .
[0079] Step 512 includes determining semantic similarity between objects depicted in the digital images.
[0080] Determining semantic similarity may include determining a plurality of pairs of objects depicted in the digital images.
[0081] Determining the semantic similarity may include determining a semantic similarity value for each of the plurality of pairs.
[0082] Determining the semantic similarity may include determining the semantic similarity based on the semantic similarity values.
[0083] For example, the semantic similarity is determined according to a weighted sum of the semantic similarity values.
[0084] For example, the semantic similarity is determined according to the minimum value among the semantic similarity values.
[0085] For example, the semantic similarity is determined according to the maximum value among the semantic similarity values.
[0086] Afterwards, step 514 is executed.
[0087] Step 514 includes determining a likelihood indicating a normal or abnormal geometric relationship between objects in the digital image based on the geometric relationship between the objects and the knowledge.
[0088] Determining the likelihood may include determining a plurality of pairs of objects depicted in the digital image.
[0089] Determining the likelihood may include determining a likelihood value for each of the plurality of pairs.
[0090] Determining the likelihood may include determining the likelihood based on the likelihood values.
[0091] For example, the likelihood is determined based on a weighted sum of the likelihood values.
[0092] For example, the likelihood is determined based on the minimum value among the likelihood values.
[0093] For example, the likelihood is determined based on the maximum value among the likelihood values.
[0094] When a rule is found in the rule set that determines that the first object and the second object are in a normal geometric relationship, a likelihood value indicating normality may be determined for the first object and the second object.
[0095] When a rule is found in the rule set that determines that the first object and the second object are in an unusual geometric relationship, a likelihood value indicating unusualness may be determined for the first object and the second object.
[0096] When it is found that the knowledge graph indicates that the first object and the second object are in a normal geometric relationship, a likelihood value indicating normality may be determined for the first object and the second object.
[0097] When it is discovered that the knowledge graph indicates that the first object and the second object are in an abnormal geometric relationship, a likelihood value indicating abnormality may be determined for the first object and the second object.
[0098] A likelihood value indicating normality can be determined for a first object and a second object when an edge is found in a knowledge graph that satisfies the following conditions: the edge connects a first node and a second node and the edge represents a normal geometric relationship, and the edge corresponds to the geometric relationship determined between the first object and the second object.
[0099] When it is found that the knowledge graph including the edge for the abnormal geometric relationship lacks an edge that satisfies the following conditions, a likelihood value indicating normality can be determined for the first object and the second object: the edge connects the first node and the second node and the edge represents the geometric relationship determined between the first object and the second object.
[0100] A likelihood value indicating abnormality can be determined for a first object and a second object when an edge is found in a knowledge graph that satisfies the following conditions: the edge connects a first node and a second node and the edge represents an abnormal geometric relationship, and the edge corresponds to the geometric relationship determined between the first object and the second object.
[0101] When it is found that the knowledge graph including the edge for the normal geometric relationship is missing an edge that satisfies the following conditions, a likelihood value indicating abnormality can be determined for the first object and the second object: the edge connects the first node and the second node and the edge represents the geometric relationship determined between the first object and the second object.
[0102] The first embodiment includes querying the knowledge graph for pairs (i, j) of identified objects, determining their likelihood values, and determining likelihoods based on these values.
[0103] Afterwards, execute step 516.
[0104] Step 516 includes classifying the likelihood using a classifier that indicates abnormality or normality based on the likelihood.
[0105] Optionally, step 516 includes classifying the likelihood and the semantic similarity together using a classifier, the classifier indicating abnormality or normality based on the likelihood and the semantic similarity.
[0106] Afterwards, execute step 518.
[0107] Step 518 includes detecting abnormality or normality based on the likelihood.
[0108] In particular, the detected result (ie, abnormality or normality) may be output via the output terminal 110 .
[0109] Optionally, the operation of the device 100 , for example, the action of the device 100 , is controlled according to the detected result.
[0110] Detection is improved by injecting exogenous information through expert-defined rules.
[0111] Figure 6 A flow chart of a corresponding second embodiment of a method for processing a digital image is depicted.
[0112] The method includes step 602 .
[0113] In step 602, a digital image is provided.
[0114] For example, the image is provided via the interface 106. For example, the image is captured by the sensor 108.
[0115] Afterwards, execute step 604.
[0116] In step 604, objects depicted in the digital image are detected.
[0117] In this example, multiple objects are detected.
[0118] Thereafter, objects among the plurality of objects are identified in step 606 and their positions and scene graphs are determined in step 608 .
[0119] Step 608 includes determining a scene graph based on the location.
[0120] Step 608 includes determining geometric relationships between objects depicted in the digital image based on the scene graph.
[0121] In one example, determining the geometric relationship includes determining positions of the objects and determining the geometric relationship based on their positions.
[0122] For example, a geometric relationship between a first object i and a second object j is determined.
[0123] Thereafter, steps 610 and 612 are performed to provide knowledge about normal and / or abnormal geometric relationships between objects.
[0124] Step 610 includes providing a knowledge graph.
[0125] Step 610 includes providing at least one rule.
[0126] The rule determines a normal geometric relationship or an abnormal geometric relationship for the first object and the second object.
[0127] When the first object and the second object are found to be in a normal geometric relationship according to the rule, likelihood values indicating normality are determined for the first object and the second object.
[0128] When the first object and the second object are found to be in an abnormal geometric relationship according to the rule, likelihood values indicating abnormality are determined for the first object and the second object.
[0129] For the first object and the second object, satisfying a rule in the rule set indicating a normal geometric relationship may result in determining a likelihood value indicating normality.
[0130] For the object and the second object, violating a rule in the rule set indicating an unusual geometric relationship may result in determining a likelihood value indicating unusualness.
[0131] Optionally, the method includes step 614 .
[0132] Step 614 includes determining semantic similarity between objects depicted in the digital images.
[0133] Determining semantic similarity may include determining a plurality of pairs of objects depicted in the digital images.
[0134] Determining the semantic similarity may include determining a semantic similarity value for each of the plurality of pairs.
[0135] Determining the semantic similarity may include determining the semantic similarity based on the semantic similarity values.
[0136] For example, the semantic similarity is determined according to a weighted sum of the semantic similarity values.
[0137] For example, the semantic similarity is determined according to the minimum value among the semantic similarity values.
[0138] For example, the semantic similarity is determined according to the maximum value among the semantic similarity values.
[0139] Afterwards, execute step 616.
[0140] Step 616 includes determining a likelihood indicating a normal or abnormal geometric relationship between objects in the digital image based on the geometric relationship between the objects and the knowledge.
[0141] Determining the likelihood may include determining a plurality of pairs of objects depicted in the digital image.
[0142] Determining the likelihood may include determining a likelihood value for each of the plurality of pairs.
[0143] Determining the likelihood may include determining the likelihood based on the likelihood values.
[0144] For example, the likelihood is determined based on a weighted sum of the likelihood values.
[0145] For example, the likelihood is determined based on the minimum value among the likelihood values.
[0146] For example, the likelihood is determined based on the maximum value among the likelihood values.
[0147] A likelihood value indicating normality can be determined for a first object and a second object when an edge is found in a knowledge graph that satisfies the following conditions: the edge connects a first node and a second node and the edge represents a normal geometric relationship, and the edge corresponds to the geometric relationship determined between the first object and the second object.
[0148] When it is found that the knowledge graph including the edge for the abnormal geometric relationship lacks an edge that satisfies the following conditions, a likelihood value indicating normality can be determined for the first object and the second object: the edge connects the first node and the second node and the edge represents the geometric relationship determined between the first object and the second object.
[0149] A likelihood value indicating abnormality can be determined for a first object and a second object when an edge is found in a knowledge graph that satisfies the following conditions: the edge connects a first node and a second node and the edge represents an abnormal geometric relationship, and the edge corresponds to the geometric relationship determined between the first object and the second object.
[0150] When it is found that the knowledge graph including the edge for the normal geometric relationship is missing an edge that satisfies the following conditions, a likelihood value indicating abnormality can be determined for the first object and the second object: the edge connects the first node and the second node and the edge represents the geometric relationship determined between the first object and the second object.
[0151] A second embodiment includes evaluating rules from a rule set for the pair (i, j) of identified objects, determining their likelihood values, and determining a likelihood from these values.
[0152] Afterwards, execute step 618.
[0153] Step 618 includes classifying the likelihood using a classifier that indicates abnormality or normality based on the likelihood.
[0154] Optionally, step 618 includes classifying the likelihood and the semantic similarity together using a classifier that indicates abnormality or normality based on the likelihood and the semantic similarity.
[0155] Afterwards, execute step 620.
[0156] Step 620 includes detecting abnormality or normality based on the likelihood.
[0157] The second embodiment includes querying the knowledge graph for pairs of identified objects (i, j), and additionally checking the object pairs (i, j) and their geometric relationships using rules. For example, the rules are learned or expert-defined. The rules can be defined by experts or learned via ILP (Inductive Logic Programming) by training a model with positive and negative examples from the scene.
[0158] In particular, the detected result (ie, abnormality or normality) may be output via the output terminal 110 .
[0159] Optionally, the operation of the device 100 , for example, the action of the device 100 , is controlled according to the detected result.
[0160] For example, a two-dimensional object detection model is used to determine the position as a two-dimensional position x, y, or a three-dimensional object detection model is used to determine the position as a three-dimensional position x, y, z.
[0161] For example, the positions of a first object i and a second object j are determined in an image k.
[0162] Using the knowledge graph or rules including the first object i and the second object j, determine the likelihood of the first object i and the second object j in the geometric relationship r
[0163] In one case, by checking the triple<i,r,j> Whether it exists in the knowledge graph or rule set to determine the likelihood
[0164] For example, given an object pair of first object i = pedestrian and second object j = street, and a geometric relationship r = above ... is detected between the first object i and the second object j based on image k, if <pedestrian, above ..., street> is in the knowledge graph, the result is the likelihood otherwise
[0165] For a given rule set, in this case, if the rule "pedestrian, street → above" is in the rule set, the result can be the likelihood otherwise
[0166] The knowledge graph or rules can include two-dimensional or three-dimensional geometric relationships.
[0167] The 3D object detection model can be used to determine the 3D geometric relationship between two objects, and to query the 3D geometric relationship from the knowledge graph.
[0168] For a pedestrian and a traffic light, an example of a result of the query may be a likelihood that indicates a normal relationship of a pedestrian depicted in a digital image behind a traffic light.
[0169] For example, semantic similarity is determined by relying on the cosine similarity of vectors corresponding to nodes representing objects in the knowledge graph.
[0170] Semantic similarity and / or likelihood are determined based on image-level features.
[0171] A classifier can be trained for anomaly detection using a supervised anomaly classification method, where abnormal and normal data are labeled. In one example, a classifier is trained as an anomaly detector to determine a likelihood and compare it to an abnormality threshold. The classifier can be trained to detect normality if the likelihood exceeds the threshold, and abnormality otherwise. The classifier can be trained to detect abnormality if the likelihood exceeds the threshold, and normality otherwise.
[0172] A classifier can be trained for normality detection using a supervised abnormality classification method, where abnormal and normal data are labeled. In one example, a classifier is trained as a normality detector, determining a likelihood and comparing it to an abnormality threshold. The classifier can be trained to detect normality if the likelihood exceeds the threshold, and abnormality otherwise. The classifier can be trained to detect abnormality if the likelihood exceeds the threshold, and normality otherwise.
[0173] Classifiers can include decision trees or decision forests, neural networks, or logistic regression models.
[0174] For example, in the manufacturing field, a classifier can be trained and used to classify the geometric relationships of product components as normal or abnormal. Based on the classification results, the method can be used to distinguish whether the detected geometric relationships of the components represent normal or abnormal geometric relationships of the components. For example, if an abnormal geometric relationship is detected, the method can identify a problem with the product. In particular, when products are manufactured automatically, the method can include automatically marking or disposing of the product if a problem is identified.
[0175] For example, a problem is identified when it is recognized through the geometrical relationship between objects that these objects are incorrectly positioned on a printed circuit board (PCB board).< / stop>
Claims
1. A computer-implemented method for processing digital images for abnormality or normality detection, characterized in that The method includes: providing (502, 602) the digital image; determining (508, 608) a geometric relationship between objects depicted in the digital image based on the digital image; providing (510, 610, 612) knowledge about normal and / or abnormal geometric relationships between objects; determining (514, 616) a likelihood indicating normal or abnormal geometric relationships between objects in the digital image based on the geometric relationships between the objects and the knowledge; and detecting (518, 620) abnormality or normality based on the likelihood.
2. The method according to claim 1, characterized in that Determining (514) the likelihood includes determining, for a plurality of pairs of objects depicted in the digital images, pairwise likelihood values, and determining the likelihood based on the likelihood values.
3. The method according to claim 2, characterized in that Determining (514) the likelihood based on the likelihood values includes: determining the likelihood based on a weighted sum of the likelihood values; determining the likelihood based on a minimum value among the likelihood values; and determining the likelihood based on a maximum value among the likelihood values.
4. The method according to claim 2 , wherein: The method comprises determining (508) the geometric relationship between the first object and the second object, in particular based on a scene graph, wherein the knowledge comprises a knowledge graph defining allowed and / or disallowed relationships, wherein determining (514) the likelihood value for the first object and the second object comprises determining the likelihood value for indicating normality when it is found that the geometric relationship satisfies the knowledge about allowed relationships from the knowledge graph or violates the knowledge about disallowed relationships from the knowledge graph, or determining the likelihood value for indicating abnormality when it is found that the geometric relationship violates the knowledge about allowed relationships from the knowledge graph or satisfies the knowledge about disallowed relationships from the knowledge graph.
5. The method according to claim 1, wherein The method comprises determining (608) the geometric relationship between a first object and a second object, wherein the knowledge comprises a rule for determining whether the first object and the second object are in a normal geometric relationship or an abnormal geometric relationship, wherein determining (616) the likelihood value for the first object and the second object comprises determining the likelihood value for indicating normality when the first object and the second object are found to be in a normal geometric relationship according to the rule, or determining the likelihood value for indicating abnormality when the first object and the second object are found to be in an abnormal geometric relationship according to the rule.
6. The method according to any one of the preceding claims, characterized in that The method includes classifying the likelihood using a classifier (516, 618), the classifier indicating abnormality or normality based on the likelihood.
7. The method according to any one of claims 1 to 5, characterized in that The method includes determining (512, 614) semantic similarities between objects depicted in the digital images, wherein the method includes classifying (516, 618) the likelihoods and the semantic similarities using a classifier that indicates abnormality or normality based on the likelihoods and the semantic similarities.
8. The method according to any one of the preceding claims, characterized in that Determining (508, 608) the geometric relationship includes determining a position of the object, determining a scene graph based on the position, and determining the geometric relationship based on the scene graph.
9. A device (100) for processing digital images for abnormality or normality detection, characterized in that The device (100) comprises at least one processor (102) and at least one memory (104), wherein the at least one processor (102) is configured to execute instructions which, when executed by the at least one processor (102), cause the device (100) to perform the method according to one of claims 1 to 8, and wherein the at least one memory (104) is configured to store the instructions.
10. A computer program, characterized in that The computer program comprises instructions which, when executed by a computer, cause the computer to perform the method according to one of claims 1 to 8 .