Processing digital images for the detection of abnormalities or normalities

Abstract

An apparatus and computer-implemented method for processing a digital image for detecting abnormalities or normalities includes providing a digital image (502), determining geometric relationships between objects depicted in the digital image (508) dependent on the digital image, providing knowledge about normal and / or abnormal geometric relationships between the objects (510), determining likelihoods indicative of normal or abnormal geometric relationships between the objects in the digital image dependent on the geometric relationships between the objects and the knowledge (514), and detecting abnormalities or normalities dependent on the likelihoods (518).

Description

[Technical Field]

[0001] background The present invention relates to a method and apparatus for processing digital images for the detection of abnormalities or normalities. [Background technology]

[0002] Biase, GD, Blum, H., Siegwart, R., Cadena, C.: Pixel-wise anomaly detection in complex driving scenes. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021. pp. 16918-16927. Computer Vision Foundation / IEEE (2021) discloses determining whether a given image is anomalous at the pixel level.

[0003] Eiter, T., Kaminski, T.: Exploiting contextual knowledge for hybrid classification of visual objects. In: JELIA. Lecture Notes in Computer Science, vol. 10021, pp. 223-239 (2016) discloses classifying images based on manually specified constraints, ensuring that none of the constraints are violated. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Biase, GD, Blum, H., Siegwart, R., Cadena, C.: Pixel-wise anomaly detection in complex driving scenes. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021. pp. 16918-16927. Computer Vision Foundation / IEEE (2021) [Non-patent document 2] Eiter, T., Kaminski, T.: Exploiting contextual knowledge for hybrid classification of visual objects. In: JELIA. Lecture Notes in Computer Science, vol. 10021, pp. 223-239 (2016) Summary of the Invention [Means for solving the problem]

[0005] Disclosure of the Invention A computer-implemented method for processing a digital image for detecting anomalies or normalities includes providing a digital image, determining geometric relationships between objects shown in the digital image depending on the digital image, providing knowledge about normal and / or abnormal geometric relationships between the objects, determining a likelihood indicating normal or abnormal geometric relationships between the objects in the digital image depending on the geometric relationships between the objects and the knowledge, and detecting anomalies or normalities depending on the likelihood. The geometric relationships between the objects provide a good basis for detecting anomalies. This information is available from the knowledge used to improve the detection of anomalies or normalities.

[0006] Determining the likelihood may include determining a likelihood value for each of a plurality of object pairs shown in the digital image and determining the likelihood in dependence on the likelihood values, thereby providing a likelihood value for the image based on the pairwise values ​​for the objects shown in the image, thereby further improving detection.

[0007] Determining the likelihood depending on the likelihood values ​​may include determining the likelihood depending on a weighted sum of the likelihood values, depending on a minimum value of the likelihood values, and depending on a maximum value of the likelihood values.

[0008] The method may include, inter alia, relying on a scene graph to determine a geometric relationship between a first object and a second object, where the knowledge includes a knowledge graph defining allowed and / or disallowed geometric relationships, and determining likelihood values ​​for the first object and the second object includes determining that the likelihood value indicates normality if the geometric relationship is found to conform to the knowledge of allowed relationships from the knowledge graph or to violate the knowledge of disallowed relationships from the knowledge graph, or determining that the likelihood value indicates abnormality if the geometric relationship is found to violate the knowledge of allowed relationships from the knowledge graph or to conform to the knowledge of disallowed relationships from the knowledge graph. This integrates the knowledge graph to further improve detection. The scene graph presents the geometric relationships of all objects, and the knowledge graph defines which of these geometric relationships are normal or abnormal.

[0009] The method may include determining a geometric relationship between a first object and a second object, wherein the knowledge includes rules for determining whether the first object and the second object are in a normal geometric relationship or an abnormal geometric relationship, and determining likelihood values ​​for the first object and the second object includes determining that the likelihood value is indicative of normality if the first object and the second object are found to be in a normal geometric relationship according to the rules, or determining that the likelihood value is indicative of abnormality if the first object and the second object are found to be in an abnormal geometric relationship according to the rules, thereby integrating the rules to further improve detection.

[0010] The method may include classifying the likelihood using a classifier that indicates abnormality or normality depending on the likelihood.

[0011] The method may include determining a semantic similarity between objects depicted in the digital image, and classifying the likelihood and semantic similarity using a classifier that indicates abnormality or normality depending on the likelihood and semantic similarity, whereby the semantic similarity and likelihood are aggregated, and the additional information provided by the semantic similarity further improves detection.

[0012] Determining the geometric relationships may include determining positions of objects, determining a scene graph depending on the positions, and determining the geometric relationships depending on the scene graph. The positions reliably indicate the geometric relationships between the objects.

[0013] An apparatus for processing digital images for the detection of abnormalities or normalities comprises at least one processor and at least one memory, the at least one processor configured to execute instructions that, when executed by the at least one processor, cause the apparatus to perform the method, and the at least one memory configured to store the instructions.

[0014] The computer program comprises instructions for causing a computer to carry out the method when the computer program is executed by a computer.

[0015] Further advantageous embodiments can be derived from the following description and drawings. [Brief explanation of the drawings]

[0016] [Figure 1] 1 shows a schematic diagram of an apparatus for processing digital images; [Figure 2] FIG. 1 shows a schematic diagram of a normal situation. [Figure 3] FIG. 1 is a schematic diagram illustrating an abnormal situation. [Figure 4] FIG. 1 shows a schematic representation of a printed circuit board containing different objects. [Figure 5] 1 is a flowchart of a first embodiment of a method for processing a digital image. [Figure 6] 4 is a flowchart of a second embodiment of a method for processing a digital image. DETAILED DESCRIPTION OF THE INVENTION

[0017] FIG. 1 shows a schematic diagram of an apparatus 100 for processing digital images.

[0018] The device 100 comprises at least one processor 102 and at least one memory 104 .

[0019] The device 100 comprises an interface 106 for a sensor 108 and / or the sensor 108. In the example shown in FIG.

[0020] The sensor 108 may be, for example, a camera, a radar sensor, a LiDAR sensor, a motion sensor, an infrared sensor, or an ultrasonic sensor.

[0021] The sensors 108 are configured to capture digital images or to reconstruct digital images using geometric information captured by the sensors 108. For example, LiDAR or radar points are processed to reconstruct a digital image of a driving scene. In one example, the sensors 108 are configured to capture sensor data from one or more sensors 108, for example, to determine a digital image of an environment of the device 100. The digital image may be determined by the device 100 depending on the sensor data.

[0022] The digital image represents visual data, which may be captured or reconstructed by a sensor 108, such as a camera, radar, LiDAR, or ultrasonic sensor.

[0023] The device 100 may be configured to detect objects in digital images. In one example, the device 100 is configured to detect objects from a set of digital images captured by the sensor 108.

[0024] The apparatus 100 may be configured to detect objects using an object detection model, such as an R-CNN, YOLO, CenterNet, or DETR model. The object detection model applied to the test data may be trained using training data from a similar distribution. The object detection model applied to the test data may be trained based on other training data or may be applied directly to the test data (zero-shot transfer).

[0025] The device 100 may be configured to detect classes of objects in digital images. In one example, the device 100 is configured to detect classes from a collection of digital images captured by the sensor 108.

[0026] Output from the object detection model may include identified objects such as cars, pedestrians, and traffic lights. Output from the object detection model may include the location of the object, such as xmin, xmax, ymin, and ymax of a two-dimensional bounding box, or the location (x, y, z) of a three-dimensional central box with respective box dimensions x, y, and z. Output from the object detection model may include a classification score for the object.

[0027] The output from the object detection model can be further used to generate a scene graph.

[0028] The scene graph is determined, for example, as disclosed in "Scene Graph Generation: A Comprehensive Survey" https: / / arxiv.org / pdf / 2201.00443.pdf.

[0029] The present disclosure relates to the problem of semantic anomaly detection for determining whether a particular measured scene or scenario is normal or anomalous. Given some input data, e.g., a collection of images, the device 100 is configured to determine whether the input data exhibits realistic or normal geometric relationships of objects or unrealistic or abnormal geometric relationships.

[0030] As an example, the geometric relationship of objects, such as one car driving in front of another, corresponds to a normal situation.

[0031] FIG. 2 shows a schematic representation of a normal situation where a first vehicle 202 is in front of a second vehicle 204 .

[0032] As an example, a geometric relationship in which one vehicle is at least partially obscured by another vehicle is considered an abnormal situation in this disclosure.

[0033] FIG. 3 shows a schematic representation of an abnormal situation in which a first vehicle 302 is partially obscured by a second vehicle 304 .

[0034] The problems considered are important and relevant in many applications, for example in autonomous driving or in the visual inspection of products assembled by robots.

[0035] For example, in the field of autonomous driving, the device 100 may be or be part of an autonomous vehicle configured to reliably distinguish between normal geometric relationships of objects and abnormal geometric relationships of objects.

[0036] In an adversarial attack, an attacker could place a sticker on the back of a car in an attempt to cause device 100 to misclassify the sticker as a <Stop> sign.

[0037] The constraints to mitigate this misclassification are as follows: <-locatedIn(X,Y),type(X,“StopSign”),type(Y,“Car”)

[0038] This means that a sticker misclassified as a <Stop> sign would violate the constraint LocatedIn(StopSign, Car).

[0039] The apparatus 100 is configured to base its determination on a normal geometric relationship of objects. In one example, the apparatus 100 is configured to detect a scene having an unusual geometric relationship of objects as a normal scene. The apparatus 100 is configured to detect a scene including a vehicle carrier loaded with a vehicle as an object as a normal scene. In one example, the apparatus 100 is configured to detect a scene having an unusual geometric relationship of objects as an abnormal scene. In one example, the apparatus 100 is configured to determine whether the unusual geometric relationship of objects indicates a normal scene or an abnormal scene.

[0040] For example, in the manufacturing field, the device 100 may be an assembly or part thereof configured to detect geometric relationships between parts of a product, particularly an automated manufactured product. The device 100 may be configured to determine whether the detected geometric relationships between the parts indicate a normal geometric relationship between the parts or an abnormal geometric relationship between the parts. The device 100 may be configured, for example, to detect that the product includes a plastic top and a metal bottom. The device 100 may be configured, for example, to detect that this geometric relationship is abnormal for a particular electronic control unit architecture. The device 100 may be configured to identify a problem with the product when the abnormal geometric relationship is detected.

[0041] 4 shows a schematic representation of a printed circuit board 400, or PCB board 400, that includes different objects, such as LEDs L1,...,L8, resistors R1,...,R17, and capacitors C1,...,C6, located at specific locations.

[0042] The device 100 is configured to detect and classify a variety of different objects.

[0043] The device 100 is configured to evaluate knowledge graphs and / or expert-defined rules that specify which objects are expected to be where relative to one another.

[0044] 4, for example, the knowledge graph and / or expert-defined rules define that the resistor is to the right of the LED. If the expected placement of an object is not detected, this is likely an outlier, i.e., a part manufactured with a defect, which can be identified by the classifier.

[0045] The device 100 in one example comprises a classifier configured to detect abnormalities or normalities.

[0046] In the example shown in FIG. 4, the apparatus 100 may be configured to detect outliers, i.e., parts manufactured with defects.

[0047] The device 100 may include an output unit 110. The output unit 110 is configured to, for example, output a result of the detection of normality or abnormality. The output unit 110 may be configured to control the operation of the device 100, for example, an action taken by the device 100.

[0048] Detecting anomalies in images can often be challenging due to the very small number of anomalous images available to meaningfully train a detector. Models trained with sufficient data, e.g., open-world data like ChatGPT, cannot be adapted to embedded devices or devices mounted on vehicles. Auxiliary approaches based on logical rules could compensate for this. Furthermore, without auxiliary extrinsic information, the quality of the detector depends heavily on the training data.

[0049] A method for detecting anomalies or normalities can use outputs from an object detection model that include identified objects and object locations.

[0050] Detection is improved by injecting exogenous information via rules or knowledge graphs.

[0051] FIG. 5 shows a flow chart of a corresponding first embodiment of a method for processing a digital image.

[0052] The method includes step 502 .

[0053] In step 502, a digital image is provided.

[0054] The image is provided, for example, via the interface 106. The image is captured, for example, by the sensor 108.

[0055] Then, step 504 is executed.

[0056] In step 504, objects shown in the digital image are detected.

[0057] In this example, multiple objects are detected.

[0058] Thereafter, in step 506, a plurality of objects of the plurality of objects are identified, and in step 508, the locations and scene graphs of the objects are determined.

[0059] Step 508 involves relying on the scene graph to determine geometric relationships between objects shown in the digital image.

[0060] In one example, determining the geometric relationship includes determining the positions of objects and determining the geometric relationship depending on the positions of the objects.

[0061] As an example, the geometric relationship between a first object i and a second object j is determined.

[0062] Then, step 510 is performed.

[0063] Step 510 involves providing knowledge about normal and / or abnormal geometric relationships between objects.

[0064] In a first embodiment, providing knowledge includes providing a knowledge graph or a set of rules.

[0065] As an example, a rule may be configured to determine whether a first object i and a second object j are in a permissible or impermissible geometric relationship. The rules may be defined by an expert or may be learned via ILP (inductive logic programming) by training a model with positive and negative examples from the scene.

[0066] The knowledge graph and set of rules define the allowed or disallowed geometric relationships for the objects.

[0067] A knowledge graph includes nodes that represent objects and edges that represent relationships between the objects.

[0068] According to one example, a knowledge graph includes edges connecting a first node representing a first object to a second node representing a second object, the edges corresponding to geometric relationships between the first and second objects.

[0069] According to one example, the knowledge graph lacks an edge connecting a first node to a second node, which can correspond to a lack of information about the geometric relationship between the first object and the second object.

[0070] The absence of an edge can correspond to a geometric relationship between the first object and the second object. For example, in a knowledge graph that uses edges to capture normal geometric relationships, the absence of an edge can indicate an abnormal geometric relationship. For example, in a knowledge graph that uses edges to capture abnormal geometric relationships, the absence of an edge can indicate a normal geometric relationship.

[0071] As an example, the knowledge graph includes a first node representing a first object i and a second node representing a second object j.

[0072] Optionally, the method includes step 512 .

[0073] Step 512 involves determining the semantic similarity between objects shown in the digital images.

[0074] Determining the semantic similarity may include determining a number of pairs of objects that are shown in the digital image.

[0075] Determining the semantic similarity may include determining a semantic similarity value for each pair of the plurality of pairs.

[0076] Determining the semantic similarity may include determining the semantic similarity depending on the semantic similarity values.

[0077] The semantic similarity is determined, for example, depending on a weighted sum of the semantic similarity values.

[0078] The semantic similarity is determined, for example, depending on the minimum value of the semantic similarity values.

[0079] The semantic similarity is determined, for example, depending on the maximum value of the semantic similarity values.

[0080] Step 514 is then executed.

[0081] Step 514 involves determining likelihoods of normal or abnormal geometric relationships between objects in the digital image depending on the geometric relationships between the objects and knowledge.

[0082] Determining the likelihood may include determining a number of object pairs that are represented in the digital image.

[0083] Determining the likelihood may include determining a likelihood value for each pair of the plurality of pairs.

[0084] Determining the likelihood may include determining the likelihood dependent on the likelihood values.

[0085] The likelihood is determined, for example, depending on a weighted sum of the likelihood values.

[0086] The likelihood is determined, for example, depending on the smallest value among the likelihood values.

[0087] The likelihood is determined, for example, depending on the maximum value of the likelihood values.

[0088] The likelihood values ​​for the first object and the second object may be determined to be indicative of normality if a rule is found in the set of rules that determines that the first object and the second object are in a normal geometric relationship.

[0089] The likelihood values ​​for the first object and the second object may be determined to be indicative of an abnormality if a rule is found in the set of rules that determines that the first object and the second object are in an abnormal geometric relationship.

[0090] The likelihood values ​​for the first object and the second object may be determined to be indicative of normality if it is found that the knowledge graph indicates that the first object and the second object are in a normal geometric relationship.

[0091] The likelihood values ​​for the first object and the second object may be determined to indicate an anomaly if the knowledge graph is found to indicate that the first object and the second object are in an abnormal geometric relationship.

[0092] The likelihood values ​​for the first object and the second object may be determined to be indicative of normality if an edge is found to exist in the knowledge graph connecting the first node and the second node, representing a normal geometric relationship, and corresponding to the determined geometric relationship between the first object and the second object.

[0093] The likelihood values ​​for the first object and the second object may be determined to be indicative of normality when a knowledge graph including an edge for an abnormal geometric relationship is found to be missing an edge connecting the first node and the second node and representing the determined geometric relationship between the first object and the second object.

[0094] The likelihood values ​​for the first object and the second object may be determined to indicate an anomaly if an edge is found to exist in the knowledge graph connecting the first node and the second node, representing an abnormal geometric relationship, and corresponding to the determined geometric relationship between the first object and the second object.

[0095] The likelihood values ​​for the first object and the second object may be determined to indicate an anomaly if an edge connecting the first node and the second node in a knowledge graph containing edges for normal geometric relationships and representing a determined geometric relationship between the first object and the second object is found to be absent.

[0096] The first embodiment involves querying pairs (i,j) of identified objects from the knowledge graph, determining likelihood values ​​for those pairs, and determining likelihoods from those values.

[0097] Step 516 is then executed.

[0098] Step 516 includes classifying the likelihood using a classifier that indicates abnormality or normality depending on the likelihood.

[0099] Optionally, step 516 includes classifying the likelihood and semantic similarity together using a classifier that indicates abnormality or normality depending on the likelihood and the semantic similarity.

[0100] Step 518 is then executed.

[0101] Step 518 involves detecting abnormality or normality depending on the likelihood.

[0102] The detected result, i.e., abnormality or normality, may be output in particular via the output unit 110 .

[0103] Optionally, the operation of the device 100, for example an action taken by the device 100, is controlled depending on the detected result.

[0104] Detection is improved by injecting extrinsic information via expert-defined rules.

[0105] FIG. 6 shows a flow chart of a corresponding second embodiment of a method for processing a digital image.

[0106] The method includes step 602 .

[0107] In step 602, a digital image is provided.

[0108] The image is provided, for example, via the interface 106. The image is captured, for example, by the sensor 108.

[0109] Then, step 604 is performed.

[0110] In step 604, objects shown in the digital image are detected.

[0111] In this example, multiple objects are detected.

[0112] Thereafter, in step 606, a plurality of objects of the plurality of objects are identified, and in step 608, the locations and scene graphs of the objects are determined.

[0113] Step 608 involves determining the scene graph depending on the location.

[0114] Step 608 involves relying on the scene graph to determine the geometric relationships between objects shown in the digital image.

[0115] In one example, determining the geometric relationship includes determining the positions of objects and determining the geometric relationship depending on the positions of the objects.

[0116] As an example, the geometric relationship between a first object i and a second object j is determined.

[0117] Steps 610 and 612 are then performed to provide knowledge about normal and / or abnormal geometric relationships between objects.

[0118] Step 610 includes providing a knowledge graph.

[0119] Step 610 includes providing at least one rule.

[0120] The rule determines whether the first object and the second object have a normal or abnormal geometric relationship.

[0121] The likelihood values ​​for the first object and the second object are determined to be indicative of normality if the first object and the second object are found to be in a normal geometric relationship according to the rules.

[0122] The likelihood values ​​for the first object and the second object are determined to indicate an abnormality if the first object and the second object are found to be in an abnormal geometric relationship according to the rule.

[0123] As a result, it can be determined that the likelihood value indicates normality by conforming to a rule for the first object and the second object from a set of rules that indicate normal geometric relationships.

[0124] A rule for the first object and the second object among a set of rules indicating an abnormal geometric relationship is met, and as a result, it can be determined that the likelihood value indicates an abnormality.

[0125] Optionally, the method includes step 614 .

[0126] Step 614 involves determining the semantic similarity between objects shown in the digital images.

[0127] Determining the semantic similarity may include determining a number of pairs of objects that are shown in the digital image.

[0128] Determining the semantic similarity may include determining a semantic similarity value for each pair of the plurality of pairs.

[0129] Determining the semantic similarity may include determining the semantic similarity depending on the semantic similarity values.

[0130] The semantic similarity is determined, for example, depending on a weighted sum of the semantic similarity values.

[0131] The semantic similarity is determined, for example, depending on the minimum value of the semantic similarity values.

[0132] The semantic similarity is determined, for example, depending on the maximum value of the semantic similarity values.

[0133] Step 616 is then executed.

[0134] Step 616 involves determining likelihoods of normal or abnormal geometric relationships between objects in the digital image depending on the geometric relationships between the objects and knowledge.

[0135] Determining the likelihood may include determining a number of object pairs that are represented in the digital image.

[0136] Determining the likelihood may include determining a likelihood value for each pair of the plurality of pairs.

[0137] Determining the likelihood may include determining the likelihood dependent on the likelihood values.

[0138] The likelihood is determined, for example, depending on a weighted sum of the likelihood values.

[0139] The likelihood is determined, for example, depending on the smallest value among the likelihood values.

[0140] The likelihood is determined, for example, depending on the maximum value of the likelihood values.

[0141] The likelihood values ​​for the first object and the second object may be determined to be indicative of normality if an edge is found to exist in the knowledge graph connecting the first node and the second node, representing a normal geometric relationship, and corresponding to the determined geometric relationship between the first object and the second object.

[0142] The likelihood values ​​for the first object and the second object may be determined to be indicative of normality when a knowledge graph including an edge for an abnormal geometric relationship is found to be missing an edge connecting the first node and the second node and representing the determined geometric relationship between the first object and the second object.

[0143] The likelihood values ​​for the first object and the second object may be determined to indicate an anomaly if an edge is found to exist in the knowledge graph connecting the first node and the second node, representing an abnormal geometric relationship, and corresponding to the determined geometric relationship between the first object and the second object.

[0144] The likelihood values ​​for the first object and the second object may be determined to indicate an anomaly if an edge connecting the first node and the second node in a knowledge graph containing edges for normal geometric relationships and representing a determined geometric relationship between the first object and the second object is found to be absent.

[0145] A second embodiment involves evaluating rules from a set of rules for pairs of identified objects (i, j), determining likelihood values ​​for those pairs, and determining a likelihood from those values.

[0146] Step 618 is then executed.

[0147] Step 618 includes classifying the likelihood using a classifier that indicates abnormality or normality depending on the likelihood.

[0148] Optionally, step 618 includes classifying the likelihood and semantic similarity together using a classifier that indicates abnormality or normality depending on the likelihood and the semantic similarity.

[0149] Step 620 is then executed.

[0150] Step 620 involves detecting abnormality or normality depending on the likelihood.

[0151] A second embodiment involves querying the identified object pairs (i, j) from the knowledge graph and additionally checking the object pairs (i, j) and their geometric relationships via rules. The rules may be, for example, learned or expert-defined. The rules may be expert-defined or learned via ILP (inductive logic programming) by training a model with positive and negative examples from the scene.

[0152] The detected result, i.e., abnormality or normality, may be output in particular via the output unit 110 .

[0153] Optionally, the operation of the device 100, for example an action taken by the device 100, is controlled depending on the detected result.

[0154] The position may be determined, for example, as a two-dimensional position x,y using a two-dimensional object detection model, or as a three-dimensional position x,y,z using a three-dimensional object detection model.

[0155] For example, the positions of a first object i and a second object j in an image k are determined.

[0156] the likelihood of a first object i and a second object j being in a geometric relationship r

number

[0157] In one case, triples in a knowledge graph or set of rules<i,r,j> By checking whether there exists

number

[0158] For example, given an object pair consisting of a first object i=pedestrian and a second object j=street, and a geometric relation r=above that is detected between the first object i and the second object j based on image k, the result can be represented in the knowledge graph as<pedestrian,above,street> If exists, the likelihood

number

number

[0159] Given a set of rules, the outcome in this case is that if pedestrian,street→above exists in the set of rules, then likelihood

number

number

[0160] The knowledge graph or rules may include two-dimensional or three-dimensional geometric relationships.

[0161] The 3D object detection model can be used to determine the 3D geometric relationship between two objects, and the 3D geometric relationship can be queried from the knowledge graph.

[0162] An example of the result of this query for pedestrians and traffic lights may be the likelihood of a normal relationship for a pedestrian to be shown behind a traffic light in a digital image.

[0163] Semantic similarity is determined, for example, by relying on the cosine similarity of vectors corresponding to nodes representing objects in the knowledge graph.

[0164] The semantic similarity and / or likelihood is determined based on image-level features.

[0165] For anomaly detection, a classifier can be trained using a supervised anomaly classification approach, in which abnormal and normal data are labeled. In one example, the classifier is trained as an anomaly detector to determine a likelihood and compare the likelihood to an anomaly threshold. The classifier may be trained to detect normality if the likelihood exceeds the threshold, or to detect anomaly otherwise. The classifier may be trained to detect anomaly if the likelihood exceeds the threshold, or to detect normality otherwise.

[0166] For normality detection, a classifier can be trained using a supervised anomaly classification approach, in which abnormal and normal data are labeled. In one example, the classifier is trained as a normality detector to determine a likelihood and compare this likelihood to an anomaly threshold. The classifier may be trained to detect normality if the likelihood exceeds the threshold, or to detect abnormality otherwise. The classifier may be trained to detect an anomaly if the likelihood exceeds the threshold, or to detect normality otherwise.

[0167] The classifier may include a decision tree or forest, a neural network, or a logistic regression model.

[0168] For example, in the field of manufacturing, a classifier can be trained and the method can be used to classify geometric relationships between parts of a product as normal or abnormal. The method can be used to determine, depending on the results of the classification, whether the detected geometric relationships between the parts indicate a normal geometric relationship between the parts or an abnormal geometric relationship between the parts. The method can, for example, identify a problem with the product when an abnormal geometric relationship is detected. In particular, when the product is manufactured using automation, the method can include automatically labeling the product or automatically disposing of the product when a problem with the product is identified.

[0169] For example, a problem is identified when the geometric relationships between objects identify that the objects are incorrectly placed on a printed circuit board, ie, a PCB board.

Claims

1. 1. A computer-implemented method for processing a digital image for detection of abnormalities or normalities, comprising: The method comprises: Providing a digital image (502, 602); determining (508, 608) geometric relationships between objects shown in the digital images in dependence on the digital images; Providing knowledge about normal and / or abnormal geometric relationships between objects (510, 610, 612); determining (514, 616) likelihoods indicative of normal or abnormal geometric relationships between objects in the digital image depending on the geometric relationships between the objects and the knowledge; Detecting abnormalities or normalities depending on the likelihood (518, 620); A method comprising:

2. Determining the likelihood (514) determining a likelihood value for each of a plurality of object pairs represented in the digital image; determining the likelihood dependent on the likelihood value; The method of claim 1 , comprising:

3. 3. The method of claim 2, wherein determining the likelihood depending on the likelihood values ​​(514) comprises determining the likelihood depending on a weighted sum of the likelihood values, depending on a minimum value of the likelihood values, and depending on a maximum value of the likelihood values.

4. The method comprises: determining (508) said geometric relationship between the first object and the second object, in particular relying on the scene graph; Including, the knowledge includes a knowledge graph defining permissible and / or disallowed geometric relationships; Determining 514 the likelihood values ​​for the first object and the second object includes: determining that the likelihood value is indicative of normality if the geometric relationship is found to conform to the knowledge of permissible relationships from the knowledge graph or to violate the knowledge of disallowed relationships from the knowledge graph; or determining that the likelihood value indicates an anomaly if the geometric relationship is found to violate the knowledge of permissible relationships from the knowledge graph or conform to the knowledge of disallowed relationships from the knowledge graph. The method of claim 2 or 3, comprising:

5. The method comprises: determining (608) the geometric relationship between the first object and the second object; the knowledge includes rules that determine whether the first object and the second object are in a normal geometric relationship or an abnormal geometric relationship; Determining 616 the likelihood values ​​for the first object and the second object includes: determining that the likelihood value indicates normality if the first object and the second object are found to be in a normal geometric relationship according to the rule; or determining that the likelihood value indicates an abnormality when the first object and the second object are found to be in an abnormal geometric relationship according to the rule; 5. The method of claim 1, comprising:

6. The method comprises: Classifying the likelihood using a classifier that indicates abnormality or normality depending on the likelihood (516, 618).

6. The method of claim 1, comprising:

7. The method comprises: Determining semantic similarities between objects shown in the digital images (512, 614). Including, The method comprises: classifying the likelihood and the semantic similarity using a classifier that indicates abnormality or normality depending on the likelihood and the semantic similarity (516, 618); 6. The method of claim 1, comprising:

8. Determining the geometric relationship (508, 608) includes: determining a position of the object; determining a scene graph dependent on said location; determining the geometric relationship in dependence on the scene graph; 8. The method of claim 1, comprising:

9. An apparatus (100) for processing digital images for the detection of abnormalities or normalities, comprising: The device (100) comprises: at least one processor (102); At least one memory (104); Equipped with The at least one processor (102) is configured to execute instructions that, when executed by the at least one processor (102), cause the apparatus (100) to perform the method of any one of claims 1 to 8; the at least one memory (104) is configured to store the instructions; 1. An apparatus (100) comprising:

10. 9. A computer program comprising instructions for causing a computer to carry out the method of any one of claims 1 to 8 when the computer program is run by the computer.

Images