Image classification method and apparatus, and method and apparatus for improving the training of an image classifier.

Abstract

To provide an image classification method and an apparatus, and a method and an apparatus for improving training of an image classifier.SOLUTION: An image classification method includes the steps of: extracting a logic program being a symbolic approximation of outputs of kernels at an extraction layer of a CNN trained to classify features in images; deriving kernel-based classification rules; forward-propagating pairs of images to the logic program to obtain kernel activations at the extraction layer for features in the images, where the scene in one of the pair contains a particular feature and the other does not have the feature; calculating the correlation between each kernel and each feature in feature-labeled images using the kernel activations and the features associated with the feature-labeled images; assigning to each kernel the label of the feature with which the kernel has the highest correlation; and applying the assigned kernel labels to the kernels in the rules to obtain kernel-labeled rules.SELECTED DRAWING: Figure 4

Description

Technical Field

[0004]

[0001] Embodiments relate to an image classification method and apparatus, and a method and apparatus for improving the training of an image classifier.

Background Art

[0002] The field of neural-symbolic integration relates to the relationship between symbolic models, such as propositional logic programs, and neural networks. In other words, this relates to explainable artificial intelligence with respect to neural networks. The main part of this work includes tasks of converting knowledge from one form of representation to another, for example, converting a logical program into a neural network that can be inductively trained by observing training samples, or converting the weights of a trained neural network into a logical program to make the decisions made by the neural network more easily understandable by humans. Rules generally describe how the relationships between individual features (represented by individual neurons) contribute to the activation of individual classes or the observation of other features, as evidenced by the activation of the corresponding neurons.

[0003] In recent years, convolutional neural networks (CNNs) have become a common method for performing image classification. Most of the effort to explain the behavior of CNNs involves visualizing the regions of an input image that are most important or relevant to a given classification. While useful, there are some limitations. · Such explanations are only local in that they explain individual samples and not the model as a whole (a "global" explanation). · They do not provide sufficient insight into the relationships between features, as was the case with early neural-symbolic models. · They do not directly provide sufficient insight into the internal workings of CNNs.

[0004] Regarding the last point, there are several ways to visualize what individual kernels respond to. A simple way to do this is to visualize the kernel output (i.e., create an image corresponding to the kernel output) and use this to generate a mask over the original image, but a more sophisticated method is to backpropagate the signal from the kernel, through the weights and activations that led to the kernel, and back to the input image. These allow for the decomposition of the model for a better understanding of its internal workings, but these methods still do not provide insight into the interactions between features represented by other kernels.

[0005] There are ways to describe the classification decisions of a CNN in the form of a graph or tree. These describe the relationships between different features. However, these do not allow for the representation of negation symbols (e.g., ¬A as the opposite of A). Methods have also been proposed to train kernels so that they can be interpreted. However, using this method, the symbolic concepts represented by kernels belong to only one class. It also assumes that the CNN was trained in a particular way. Another way to describe the behavior of a CNN is to train a prototype layer, which represents the input in terms of components similar to the training input, with each component represented by a specific kernel within the prototype layer. However, this again assumes a particular training method and a particular type of layer. However, there may be situations where we want to describe an arbitrary CNN, not just one with a particular architecture and / or one trained in any particular way.

[0006] EP3291146 proposes a method for extracting logic programs from convolutional neural networks so that these logic programs can be considered to explain the behavior of the corresponding CNN. This overcomes the problems enumerated in the previous paragraph; namely, it allows for symbol negation, enables symbols represented by kernels to be associated with multiple classes, and does not assume any specific training methods or architectures beyond those common in CNNs (although such training methods may still improve accuracy).

[0007] As shown in Figure 1 of the attached diagram, in this method, each kernel in the CNN is first mapped to a single value considered to be the activation value of that kernel by applying an L1 or L2 norm to its activation map, and then quantized by applying a binary threshold to that activation.

[0008] Figure 2 of the attached diagram shows an exemplary CNN M and the corresponding extracted logic program M', extracted using the method described above. A logic program that approximates the behavior of CNN M is extracted by first applying a quantization function to all kernels involved in the program, and then applying a decision tree extractor to each binarized kernel and its input to discover logical rules that describe the conditions under which each of these kernels "activates". The aggregation of these rules constitutes the overall description of the CNN. In Figure 2, the program is extracted only to describe the third layer ("extraction layer"), but it can contain multiple layers. Any convolutional layer preceding the extraction layer remains in M', thereby allowing kernel activation for quantization to binary truth, as in Figure 1.

[0009] However, the extracted rules are meaningless without labels assigned to the kernels, which leaves an unresolved problem. The problem of labeling convolutional kernels is a CNN-specific version of the more general "symbol grounding problem," namely, the question of the origin of the meaning of symbols.

[0010] It has been proposed that labels can be assigned by visualizing the kernel output and providing this visualization as input to a second classifier trained on a finer-grained dataset (hereinafter referred to as the "support" dataset), thereby assigning these class labels to these kernels. This is shown in Figure 3 of the attached diagram. The visualization method may be direct or a more sophisticated visualization method.

[0011] Among more sophisticated visualization methods are perturbation-based methods, which estimate feature importance based on the impact on the classifier output when features are added, removed, or modified. For example, an image may be modified by cropping a region of the image, and if this changes the network's decision regarding the input's class, that region is considered important; otherwise, it is not.

[0012] In recent years, methods for performing perturbations via inpainting have been proposed. Inpainting is a method often used to automate the generation of training data. Models are designed and / or trained to be used to generate datasets for the purpose of drawing features into images or painting over features in images, and thus training another model to detect the presence or absence of those features. However, in this case, it is used to determine the importance of features with respect to the decision-making process of the trained network, rather than (not necessarily) for training purposes. This allows for a more "practical" perturbation than simply cropping regions of an image.

[0013] Previously proposed methods for assigning labels to kernels using a support dataset are computationally expensive because both the visualization method and the kernel classifier must be applied once for each image in the support dataset and once for each kernel to which a label is assigned. This is particularly expensive for more sophisticated visualization methods that backpropagate some signal from the kernel so that it is visualized back onto the input image.

[0014] Furthermore, inpainting is an expensive process if it is not needed. For example, it is only used to generate training data if a suitable training set has not been obtained. The same applies to using it as a means of perturbation-based feature importance calculation. Using actual photographic datasets that represent the presence or absence of features in a otherwise unchanging scene would be better, if such data is available. [Overview of the project]

[0015] Therefore, it is desirable to be able to more efficiently assign meaningful labels to the kernel in the extracted rules.

[0016] According to one embodiment of the first aspect, a computer-based image classification method is provided, the method comprising: obtaining a convolutional neural network (CNN) trained to classify features in images using a training image dataset; extracting a logic program from the CNN, the logic program being a symbolic approximation of the kernel output in the extraction layer of the CNN; deriving rules from the logic program for using the kernel to explain the image classification by the CNN; obtaining a feature-labeled image dataset and a record of each feature associated with each feature-labeled image in the dataset, the images in the dataset comprising pairs of feature-labeled images, one of which is a scene containing a feature, and the other of which is the same scene without a feature; and forward-propagating the pairs of feature-labeled images through the logic program to activate the kernel in the extraction layer for features in the images. The process includes the steps of obtaining activations, calculating the correlation between each kernel in the logic program and each feature in the feature-labeled image using the obtained kernel activations and the features associated with the feature-labeled image, assigning each kernel in the logic program the label of the feature with which the kernel has the highest correlation, and applying the assigned kernel label to the kernel in the derived rule to obtain a kernel-labeled rule.

[0017] The embodiment provides a novel approach to assign symbolic labels to kernels within a convolutional neural network (CNN), allowing these labeled kernels to be manipulated by a logic program. In contrast to previous approaches that proposed classifying the output of individual kernels for each image from a labeled "support dataset" reserved for this purpose, performance is improved by requiring the kernel classifier to be applied only once per kernel after all support images have been processed.

[0018] In particular, in both the conventional methods described above and this embodiment, kernel labeling may include forward propagation of labeled training images from a support dataset and quantization of the kernel output. However, in previously proposed methods, classification must be applied to each kernel and to each support image, whereas in one embodiment, it is possible to simply annotate a table to identify which kernels were activated in that image. After all support images have been processed, classification only needs to be performed once per kernel by selecting the tag assigned to the image that most strongly correlates with the activation of that kernel as the label (e.g., during manual labeling). This is based on the assumption that a kernel whose activation changes dramatically between two images can be said to correspond to its tag / feature. To achieve this, the network is presented with at least two versions of images from the support dataset during the labeling process, one with a given feature and the other without the given feature but otherwise identical.

[0019] Therefore, the complexity of previously proposed approaches to kernel labeling is reduced because it is no longer necessary to apply the classifier once per image per kernel, which in turn reduces the computational resource requirements.

[0020] According to one embodiment of the second aspect, a computer-based method is provided for improving the training of an image classifier, the method comprising: obtaining kernel-labeled rules derived from a convolutional neural network (CNN) trained to classify features in images using a method embodying the first aspect; obtaining at least one image classification determined by the CNN for at least one image that does not form part of the training image dataset used to train the CNN or the feature-labeled image dataset used to derive the kernel-labeled rules, wherein the classification is evaluated as inaccurate; identifying a rule among the kernel-labeled rules associated with the inaccurate classification; and retraining the CNN with further training images containing features corresponding to the kernel labels of the rules associated with the inaccurate classification.

[0021] According to one embodiment of the third aspect, a computer program is provided which, when executed on a computer, causes the computer to perform a method of embodying the first and / or second aspects.

[0022] According to one embodiment of the fourth aspect, an image classification device is provided, the device having at least one memory that stores (a) a convolutional neural network (CNN) trained to classify features in images using a training image dataset, and (b) a feature-labeled image dataset and a record of each feature associated with each feature-labeled image in the dataset, wherein the images in the dataset comprise pairs of feature-labeled images, where one feature-labeled image in the pair is of a scene containing a feature, and the other feature-labeled image in the pair is of the same scene without a feature, and at least one processor connected to the memory and logical processing from the CNN stored in the memory The system includes at least one processor that extracts a program, the logic program is a symbolic approximation of the output of a kernel in the extraction layer of a CNN, derives rules from the logic program that use kernels to describe the classification of an image by a CNN, forward propagates pairs of feature-labeled images from a feature-labeled dataset stored in memory via the logic program to obtain kernel activations in the extraction layer for features in the images, calculates the correlation between each kernel in the logic program and each feature in the feature-labeled image using the obtained kernel activations and the features associated with the feature-labeled image, assigns each kernel in the logic program the label of the feature with which the kernel has the highest correlation, and applies the assigned kernel label to the kernel in the derived rules to obtain kernel-labeled rules.

[0023] According to one embodiment of the fifth aspect, an apparatus is provided for improving the training of an image classifier, the apparatus comprising: at least one memory, which stores kernel-labeled rules derived from a convolutional neural network (CNN) trained to classify features in images using an apparatus or method embodying the first aspect, and at least one image classification determined by the CNN for at least one image that does not form part of the training image dataset used to train the CNN or the feature-labeled image dataset used to derive the kernel-labeled rules, the classification of which is evaluated as inaccurate; and at least one processor, which is connected to the memory, which identifies the rules among the kernel-labeled rules that are associated with the inaccurate classification, and causes the CNN to be retrained with further training images that include features corresponding to the kernel labels of the rules associated with the inaccurate classification.

[0024] In an apparatus that embodies the first embodiment or the fourth embodiment of the method, the images in the feature-labeled image dataset may include still frames from at least one video recording. The at least one video recording may be captured by a closed-circuit television (CCTV) camera.

[0025] The manual effort required to label the support dataset can be reduced by leveraging the fact that frames from still video / CCTV cameras capture the differences between the presence, absence, or change of entities within the camera view when the background is stationary, and that kernel activations vary accordingly. The use of video from still cameras is a more efficient way to generate scene perturbations as the perturbations are natural and there is no need to manually place objects. For example, it would be cumbersome to drive a car inside and outside the camera shot to acquire "with" and "without" images to identify kernels related to cars. However, CCTV in parking lots constantly observes the arrival and departure of cars.

[0026] By capturing training frames from static video cameras as the support dataset, three drawbacks can be overcome. · Labeling frames from such videos is a less cumbersome task than labeling an equal number of images corresponding to the total number of frames across all videos, because it suffices to label instances of appearance or disappearance of features, i.e., frames only need to be annotated when objects enter or leave the scene, in contrast to manually tagging individual frames. · The lack of realism of naive perturbations such as cutting out or blurring tagged objects is avoided. · Reduces the computational load required by automatic inpainting methods for adding or removing objects. That is, the additional processing for inpainting methods that artificially add or remove tagged objects is avoided, because objects in the video enter and leave the scene spontaneously.

[0027] Nevertheless, the user may still, if desired, use in the embodiments still images and / or images or frames artificially perturbed.

[0028] The use of images from a still camera to improve performance can be relevant to anyone, in particular, who does the job of handling such hardware. For example, the embodiments may be applied to obtain an explainable classification of CCTV videos taken at airports, transportation networks, and the like.

[0029] In a method embodying the first aspect or an apparatus embodying the fourth aspect, the feature-labeled image dataset may include images annotated for semantic segmentation. The record of each feature associated with each feature-labeled image in the dataset may include a value corresponding to the total area occupied by the feature in the image.

Brief Description of the Drawings

[0030] Next, as an example, reference is made to the accompanying drawings. [Figure 1] The above-mentioned FIG. 1 is a diagram used to explain the quantization of each kernel in a CNN by a prior art method. [Figure 2] The above-mentioned FIG. 2 is a diagram used to explain the extraction of a logical program from a CNN by a prior art method. [Figure 3] The above-mentioned FIG. 3 is a diagram used to explain a prior art method of assigning labels to the kernels of a CNN. [Figure 4] It is a flowchart of a method according to an embodiment. [Figure 5] It is a flowchart of a preprocessing method used with a method according to an embodiment. [Figure 6] It is a diagram used to explain the preprocessing method of FIG. 5. [Figure 7] It is a flowchart of a process used in a method according to an embodiment. [Figure 8] This diagram shows a series of static CCTV frames and their associated kernel activations. [Figure 9] This table shows the completed existence matrix corresponding to the frame in Figure 8. [Figure 10] This table shows the completed kernel activation matrix corresponding to the frame in Figure 8. [Figure 11] Figure 8 shows a series of static CCTV frames and their associated kernel activations, represented using segmented regions. [Figure 12] This table shows the completed existence matrix and the completed kernel activation matrix corresponding to the frame in Figure 11. [Figure 13] This is a table of feature and kernel correlation values ​​corresponding to the frames in Figures 8 and 11. [Figure 14] This is a table of extracted rules interpreted according to the process shown in Figure 7. [Figure 15] This is a diagram used to explain misclassified images. [Figure 16] This is a flowchart of a method according to one embodiment. [Figure 17] This is a block diagram of a computing device suitable for performing a method according to one embodiment. [Modes for carrying out the invention]

[0031] In one embodiment described below, initial training of a CNN is performed as in the prior art, and a representative logic program is extracted. To label the atoms within the logic program, their corresponding kernels are isolated, and changes in their kernel activation are observed as images from a support dataset are sequentially classified by the CNN. For example, a binary inversion of the kernel's quantized activation when an object enters and then leaves the scene is considered evidence that the kernel is activated in response to the presence of that object.

[0032] Figure 4 shows a high-level overview of the overall methodology, which is explained below.

[0033] 1. Dataset and Preprocessing In Step 1, the problem dataset for training the CNN is obtained.

[0034] Obtaining the problem dataset The problem dataset is the original classification task that the CNN to be described will be trained to perform. Thus, the problem dataset is a set of images with their corresponding class labels, all partitioned into training, validation, and test partitions. These images may also be video frames.

[0035] In this example, additional support datasets may be obtained in step 1. Alternatively, these may be obtained at any time before step 3.

[0036] Obtaining support datasets Figure 5 shows the preprocessing pipeline for the support dataset.

[0037] The support dataset is used to label the kernel. It may be the same as the problem dataset, provided that the problem dataset meets the criteria for a support dataset as described in the following paragraph.

[0038] For each instance of a feature, the support dataset must contain instances of the scene with and without that feature, with all other features in the image remaining otherwise identical. Therefore, if the number of object instances is N, the support dataset should ideally contain at least 2 × N images, plus a "presence matrix" that identifies which images correspond to the presence or absence of the feature. Methods for obtaining such data include (but are not limited to) the following: • Objects enter / exit the scene, and frames are obtained from a still video camera (the method assumed hereafter). • Take a picture of an object in the scene, remove this object, and then take another picture. • In the case of still images, use inpainting to add or remove objects.

[0039] If a completed existence matrix is ​​not yet provided, there are several options for completing the existence matrix for the support dataset. These options include, but are not limited to, the following: Regarding the example in Figure 8, as shown in Table 1 (Figure 9), the appearance or disappearance of the feature of interest is marked with 1 or -1 for the frames in which these corresponding transitions occur (Step 1.1 in Figure 5). These intervals are automatically filled with 1 in the existence matrix (Step 1.2 in Figure 5). This is the assumed method used in the examples described later in this document. If the dataset is already annotated for semantic segmentation (whether video or still images), a value of 1 is added to the existence matrix for each object (i.e., feature) present in each frame. Alternatively, a value corresponding to the total area occupied by each object (feature) may be entered (see Figure 9, Table 3 (Figure 12)).

[0040] The frames and their corresponding existence matrices may be generated from multiple videos, in which case they are combined into a single dataset of n images / rows corresponding to one image set and one existence matrice, as shown in Figure 6 (Step 1.3 in Figure 5).

[0041] 2. Initial training for CNN In step 2, CNN M is trained on the problem dataset using the usual method.

[0042] 3. Knowledge Extraction In step 3, a logical program M', which is a quantized / symbolic approximation of M, is extracted from M according to, for example, one of the extraction methods described above (e.g., as shown in Figure 2), or any method that generates a quantized approximation of the kernel output. The rules describing the classification of the CNN are derived from the logical program. However, the derived rules do not have labels assigned to the kernel.

[0043] 4. Labeling the kernel In step 4, the kernel in the symbolic approximation M' is labeled. Figure 7 shows an overview of the kernel labeling process performed in step 4.

[0044] In step 4.1 of Figure 7, each support image is forward propagated through M' to obtain quantized kernel activations in the extraction layer, and these quantizations are recorded according to the previously generated existence matrix values ​​(Table 1 (Figure 9)), as shown in Table 2 (Figure 10).

[0045] Alternatively, if segmentation annotations are available as described above, the existence matrix may be filled with the total area occupied by the segments of each class, and the kernel activation matrix may be filled with the activation values ​​before the thresholding portion of the quantization (i.e., after the L1 or L2 norm is obtained, but before thresholding). The values ​​for the current example are shown in Table 3 (Figure 12), based on the segments shown in Figure 11.

[0046] Note that some kernels in the original CNN M may not have corresponding literals in the symbolic approximation M'. Therefore, it is not necessary to generate labels for these literals / kernels.

[0047] Next, in step 4.2, the correlation between each observed feature and each kernel is calculated, for example, according to the Phi coefficient, Spearman's rank correlation coefficient, Kendall rank correlation coefficient, or any other known method for calculating the correlation between two binary variables. Features and kernels that do not show any change in presence or activation may be excluded from this process.

[0048] In step 4.3, each kernel is assigned the label of the feature that produces the highest correlation. The symbolic approximation M' plus the newly assigned label is now referred to as M''.

[0049] In the case of segmented image datasets, labels may be assigned to kernels based on the correlation between kernel activation intensity and the area of ​​segmented regions belonging to the same label class, provided that the correlation metric used in this case can be applied to a continuous variable (e.g., Pearson or Spearman).

[0050] Therefore, the rule M' can then be translated into a term that uses the assigned kernel label.

[0051] 5. Reasoning In step 5, inference is performed. If the symbolic approximation M' is used for logical inference, the classification made by M can be explained by performing the symbolic approximation M' in parallel (as in prior art). However, we now use M'' instead of M', and therefore, as mentioned above, the atoms used in the rules / explanations use labels assigned using the proposed kernel labeling process.

[0052] During inference, live or test images are used, and if it is found that an image was misclassified by the trained CNN, the CNN is retrained with additional training images. In this case, the rules associated with the misclassification are used to determine which features should be shown in the additional training images. In particular, since the rules indicate features that lead to the activation of the relevant kernels, resulting in the misclassification of the image, retraining the CNN with more images that show the relevant features helps the retrained CNN avoid such misclassifications in the future.

[0053] In particular, as shown in the process in Figure 16, the training of an image classifier can be improved by, for a CNN trained to classify features in an image, obtaining kernel-labeled rules derived from the CNN using the method according to the above embodiment in step S161, obtaining at least one inaccurate classification of an image determined by the CNN for at least one image that does not form part of the training image dataset used to train the CNN or the feature-labeled image dataset used to derive the kernel-labeled rules in step S162, identifying the rule among the kernel-labeled rules that is associated with the inaccurate classification, and in step S163, retraining the CNN with further training images that include features corresponding to the kernel labels of the rules associated with the inaccurate classification.

[0054] The embodiments may be applied to any scenario in which classification is performed using video data. One example is CCTV security cameras for detecting security risks. These may need to be debugged if they produce false positives that lead to innocent parties being wrongly charged. Another application may be automated video tagging, i.e., to explain and diagnose inaccurate tags.

[0055] A further application lies in Advanced Driver-Assistance Systems (ADAS). ADAS can be trained to recognize road scenes, thereby inferring local driving restrictions when traffic signs are not visible and GPS connectivity is lost (which is typically used to obtain local information). For example, in the UK, if ADAS recognizes the local scene as a residential area within the UK, the safest assumption would be a 30 mph speed limit. If it sees a motorway (main road), a 70 mph limit might be assumed. Near a school, perhaps a 20 mph limit.

[0056] If a scene is misclassified, there is a risk that the vehicle may be driven at an unsafe speed. For example, 70 mph in a residential area or 30 mph on a highway are both dangerous. Therefore, whether such instances are observed during development and testing or by users using the deployed system, these errors must be understood and corrected by the manufacturer.

[0057] Next, the application of the method according to one embodiment to ADAS will be described.

[0058] [Examples] CNN training and knowledge extraction The CNN is trained on a scene classification dataset (problem dataset), and the rules that explain the CNN's classification are extracted using the conventional or similar methods described with reference to Figure 2. However, the extracted rules do not have labels assigned to the kernels. By default, the kernels are labeled with alphabetical characters, as shown in Figure 2. For example, the rule for identifying a street appears as G∧E→street. The example presented in Figure 2 is used for the remainder of this scenario.

[0059] Kernel labeling A dataset of video from a fixed CCTV camera is selected as the support dataset. The system iterates through each video using the difference between frames to support the kernel labeling described in the example below.

[0060] In this example, note that since the extracted logical program M' does not contain literals D, H, I, or L, the corresponding kernels are excluded from the process and therefore do not need to be labeled.

[0061] Figure 8 shows eight frames from a video captured by a CCTV camera fixed to the side of a building beside a road, and the corresponding changes in kernel activation. Table 1 (Figure 9) shows the existence matrix generated based on raw annotations when objects enter and leave the scene. The following explanation describes annotations and kernel activation in parallel, but it is assumed that the annotation matrix (Table 1, left) would have been completed before the extraction process was performed.

[0062] • t=0: At the beginning of the video, the door, tree, and several windows are already in view, and therefore the annotator will mark "1" under these headings in the "raw annotation" matrix. The three kernels "C", "E", and "J" are already active according to their magnitude relative to the global threshold, which means they are related to visible objects.

[0063] • t=1: A person emerges from the door, and therefore the annotator marks the "person" with a "1". The door, tree, and window are still in the view, but there was no need to mark them with a "1" again, because the system assumes they still exist unless otherwise notified. This is reflected in the existence matrix (Table 1, right) which is automatically generated from the annotation matrix. One further kernel "G" is activated, which suggests a relationship to the person who entered the scene.

[0064] • t=2: The person has moved further to the right of the camera view. Meanwhile, no further objects have entered or left the view, so the change in kernel activation is negligible.

[0065] • t=3: The person is still moving further towards the edge but remains within the view. A vehicle enters the left side of the screen, and the annotator marks it with a "1" to indicate that this is a van entering the scene. Kernel "A" has been activated, but only by a slight difference.

[0066] • t=4: People begin to disappear, and the activation of kernel "G" weakens (though it is still active), which further supports the evidence that this kernel corresponds to people. As the activation of kernel A becomes stronger, more of the vans appear, which again suggests a correlation. Furthermore, the vans block the tree, and kernel D becomes inactive, which suggests that E responds to the tree. The annotator marks the "tree" label with "-1" to indicate that it has disappeared, and the "tree" column in the existence matrix from t=0 to t=3 is automatically populated with 1.

[0067] • t=5: The people have now left the scene, and kernel G is inactive again. The annotator marks "-1" under "people" to indicate they have disappeared, and the "people" column in the existence matrix is ​​filled with 1s from t=1 (when the people appeared) to t=4.

[0068] • t=6: The van is still in view, but has passed the tree, which is no longer blocking it. The annotator marks a "1" under the "tree" to indicate that it reappears in this frame, and kernel E is active again. Kernel A's activation weakens as the van begins to move out of the scene.

[0069] • t=7: The van is now out of the scene, and therefore the annotator marks it as such by marking "-1" under "van". Kernel E is inactive again. The "van" column in the existence matrix for t=3 to t=6 (when the van appeared and when it disappeared, respectively) is set to 1. Furthermore, since this is the end of the video, 1 is also populated in the existence matrix columns for all entities that are still visible: namely, "door" and "window" from t=1 to t=7 (because they did not disappear), and "tree" from t=6 to t=7 (starting from when the tree reappeared). Finally, note that the scene and kernel activations have all returned to their original states as seen at t=0.

[0070] This process is repeated for further training videos, with the annotator only needing to mark when entities appear or disappear within the camera view. The presence or absence of these entities in all other frames was automatically populated into the presence matrix by the system.

[0071] Kernel labeling (segmentation-based alternative) Alternatively, if segmentation annotations are available, the existence matrix can be filled with the total area (in pixels) occupied by the segments of each class, and the kernel activation matrix can be filled with the activation values ​​before quantization. The values ​​for the current example are shown in Table 3 (Figure 12), based on the segments shown in Figure 11. In Figure 11, each number within the segmented region corresponds to a different color, and therefore to a label. In Table 3, the existence matrix and activation matrix are completed according to the frame segmentation shown in Figure 11. That is, each value in the existence matrix corresponds to the area occupied by the corresponding feature in the corresponding frame. For example, at t=4, van (7) occupies 40 pixels.

[0072] Correlation matrix After all training videos have been observed, the Pearson correlation between processed annotations and kernel activations is calculated, as shown in Table 4 (Figure 13). The maximum absolute value for each kernel is shown in bold because each kernel is assigned the label with the strongest absolute feature correlation.

[0073] Each kernel is then assigned the label with which it has the strongest correlation, according to this matrix. The rule of M' can now be interpreted as shown in Table 5 (Figure 14).

[0074] inference Later, after the trained and labeled ADAS software was deployed to the vehicle, the user noticed that while driving along a highway (main road), the car simply advised him to slow down because it thought he was in a residential area (see Figure 15). In the camera's view, someone was stuck next to a tree, waiting outside the car for roadside assistance. The driver then realized that the car had classified the scene as a "street," and the explanation associated with this classification was that it stemmed from the presence of the person and the tree: person ∧ tree → street.

[0075] The user reported this error to the manufacturer, who, upon investigating the explanation, acknowledged that the rule was based on unreasonable assumptions because it is highly likely that trees will be found near the roadside of a highway, and, less likely, that a person will be found standing on the highway in such a scenario.

[0076] This explanation reveals that the developers will need to retrain their models with more examples of highways where people might be waiting beside their cars and / or trees might be found along the roadside.

[0077] Figure 17 is a block diagram of a computing device, such as a data storage server, which may be used to embody the present invention, to carry out some or all of the operations of a method for embodying the present invention, and to perform some or all of the tasks of the apparatus of the embodiment. For example, the computing device of Figure 17 may be used to implement some or all of the processes described with reference to Figures 4, 5, 7, and / or 16.

[0078] The computing device includes a processor 993 and memory 994. Optionally, the computing device further includes a network interface 997 for communication with other such computing devices, for example, with other computing devices in embodiments of the present invention.

[0079] For example, one embodiment may consist of a network of such computing devices. Optionally, the computing devices further include one or more input mechanisms, such as a keyboard and a mouse 996, and one or more display units, such as a monitor 995. The components are connectable to each other via a bus 992.

[0080] Memory 994 may include computer-readable media, and the term may refer to a single or multiple media (e.g., a centralized or distributed database and / or associated caches and servers) configured to store information such as problem datasets, support image datasets, kernel-labeled rules, misclassified images, and / or images used for retraining, and / or to carry computer-executable instructions. Computer-executable instructions may be accessible by, for example, a general-purpose computer, a dedicated computer, or a dedicated processing device (e.g., one or more processors), and may include instructions and data that cause them to perform one or more functions or operations. For example, computer-executable instructions may include instructions for performing some or all of the steps shown in Figure 4, Figure 5, Figure 7, or Figure 16, or instructions for performing one or more processes described with reference to Figure 4, or Figure 5, or Figure 6, or Figure 7, or Figure 16. Thus, the term “computer-readable storage medium” may include any medium that can store, encode, or carry a set of instructions for machine execution and cause a machine to perform one or more of the methods of the present disclosure. Therefore, the term “computer-readable storage medium” may be considered to include, but is not limited to, solid-state memory, optical media, and magnetic media. For example, such computer-readable storage medium may include non-temporary computer-readable storage mediums, including random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), compact disk read-only memory (CD-ROM) or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, and flash memory devices (e.g., solid-state memory devices).

[0081] The processor 993 is configured to control the computing device and perform processing operations, as described, for example, with reference to Figures 4, 7, and / or 16, and executes computer program code stored in memory 994 to carry out the methods defined in the claims. Memory 994 stores data to be read from and written by the processor 993. When referred to herein, the processor may include one or more general-purpose processing devices, such as a microprocessor or a central processing unit. The processor may include a composite instruction set compute (CISC) microprocessor, a reduced instruction set compute (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or a processor that implements other instruction sets or combinations of instruction sets. Furthermore, the processor may include one or more dedicated processing devices, such as an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), or a network processor. In one or more embodiments, the processor is configured to execute instructions and operations for performing the operations discussed herein.

[0082] The display unit 995 can display a representation of data stored by the computing device, such as problem datasets, support image datasets, misclassified images, and / or images used for retraining. Furthermore, it can display a cursor, dialog boxes, and screens that enable interaction between the user and programs and data stored on the computing device. The input mechanism 996 may enable the user to input data and instructions into the computing device.

[0083] The network interface (network I / F) 997 can connect to a network such as the Internet and can connect to other such computing devices via the network. The network I / F 997 can control the data that is input and output to and from other devices via the network.

[0084] Other peripherals such as microphones, speakers, printers, power supply units, fans, cases, scanners, and tracker balls may be included in the computing device.

[0085] A method for embodying the present invention can be performed using a computing device as shown in Figure 17. Such a computing device does not need to have all the components shown in Figure 17, but may consist of a subset of these components. A method for embodying the present invention can be performed using a single computing device that communicates with one or more data storage servers over a network. The computing device may be a data storage device itself that stores at least a portion of the data.

[0086] The method for realizing the present invention may be carried out by a plurality of computing devices working together. One or more of the plurality of computing devices may be data storage servers that store at least a portion of the data.

[0087] The present invention can be implemented in digital electronic circuits, or in computer hardware, firmware, software, or a combination thereof. The present invention can be implemented as a computer program or computer program product, i.e., a computer program tangibly embodied in a non-temporary information carrier, for example in a machine-readable storage device, or in a propagating signal, for execution by one or more hardware modules, or for controlling the operation of one or more hardware modules.

[0088] Computer programs can take the form of standalone programs, computer program components, or multiple computer programs; they can be written in any form of programming language, including compiled or interpreted languages; and they can be deployed in any form, including as standalone programs or as modules, components, subroutines, or other units suitable for use in a data processing environment. Computer programs can be deployed to run on a single module, on multiple modules at a single site, or distributed across multiple sites and interconnected by a communication network.

[0089] The method steps of the present invention can be carried out by one or more programmable processors that execute a computer program to perform the functions of the present invention by operating on input data and generating an output. The apparatus of the present invention can be implemented as programmed hardware or as a dedicated logic circuit including, for example, an FPGA (Field Programmable Gate Array) or an ASIC (Application-Specific Integrated Circuit).

[0090] Processors suitable for executing computer programs include, for example, both general-purpose and specialized microprocessors, as well as any one or more processors in any type of digital computer. Generally, a processor receives instructions and data from read-only memory, random-access memory, or both. An essential element of a computer is a processor that executes instructions coupled to one or more memory devices that store instructions and data.

[0091] The above-described embodiments of the present invention may be used, advantageously, independently of any other embodiment, or in any feasible combination with one or more other embodiments.

[0092] [Glossary of terms used in this specification] ADAS - Advanced Driver Assistance Systems: A combination of software and hardware included in a vehicle that assists the driver without requiring the driver to have complete control of the vehicle. Feature attribute map (or "feature importance map"): A heatmap of an image classified by a CNN (or other method), showing the importance of each pixel in the image in relation to the output classification or the output activation of any other component of the classifier. Inpainting: A category of image processing methods that automatically fill in missing image data by estimating the lost information, or replace entities in an image with an estimated background area that they obscure. Perturbation-based feature attribute: A method for generating feature attribute maps by adding perturbations to an input image and observing changes in the output classification or activation of components of interest. Existence matrix: A term used to describe a table that shows which features or objects (columns) exist in which image / video frames (rows). Problem dataset: The dataset on which the CNN to be described was originally trained. In other words, a dataset representing the problem domain to which the CNN should be applied. Support datasets: Datasets used for kernel labeling purposes.

[0093] The following note is left regarding the above embodiment. (Note 1) A computer-based image classification method, The steps include obtaining a convolutional neural network (CNN) trained to classify features in images using a training image dataset, and The steps include: extracting a logic program from the CNN, wherein the logic program is a symbolic approximation of the output of the kernel in the extraction layer of the CNN; and deriving a rule from the logic program that uses the kernel to explain the image classification by the CNN. Steps include obtaining a feature-labeled image dataset and a record of each feature associated with each feature-labeled image in the dataset, wherein the images in the dataset include pairs of feature-labeled images, one of which is a scene containing the feature, and the other of which is a scene without the feature. The steps include forward propagation of the pair of feature-labeled images via the logic program to obtain kernel activations in the extraction layer for features within the images, A step of calculating the correlation between each kernel in the logic program and each feature in the feature-labeled image using the acquired kernel activation and the features associated with the feature-labeled image, The steps include assigning a label for the feature that has the highest correlation to each kernel in the logic program, The steps include applying the assigned kernel label to the kernel in the derived rule to obtain a kernel-labeled rule, A method that includes this. (Note 2) The method according to Appendix 1, wherein the images in the feature-labeled image dataset include still frames from at least one video recording. (Note 3) The method described in Appendix 2, wherein at least one of the video recordings is captured by a closed-circuit television (CCTV) camera. (Note 4) The method according to any one of the appendices 1 to 3, wherein the feature-labeled image dataset includes images annotated for semantic segmentation, and the record of each feature associated with each feature-labeled image in the dataset includes a value corresponding to the total area occupied by the feature in the image. (Note 5) A computer-based method for improving the training of an image classifier, For a convolutional neural network (CNN) trained to classify features in an image, the steps include obtaining kernel-labeled rules derived from the CNN using the method described in any one of the appendices 1 to 4, A step of obtaining a classification of at least one image determined by the CNN for at least one image that does not form part of the training image dataset used to train the CNN or the feature-labeled image dataset used to derive the kernel-labeled rules, wherein the classification is evaluated as inaccurate; and a step of identifying the kernel-labeled rules associated with the inaccurate classification. The steps include: retraining the CNN with further training images that include features corresponding to the kernel labels of the rules associated with the aforementioned inaccurate classifications; A method that includes this. (Note 6) A computer program that, when executed on a computer, causes the computer to perform the method described in any one of the appendices 1 to 5. (Note 7) An image classification device, At least one memory, (a) A convolutional neural network (CNN) trained to classify features in images using a training image dataset, and (b) A feature-labeled image dataset, and a record of each feature associated with each feature-labeled image in the dataset, wherein the images in the dataset include pairs of feature-labeled images, one of which is a scene containing the feature, and the other of which is a scene without the feature; a feature-labeled image dataset, and a record of each feature associated with each feature-labeled image in the dataset. At least one memory to store, At least one processor, connected to the memory, A logic program is extracted from the CNN stored in the memory, the logic program is a symbolic approximation of the output of the kernel in the extraction layer of the CNN, and a rule is derived from the logic program that uses the kernel to describe the image classification by the CNN. Through the logic program, pairs of feature-labeled images from the feature-labeled dataset stored in the memory are propagated forward to obtain kernel activations in the extraction layer for features in the images. Using the acquired kernel activations and the features associated with the feature-labeled image, the correlation between each kernel in the logic program and each feature in the feature-labeled image is calculated. Each kernel in the logic program is assigned the label of the feature that has the highest correlation with that kernel. Apply the assigned kernel label to the kernel in the derived rule to obtain a kernel-labeled rule. At least one processor, A device that includes this. (Note 8) The apparatus according to Appendix 7, wherein the images in the aforementioned feature-labeled image dataset include still frames from at least one video recording. (Note 9) The apparatus as described in Appendix 8, wherein at least one of the video recordings is captured by a closed-circuit television (CCTV) camera. (Note 10) The apparatus according to any one of appendices 7 to 9, wherein the feature-labeled image dataset includes images annotated for semantic segmentation, and the record of each feature associated with each feature-labeled image in the dataset includes a value corresponding to the total area occupied by the feature in the image. (Note 11) A device for improving the training of an image classifier, At least one memory, For a convolutional neural network (CNN) trained to classify features in an image, kernel-labeled rules derived from the CNN using the apparatus described in any one of the appendices 7 to 10, and A classification of at least one image determined by the CNN, where the classification is evaluated as inaccurate, for at least one image that does not form part of the training image dataset used to train the CNN or the feature-labeled image dataset used to derive the kernel-labeled rules. At least one memory to store, At least one processor, connected to the memory, Identify the rules among the kernel-labeled rules that are associated with the inaccurate classification. The CNN is retrained using further training images that include features corresponding to the kernel labels of the rules associated with the aforementioned inaccurate classifications. At least one processor, A device that includes this.

Claims

1. A computer-based image classification method, The steps include obtaining a convolutional neural network (CNN) trained to classify features in images using a training image dataset, and The steps include: extracting a logic program from the CNN, wherein the logic program is a symbolic approximation of the output of the kernel in the extraction layer of the CNN; and deriving rules from the logic program that use the kernel to describe the image classification performed by the CNN. Steps include obtaining a feature-labeled image dataset and a record of each feature associated with each feature-labeled image in the feature-labeled image dataset, wherein the images in the feature-labeled image dataset include pairs of feature-labeled images, the first feature-labeled image of the pair is of a scene containing the feature, and the second feature-labeled image of the pair is of the same scene but without the feature, and is the same image as the first feature-labeled image except for the portion where the feature was present. The steps include forward propagation of the pair of feature-labeled images via the logic program to obtain kernel activations in the extraction layer for features within the images, A step of calculating the correlation between each kernel in the logic program and each feature in the feature-labeled image using the acquired kernel activation and the features associated with the feature-labeled image, The steps include assigning a label for the feature that has the highest correlation to each kernel in the logic program, The steps include applying the assigned kernel label to the kernel in the derived rule to obtain a kernel-labeled rule, A method that includes this.

2. The method according to claim 1, wherein the images in the feature-labeled image dataset include still frames from at least one video recording.

3. The method according to claim 2, wherein the at least one video recording is captured by a closed-circuit television (CCTV) camera.

4. The method according to any one of claims 1 to 3, wherein the feature-labeled image dataset includes images annotated for semantic segmentation, and the record of each feature associated with each feature-labeled image in the feature-labeled image dataset includes a value corresponding to the total area occupied by the feature in the image.

5. A computer-based method for improving the training of an image classifier, A convolutional neural network (CNN) trained to classify features in an image, comprising the steps of obtaining kernel-labeled rules derived from the CNN using the method described in claim 1, A step of obtaining a classification of at least one image determined by the CNN for at least one image that does not form part of the training image dataset used to train the CNN or the feature-labeled image dataset used to derive the kernel-labeled rules, wherein the classification is evaluated as inaccurate; and a step of identifying the kernel-labeled rules associated with the inaccurate classification. The steps include: retraining the CNN with further training images that include features corresponding to the kernel labels of the rules associated with the aforementioned inaccurate classifications; A method that includes this.

6. A computer program that, when executed on a computer, causes the computer to perform the method according to claim 1 or 5.

7. An image classification device, At least one memory, (a) A convolutional neural network (CNN) trained to classify features in images using a training image dataset, and (b) A feature-labeled image dataset, and a record of each feature associated with each feature-labeled image in the feature-labeled image dataset, wherein the images in the feature-labeled image dataset include pairs of feature-labeled images, the first feature-labeled image of the pair being of a scene containing the feature, and the second feature-labeled image of the pair being of the same scene but without the feature, and being the same image as the first feature-labeled image except for the portion where the feature was present; a feature-labeled image dataset, and a record of each feature associated with each feature-labeled image in the feature-labeled image dataset. At least one memory to store, At least one processor, connected to the memory, A logic program is extracted from the CNN stored in the memory, the logic program is a symbolic approximation of the kernel output in the extraction layer of the CNN, and rules are derived from the logic program that use the kernel to explain the image classification by the CNN. Through the logic program, pairs of feature-labeled images from the feature-labeled dataset stored in the memory are propagated forward to obtain kernel activations in the extraction layer for features in the images. Using the acquired kernel activations and the features associated with the feature-labeled image, the correlation between each kernel in the logic program and each feature in the feature-labeled image is calculated. Each kernel in the logic program is assigned the label of the feature that has the highest correlation with that kernel. Apply the assigned kernel label to the kernel in the derived rule to obtain a kernel-labeled rule. At least one processor, A device that includes this.

8. The apparatus according to claim 7, wherein the images in the feature-labeled image dataset include still frames from at least one video recording.

9. The apparatus according to claim 8, wherein the at least one video recording is captured by a closed-circuit television (CCTV) camera.

10. The apparatus according to any one of claims 7 to 9, wherein the feature-labeled image dataset includes images annotated for semantic segmentation, and the record of each feature associated with each feature-labeled image in the feature-labeled image dataset includes a value corresponding to the total area occupied by the feature in the image.

11. A device for improving the training of an image classifier, At least one memory, A convolutional neural network (CNN) trained to classify features in an image, wherein a kernel-labeled rule is derived from the CNN using the apparatus described in any one of claims 7 to 9, and A classification of at least one image determined by the CNN, which does not form part of the training image dataset used to train the CNN or the feature-labeled image dataset used to derive the kernel-labeled rules, and which is evaluated as inaccurate. At least one memory to store, At least one processor, connected to the memory, Identify the rules among the kernel-labeled rules that are associated with the inaccurate classification. The CNN is retrained using further training images that include features corresponding to the kernel labels of the rules associated with the aforementioned inaccurate classifications. At least one processor, A device that includes this.

Images