Layered image decomposition for defect detection

Abstract

Aspects of the present invention disclose a method, computer program product, and system for improving object detection in high resolution images at inference time. The method includes one or more processors receiving a high resolution image. The method also includes the one or more processors decomposing the received image into hierarchically organized image layers. Each layer includes at least one image patch of the received image. Each image patch has a corresponding resolution suitable for a baseline image recognition algorithm. The method also includes the one or more processors applying the baseline algorithm to each image patch of each layer. The method also includes the one or more processors performing a result aggregation of the baseline algorithm application results to the image patches of the layer.

Description

BACKGROUND

[0001] The present invention relates generally to the field of object detection, and more specifically to object detection in high resolution images.

[0002] Research organizations and enterprises have made tremendous efforts in implementing Al (Artificial Intelligence) driven applications in order to automate processes, implement more humanized user interfaces or help analyze large amounts of data. Deep learning methods have proven significant success and have outperformed classical machine learning solutions. Two main factors, mainly the availability of performance computing infrastructure and the availability of large labeled datasets, have driven the success of deep learning techniques. Deep learning methods are often used for image classification, object detection, video analysis, text translation and audio type classification, to name a few. In particular, recent and more advanced models that operate with pixel data can leverage convolutional neural networks. As such, deep learning methods that process image data can be divided into three main tasks: a) classification, b) detection, and c) segmentation. All three tasks share a single input, but they define what the method has to produce. For classification, a single class label is predicted (e.g., an image showing a guard dog), for detection, a bounding box is produced (e.g., the guard dog is located in the rectangle [X, Y, dX, dY]); for segmentation, pixels belonging to the intended object are predicted (e.g., pixels pi, p2, p3,..., pN represent the guard dog).

[0003] Automated defect detection defines a subset of the general task of object detection, where the goal is to identify (detect and / or segment) defects on industrial images. Applications can include use cases from various domains, including the medical domain (e.g., identifying anatomical structures of a human from an X-ray scan), the material manufacturing industry (e.g., identifying defects on produced steel or other products) or defect detection on civil infrastructure (e.g., bridges or high-rise buildings). SUMMARY

[0004] Aspects of the present invention disclose a method, computer program product, and system for improving object detection in high resolution images with inference time. The method includes receiving a high resolution image by one or more processors. The method also includes decomposing, by the one or more processors, the received image into hierarchically organized layers of the image. Each layer includes at least one image patch of the received image. Each image patch has a corresponding resolution suitable for a baseline image recognition algorithm. The method also includes applying, by the one or more processors, the baseline algorithm to each image patch of each layer. The method also includes performing, by the one or more processors, a result aggregation on results of applying the baseline algorithm to the image patches of the layers.

[0005] In another embodiment, the result aggregation of the results of the baseline algorithm application to the image tiles of the layers further comprises the one or more processors aggregating the results of the baseline algorithm for each layer, the one or more processors comparing the results of the baseline algorithm for adjacent pairs of layers, and the one or more processors performing a hierarchical aggregation of the results of the baseline algorithm according to the pairs of layer comparisons. BRIEF DESCRIPTION OF DRAWINGS

[0006] It should be noted that embodiments of the application are described with reference to different subject matters. In particular, some embodiments are described with reference to method type claims whereas other embodiments are described with reference to device type claims. However, a person skilled in the art will gather from the above and the following description that, unless otherwise indicated, in addition to any combination of features belonging to one type of subject matter, any combination of features also is considered to be a combination of the features of the different types of subject matter, e.g. the combination of features of the method type claims and features of the device type claims are also considered to be disclosed herein.

[0007] The above and other aspects of the application will be apparent from the examples of embodiments to be described hereinafter and will be explained with reference to examples of embodiments. The application is not limited to the examples of embodiments described but determined by the claims appended hereto.

[0008] Figure 1 A block diagram depicting an embodiment of a method for improving object detection in high resolution images with inference time according to embodiments of the application is depicted.

[0009] Figure 2 A block diagram depicting an embodiment of a method for improving object detection in high resolution images with inference time according to embodiments of the application is depicted.

[0010] Figure 3 A block diagram depicting an embodiment of a method for improving object detection in high resolution images with inference time according to embodiments of the application is depicted.

[0011] Figure 4 A block diagram depicting an embodiment of a method for improving object detection in high resolution images with inference time according to embodiments of the application is depicted.

[0012] Figure 5 A block diagram depicting an embodiment of a method for improving object detection in high resolution images with inference time according to embodiments of the application is depicted.

[0013] Figure 6 A block diagram depicting an embodiment of a method for improving object detection in high resolution images with inference time according to embodiments of the application is depicted.

[0014] Figure 7A block diagram depicting an embodiment of a computing system including an object recognition system of the present invention, according to an embodiment of the present invention. DETAILED DESCRIPTION

[0015] Embodiments of the present invention recognize that even when using state-of-the-art deep learning methods, many use cases of object detection define very challenging instances of the problem. The reasons are manifold and can be explained by variations in light, including but not limited to: different light conditions, different image resolutions, different cameras used to capture the images (e.g., different lens distortions, sensitivity of the camera (ISO), etc.), different viewpoints, different levels of zoom, occlusions from obstacles (e.g., trees in front of the pillars of a bridge), and different backgrounds (e.g., different appearances from two bridges), unexpected obstacle objects in the background (e.g., people, cars, and boats near a bridge, where the bridge and its defects are the main subject). In addition, many defects do not have clear boundaries, which makes the defects difficult to be detected as objects.

[0016] Further embodiments of the present invention recognize that traditional deep learning methods typically operate on relatively small image sizes. For example, the state-of-the-art image classification algorithms evaluated on the CIFAR-10 dataset use inputs of shape 32 by 32 pixels. In addition, widespread image networks provide different image sizes, and most algorithms follow uniform training and evaluation settings, where images are resized to a fixed size of 224 by 224 pixels. Object detection algorithms, such as Mask R-CNN (Region-based Convolutional Neural Network), operate on a fixed scale of 1024 pixels. In contrast to this, embodiments of the present invention recognize that high-resolution images are nowadays almost freely available; many cameras support 2K, 4K, and 8K modes, and there are high-end cameras that support 16K to 64K. Capturing images under this setting results in processing images that are 2 to 64 times wider in pixels than the expected size of the original formula of the detector.

[0017] Embodiments of the invention recognize that the defects are primarily small features that are located at only a few sparse locations in the high resolution image. Thereafter, simply resizing the image to a smaller resolution is problematic as that would lose a substantial factor of the resolution. Additionally, tiling a high resolution image into separately processed smaller images does help to preserve the high resolution. However, tiling comes with the additional overhead necessary to handle overlapping regions (e.g., a partially visible defect on one tile) and it extends the workload of one image to the number of tiles extracted from the image, resulting in a larger workload. Embodiments of the invention recognize that, as required in deep learning methods, to get good generalization behavior of the deep learning method, it is important that the training and test images follow the same statistics. Experiments were conducted with different tiling settings and ensured that the statistics of one image tile in the training match the statistics of the image tile in the test scene are becoming an important task. Furthermore, embodiments of the invention recognize that the Modifi cation Cations applied in the training domain that trigger retraining of the model are computationally very intensive and thus inefficient and expensive.

[0018] Additional embodiments of the invention recognize that, to implement the first development cycle, commonly known object identification algorithms are repeatedly reused. The algorithms typically rely on a fixed image resolution. However, camera resolutions are rapidly increasing, making the assumed image resolution of the known object identification algorithms potentially lagging behind their development. Additionally, even if the object identification algorithms were to keep up with the availability of ever higher resolution cameras, embodiments of the invention recognize that the computational effort for retraining existing neural networks and their hyperparameters for reconfiguration would be enormous, which is considered a key drawback of the traditional approach. To overcome this deadlock situation, embodiments of the invention recognize the need to provide stereoscopic object identification capabilities without the need to retrain existing image recognition algorithms.

[0019] In the context of the present specification, the following conventions, terms and / or expressions can be used.

[0020] The term“object detection” can denote an activity of a system supported by a method for identifying one or more predefined items, samples or patterns in a given digital image.

[0021] The term“high resolution image” can denote an image having a higher resolution than the resolution of the input to a given algorithm used as a typical image for object detection processing. Thus, the resolution of the input required by the baseline algorithm and the resolution of the high resolution image do not match. Therefore, measures can be required to use the baseline algorithm that has been pre-trained to process the high resolution image or at least parts thereof (e.g., image tiles) without the need to retrain or reconfigure or redesign the pre-trained baseline algorithm.

[0022] The term“inference time” can denote the time in which a trained machine learning system (e.g. a convolutional neural network) can predict a class or a segmentation of an input image. In contrast to inference time is the training time. The pre-training of a machine learning system can require a lot of computational power and time, whereas the inference activities of a trained machine learning system can be optimized to work with only little computational resources.

[0023] The term“decomposed image” can denote the process of cutting a given digital image into parts (e.g. rectangular segments), which can also be denoted as image tiles.

[0024] The term“layer of a hierarchical organization” can denote a plurality of layers comprising a predetermined number of sub-images of a given original (i.e. received) digital image in a given layer. Thus, the layers can be distinguished by different resolutions. The lowest layer can be the layer with the highest resolution, i.e. the largest number of pixels available for the given digital image providing the basis for global coordinates. The highest layer can be defined as the layer with the lowest number of pixels for the given digital image, i.e. the layer with the lowest resolution.

[0025] The term“suitable resolution” (e.g. in particular suitable resolution for a baseline algorithm) can denote a resolution of a digital image optimized for an object recognition algorithm like Mask R-CNN or Fast R-CNN. For example, the algorithm works with a resolution of 224x224 pixels. Thus, a digital image with 1000x1000 pixels is not suitable for a given baseline algorithm.

[0026] The term“trained baseline image recognition algorithm” can denote an image detection and / or identification algorithm which can also be fully implemented in hardware (e.g. using crossbars of memristive devices) which has undergone a training such that hyperparameters and weighting factors (of e.g. a convolutional neural network) can be defined and a neural network model can be developed accordingly. The trained baseline algorithm can then be used for object identification tasks at inference time.

[0027] The term“intelligent result aggregation” can denote a multi-step process as a tool of the object recognition process proposed here. The intelligent result aggregation can comprise at least the following steps: (i) aggregating the results of the baseline algorithms per layer, (ii) performing a pairwise layer comparison, and (iii) performing a hierarchical aggregation of the comparison results. The details of the multi-step process are defined by the dependent claims and described in more detail in the context of the attached figures.

[0028] The term“overlapping region” can denote a part of an image tile of an image which can be a part of two adjacent image tiles of the same digital image. The image part can be a part of a left image tile and a right image tile placed side by side.

[0029] The term“intermediate image layer” can denote a layer built by layer-wise comparison of intermediate results as part of the intelligent result aggregation process. Thus, a given number of M layers (e.g. 4 layers) yields N intermediate image layers, with N = M - 1. Reference is made to Figure 4 A practical example is described in more detail.

[0030] The term“pixel-wise union” can denote a process of combining two shapes with a logical“or” function. The“or” function can be applied to binary masks encoding the shapes. Similarly, the same term can denote a process of taking the“union” of two shapes which can be encoded as polygons.

[0031] The term“pixel-wise intersection” can denote a process of combining two shapes with a logical“and” function. The“and” function can be applied to binary masks encoding the shapes. Similarly, the same term can denote a process of taking the“intersection” of two shapes which can be encoded as polygons.

[0032] The term“identified item” can denote an object in a digital image having a predefined shape (i.e. shape or other characteristic feature) to the recognition or identification of which the machine learning system used has been trained. In this sense, the identified item can be identical to the recognized object.

[0033] The term“mask R-CNN algorithm” can denote a known convolutional neural network algorithm used for segmentation, e.g. based on a known processing architecture for object detection. Thereby, an image input can be presented to a neural network; a selected search process can be run on the received digital image, and then, output regions from the selected search process can be used for future extraction and classification using a pre-trained convolutional neural network.

[0034] The term“fast R-CNN algorithm” can denote an enhanced version of the mask R-CNN algorithm. The fast R-CNN algorithm can still use a selected search algorithm to obtain region proposals, but can add a region of interest (ROI) pooling module. The fast R-CNN algorithm can extract fixed-size windows from feature maps in order to obtain final class labels and bounding boxes for a given object within the received digital image. A benefit of this approach can be that the convolutional neural network is now end-to-end trainable.

[0035] The term“pre-trained” can mean that the image or object recognition system has been trained prior to use. In particular, a pre-trained object recognition system or method can be used as a tool for processing digital images to be classified that are not directly suitable for the previously trained baseline algorithm used (e.g., due to resolution mismatch). In contrast (e.g., on traditional systems), the concept presented here uses decomposition of a given (i.e., received) digital image and intelligent result aggregation in order to overcome a mismatch in resolution of the received digital image and the resolution required by the baseline object identification algorithm.

[0036] The term“neural network model” can mean the sum of all weights of a given neural network together with the logical organization of the neural network used (i.e., the hyperparameters of the underlying machine learning system (here, a convolutional neural network)).

[0037] Below, a detailed description of the drawings will be given. All illustrations in the figures are schematic. First, a block diagram of an embodiment of the inventive method for improving object detection in high resolution images at inference time will be given. Then, further embodiments of the object recognition system for improving object detection in high resolution images at inference time as well as embodiments of the method will be described.

[0038] Figure 1 A block diagram of an embodiment of a method 100 for improving object detection (e.g., defect detection) in high resolution images at inference time according to embodiments of the present invention is shown. In an example embodiment, an object recognition system 600 (depicted in Figure 6 ) can perform the processing steps of the method 100 (i.e., perform the operations Figure 1 ) according to embodiments of the present invention. In additional example aspects, the object recognition system 600 (in conjunction with the method 100) can perform operations described in more detail and described with respect to Figures 2 to 5 below according to embodiments of the present invention.

[0039] In step 102, the method 100 receives a high resolution image. In an example embodiment, the method 100 receives a digital image having a resolution that is larger than the image resolution used by the underlying baseline image recognition algorithm (e.g., Mask R-CNN).

[0040] In step 104, the method decomposes the received image into layers of a hierarchical organization of the image. In an example embodiment, each layer comprises at least one image tile of the received image (only the image with F = max has only one image tile, all other layers have more image tiles). In additional embodiments, each image tile has a resolution that is suitable (e.g., required or recommended) for the pre-trained baseline image recognition algorithm.

[0041] In step 106, the method 100 applies the baseline algorithm to each of the image tiles of each layer. In an example embodiment, the method 100 can operate to identify regions of interest, bounding boxes of objects (i.e., rectangles that surround defects and classifications), and / or alternatively additionally mask polygons, shapes based on pre-training.

[0042] Additionally, in process 108, the method 100 performs intelligent result aggregation. For example, the method 100 intelligently aggregates the results of applying the baseline algorithm to the image tiles of the layers using a three-step approach, as described in steps 110 and 114.

[0043] In step 110, the method 100 aggregates the results of the baseline algorithm by layer. In step 112, the method 100 performs a pair-wise layer comparison of the results of the baseline algorithm on adjacent pairs of layers. In step 114, the method 100 performs a hierarchical aggregation of the results of the baseline algorithm from the pair-wise layer comparison. Thus, the method 100 can utilize a consistent scaling factor, which means that, depending on the resolution, one pixel in one resolution can be compared to four pixels and / or 16 pixels and even higher resolutions in another resolution. If the number of white or black pixels is equally distributed (i.e., 50 / 50) in a resolution higher than the lowest resolution (e.g., in a black and white image or in one or more color channels), a random decision is made for one of the two color options.

[0044] Figure 2 A block diagram 200 of one embodiment is described, in which the original, received high resolution image 202 is used as the basis for the operation of the embodiments of the present invention (e.g., the processes of the method 100). The rectangle 204 represents the fixed working size of the working resolution used and / or required by the baseline image recognition algorithm. Thus, the images corresponding to the different layers f = 1.0, f = 2.0, f = 6.0, in particular layer 1, layer 2, layer 3, must be cut into image tiles. The first layer image 206 is cut into 24 image tiles, so that each image tile has a number of pixels equal to the working size of the image of the baseline algorithm.

[0045] Correspondingly, the image 208 of layer 2, which has a lower resolution than the image of layer 1, only needs 6 image tiles, while the image 210 of layer 3, which has the lowest resolution, only needs a single image tile, since the corresponding resolution matches the working size of the baseline algorithm. The number of layers is configurable and can depend on the image resolution of the received digital image.

[0046] As a result of the chunking step, embodiments of the present application generate a set of image chunks 212, 214, and 216, where the higher the resolution of the images 206, 208, and 210, the more image chunks per set. The sets of image chunks are then used as input to a baseline algorithm. As an example, image chunk 216 is the lower right corner of image 206, and image chunk 214 is the upper middle of layer 2 image 208. The overall process will be referred to as Figure 3 and described in further detail.

[0047] Figure 3 A block diagram 300 is depicted that details the feeding of the sets of input image chunks 212, 214, and 216 (see Figure 2 ) to an object recognition baseline algorithm 302, in accordance with embodiments of the present application. In various embodiments, the object recognition baseline algorithm 302 can be used in a pre-trained form, without any additional training. For example, the object recognition baseline algorithm 302 can be used as available. The sets of outputs 304, 306, and 308 of the baseline algorithm 302 can then be merged per layer, as indicated by arrows 310 and 312. No aggregation is needed for the result set with the lowest resolution (304).

[0048] The layer results 314, 316, and 318 are then input to a smart result aggregation step 320 in order to produce the final result of the object recognition process (about Figure 4 which is described in further detail).

[0049] Figure 4 A block diagram 400 is depicted that details the pairwise layer comparison and final take-and-aggregation step used to produce the final result 402, in accordance with embodiments of the present application. In Figure 2 the depicted example of the layer concept, four different result sets 404, 406, 408, and 410 (corresponding logically to the layer results 314, 316, and 318 of Figure 3 ) are used as layer results. Adjacent instances (in the sense of different resolution layers) are compared pairwise, and the intersection (i.e., the logical AND) is built in intermediate layer data sets 412, 414, and 416, as Figure 4 shown. Embodiments of the present application can then merge the intermediate data sets 412, 414, 416 in a logical OR operation 418 to build the final result 402 and complete the smart result aggregation.

[0050] Furthermore, as Figure 4 shown, the result set labeled f=4 (410) has the lowest resolution (corresponding to the highest layer), and the result set f=1 (404) has the highest resolution, and can also represent a reference to global coordinates.

[0051] Figure 5 A diagram 500 depicting the boundary regions of the edges of the processed tiles according to embodiments of the present application. As shown, both image tile 502 and another image tile 504 have respective boundary regions. The baselines for image tile 502 and image tile 504 provide independent results for the respective boundaries / boundary regions 506 and 508 of the respective image tiles 502 and 504. The merged image tile 502 and 504 in the form of a rectangle 510 shows the results in global coordinates (i.e., in the coordinates of the original image having the highest resolution). As such, there are two separate partial detections in the result entity. The partial overlap is detected as a result of the boundary region processing (denoted with rectangle 512), and the two partial results are unified into one detection result 516. Figure 5

[0052] The large rectangle (encompassing the original image 514) shows the respective processing as part of the formal high definition / high resolution (i.e., high resolution) image or better having global coordinates. As a result, the defect (i.e., the identified object) is shown as a black scratch in the surface. Figure 5 The depiction can assume that the original image 514 is an image showing the surface of a component to be visually inspected.

[0053] The proposed method for improving object detection in high resolution images at inference time can provide a variety of advantages, technical effects, contributions, and / or improvements. In various scenarios, a high resolution image can be effectively and efficiently fed to a baseline object detection algorithm without the need to retrain the baseline algorithm that has been trained on low resolution images. However, as images with increasing resolution (i.e., more pixels per region) become available, and as the trained baseline algorithm is not suitable for processing the available resolution (e.g., more computational power is required due to longer computation cycles in the training and inference processes), special activities for producing digital image results with higher resolution are proposed in various embodiments of the present application.

[0054] Some embodiments of the present application can be fully automated and rely only on simple settings: e.g., the number of used hierarchical layers, the sensitivity threshold of the baseline model, and the pre-defined tile size (number of image tiles per digital image). Based on this, embodiments of the present application have already worked well using the default parameters of the baseline algorithm. Various embodiments of the present application also share the advantages of using deep learning techniques for defect detection, which can not require manual feature engineering, and in many cases, can outperform traditional machine learning methods with large performance margins. Further embodiments of the present application have also been validated on one defect detection task using real data by using Mask R-CNN (Region-based Convolutional Neural Network) as the baseline algorithm. ​

[0055] In particular, the ability to re-use a previously trained image identification model can be a considerable advantage when the real data has a resolution corresponding to a 4K or 8K camera, said previously trained image identification model having been trained using training images with low resolution as annotated training patterns. There is no need to re-train the baseline algorithm, so that embodiments of the present invention can be used during the inference phase. Thus, high resolution images and corresponding object identification can be processed with a baseline algorithm originally designed for and trained using lower resolution images. Some embodiments of the present invention can also be adapted to high resolution images of various resolutions by setting only a small set of configuration parameters.

[0056] Furthermore, embodiments of the present invention can also effectively address the precision (also denoted as positive predictive value) and recall (also called sensitivity of the algorithm) challenges of pattern identification, as the number of false positive detections and the number of false negative detections are minimized, respectively.

[0057] According to an advantageous embodiment of the present invention, the aggregation of the results of the baseline algorithm per layer (i.e. the first sub-step of the smart result aggregation) can also comprise extracting a polygon encoding the shape of the recognized object and mapping the local polygon coordinates for the image tiles to global coordinates for the image tile with the highest resolution. Thereby, the shape of the polygon with higher resolution in the results of the baseline algorithm is compressed, so that the compressed shape is comparable to the shape in the image tile with lower resolution. In this case, it can be noted that instead of the highest resolution coordinates available, other scaling mechanisms can also be used (i.e. abstract coordinates can be used). In addition, embodiments of the present invention can also operate to extend the image with lower resolution to match the image with high resolution. To this end, from one pixel to three or more pixels can be generated in the lower resolution image. The extraction of the polygon encoding the shape of the recognized object can also be seen as a more accurate image capturing technique than using only the minimal bounding box or a surrounding rectangle of the recognized image.

[0058] According to another advantageous embodiment of the present invention, the aggregation of the results of the baseline algorithm per layer (activity also related to the first sub-step of the smart result aggregation) can also comprise eliminating overlapping areas between adjacent image tile boundaries of the respective layer and merging detected partial objects of adjacent tiles into one detected object. For the reconstruction of seamless images, edge region handling can be advantageous. As a reminder: during decomposition, overlaps have been created.

[0059] According to a further embodiment of the present application, the result of the baseline algorithm can comprise at least one selected from the group comprising a class of the recognized item, a bounding box enclosing the identified object in the image patch, and a mask represented by a polygon surrounding the shape of the identified object in the image patch. In particular, the recognized item or object can relate to material defects or surfaces (e.g. rebar corrosion, cracks, rust, spalling and / or algae). Thus, embodiments of the present application can advantageously be used for the inspection of infrastructure components such as bridges, buildings, masts, pipes, pipelines and / or other industrial or infrastructure elements. Thus, in one embodiment, the object to be detected can be a material defect, in particular a material defect of the object to be detected.

[0060] According to a further embodiment of the present application, the baseline algorithm can be a Mask R-CNN algorithm or a Fast R-CNN algorithm. Both algorithms are known and often used in the context of object detection. Thereby, the input image is presented to a neural network, a selective search is run on the image and then the output regions from the selective search are used for classification feature extraction using a pre-trained convolutional neural network. Moreover, Fast R-CNN is based on a pre-trained convolutional neural network, wherein the last three classification layers have been replaced by new classification layers specific to the required object classes.

[0061] According to an additional embodiment of the present application, the resolution suitable for the baseline image recognition algorithm can be selected from the group comprising 224x224 pixels, 512x512 pixels, 800x800 pixels and 1024x800. Moreover, other resolutions can also be used. However, commonly used baseline algorithms usually work on a fixed resolution (e.g. 224x224), which makes it difficult to compare and determine the relationship between different resolution images.

[0062] According to a further embodiment of the present application, the baseline algorithm can be pre-trained such that the neural network model is built for the inference task of object identification. The pre-training can usually be done when using the baseline algorithm Mask R-CNN or Fast R-CNN. Thus, according to the proposed concept, the training that can have been performed on the low resolution image or filter can also be used for the high resolution image.

[0063] For completeness reasons, Figure 6A block diagram depicting an object recognition system 600 for improving object detection in high resolution images at inference time. The object recognition system 600 comprises a processor 604 communicatively coupled to a memory 602 storing instructions for causing the system to: (in particular by a receiver 606) receive a high resolution image; decompose (in particular by a decomposition unit 608) the received image into hierarchically organized layers of the image. Thereby, each layer comprises at least one image patch of the received image, and each image patch has a resolution suitable for (or in particular required, recommended or efficient for) a baseline image recognition algorithm.

[0064] The instructions further comprise (in particular by a baseline invocation module 610) applying the baseline algorithm to each of the image patches of each layer, and performing a result aggregation (in particular by a smart result aggregation unit) on the results of applying the baseline algorithm to the image patches of the layers. This particular functionality is implemented by a first aggregation module 614 adapted to aggregate the results of the baseline algorithm per layer, a pair-wise layer comparison module 616 adapted to perform a pair-wise layer comparison on the results of the baseline algorithm for adjacent pairs of layers, and a second aggregation module 618 adapted to perform a hierarchical aggregation of the results of the baseline algorithm according to the pair-wise layer comparison.

[0065] Additionally, the modules and units (in particular the memory 602, the processor 604, the receiver 606, the decomposition unit 608, the baseline invocation module 610, the smart result aggregation unit 612, the first aggregation module 614, the layer comparison module 616 and the second aggregation module 618) can be in communication contact with each other by means of a direct connection or by means of a system internal bus system 620 for data, signal and / or information exchange.

[0066] Embodiments of the present application can in fact be realized with any type of computer, regardless of the platform, as long as the platform is adapted to store and / or execute program code. Figure 7 A computing system 700 adapted to execute program code related to the proposed method is depicted as an example.

[0067] The computing system 700 is only one example of a suitable computer system and is not intended to suggest any limitation as to the scope of use or functionality of embodiments of the present application described herein, whether or not implementation is

[0068] As shown in Figure 7 The computer system / server 700 is shown in the form of a general-purpose computing device. The components of computer system / server 700 can include, but are not limited to, one or more processors or processing units 702, a system memory 704, and a bus 706 that couples various system components including system memory 704 to processor 702. Bus 706 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example, and not limiting, these architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus. Computer system / server 700 typically includes a variety of computer system readable media. Such media can be any available media that is accessible by computer system / server 700, and it includes both

[0069] The system memory 704 can include computer system readable media in the form of volatile memory, such as random access memory (RAM) 708 and / or cache memory 710. Computer system / server 700 can further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, a storage system 712 can be provided for reading from and writing to non-removable, non-volatile magnetic media (not shown and typically called a "hard drive"). Although not specifically shown, a magnetic disk drive can also be used for reading and writing to a removable, non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive can be used for reading from or writing to a removable, non-volatile optical disk (such as a CD-ROM, DVD-ROM or other optical media). Each of these devices can be connected to bus 706 by one or more data media interfaces. As will be further depicted and described below, memory 704 can include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the application.

[0070] Program / utility 716 having a set (at least one) of program modules 716 can be stored in memory 704 by way of example, and not limitation, as well as an operating system, one or more application programs, other program modules, and program data. Each of the operating system, one or more application programs, other program modules, and program data or some combination thereof, can include an implementation of a networking environment. Program modules 716 generally carry out the functions and / or methodologies of embodiments of the application as described herein.

[0071] Computer system / server 700 can also communicate with one or more external devices 718 such as a keyboard, a pointing device, a display 720, etc.; one or more devices that enable a user to interact with computer system / server 700; and / or any devices (e.g., network card, modem, etc.) that enable computer system / server 700 to communicate with one or more other computing devices. Such communication can occur via Input / Output (I / O) interface(s) 714. Still yet, computer system / server 700 can communicate with one or more networks such as a local area network (LAN), a general wide area network (WAN), and / or a public network (e.g., the Internet) via network adapter 722. As depicted, network adapter 722 can communicate with the other components of computer system / server 700 via bus 706. It should be understood that, although not specifically shown, other hardware and / or software components could also be used, such as an

[0072] Additionally, the object recognition system 600 for improving object detection in high resolution images at inference time can be attached to a bus system (e.g., bus 706).

[0073] The programs described herein are identified based upon the application for which they are implemented in a particular embodiment of the invention. However, it should be understood that any particular program identified herein can be implemented with program code that includes one or more programs stored in a computer readable storage medium (a computer program product) associated with a computer that can interface with a human operator. The computer readable storage medium for a computer program product can include one or more types of computer readable storage media, for example, but not limited to, secondary storage devices such as hard disks, floppy disks, CD-ROMs, DVD-ROMs, Blu-ray discs® or memory cards; or primary storage devices such as random access memory (RAM) or read only memory (ROM). The computer readable storage medium can also be, for example, and without limitation, a single storage device or multiple storage devices that store computer readable instructions thereon.

[0074] The present invention can be a system, a method, and / or a computer program product at any possible technical detail level of integration. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present invention.

[0075] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium can also include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

[0076] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions into the respective computing / processing device for execution by the processing device. The network adapter card or network interface can include a tangible computer readable storage medium for storing the computer readable program instructions executed by a processing device in the computing / processing device.

[0077] Computer readable program instructions for carrying out operations of the present application can be assembly instructions, instruction-set-architecture (ISA) instructions, machine- related instructions, microcode, firmware instructions, state-setting data, configuration data for an integrated circuit, or source code or object code written in any combination of one or more programming languages, including an object-oriented programming language such as Smalltalk, C++ or the like, and a procedural programming language such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present application.

[0078] Aspects of the present application are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0079] These computer readable program instructions can be provided to a processor of a computer or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and / or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including

[0080] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0081] The computer program product of the present application can be a computer program embedded in a computer readable medium such as a hard disk drive, a floppy disk, a RAM, a ROM, a flash memory, a CD-ROM, a DVD, a magnetic tape, an optical storage device, a magnetic storage device, etc. The computer readable medium can be a computer readable storage medium or a computer readable signal medium. The computer readable storage medium can include tangible storage media or memory media such as magnetic tapes, floppy disks, RAMs, ROMs, hard disks, solid state drives, flash memories, CD-ROMs, DVDs, etc. The computer readable signal medium can include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. The computer readable program code embodied on the computer readable signal medium can be transmitted using any appropriate medium, including but not limited to wireless, wire line, optical fiber cable, RF, etc. The computer program product of the present application can be a computer program tangibly embodied in a computer readable medium or a computer program transmitted by a computer readable signal medium.

[0082] The description of the various embodiments of the present application has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the application. The terminology used herein was chosen to best explain the principles of the embodiment, the practical application, or technical improvement over the existing technology, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.

[0083] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0084] All means or steps plus function elements in the claims that follow on the corresponding structure, material, act or the like are intended to embrace any structure, material, or acts for performing the function during the recited claim elements along with any normal variations thereof. The description of the application has been presented for purposes of illustration and description, but is not intended to be exhaustive or to limit the application to the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the application. The embodiment was chosen and described in order to best explain the principles of the application and the practical application, and to enable others of ordinary skill in the art to understand the application for various embodiments with various modifications as are suited to the particular use contemplated.

Claims

1. A computer-implemented method comprising: receiving, by one or more processors, a high-resolution image; decomposing, by one or more processors, the received image into hierarchically organized layers of images, wherein each layer comprises at least one image block of the received image, and wherein each of the image blocks has a corresponding resolution suitable for a baseline image recognition algorithm; applying, by one or more processors, the baseline image recognition algorithm to each of the image blocks in each layer; as well as performing, by one or more processors, result aggregation on the results of applying the baseline image recognition algorithm to the image block of the layer, Wherein performing the result aggregation on the result of applying the baseline image recognition algorithm to the image block of the layer further comprises: aggregating, by one or more processors, results of the baseline image recognition algorithm by layer; performing, by one or more processors, pairwise comparisons of the baseline image recognition algorithm's results at adjacent pairwise levels; and Hierarchical aggregation of the baseline image recognition algorithm results is performed by one or more processors based on the pairwise layer comparisons.

2. The method of claim 1 , wherein performing the result aggregation on the results of applying the baseline image recognition algorithm to the image block of the layer further comprises: extracting, by one or more processors, a polygon encoding the shape of the identified object; as well as One or more processors map local polygon coordinates for the image patch to global coordinates for the image patch having the highest corresponding resolution, thereby compressing the shapes of polygons with the higher resolution in the results of the baseline image recognition algorithm so that the compressed shapes are comparable to the shapes in the patch with the lower resolution.

3. The method according to claim 2, further comprising: Eliminating, by one or more processors, overlapping regions between boundaries of adjacent image blocks of corresponding layers; as well as One or more processors merge the detected partial objects of the adjacent image blocks into one detected object.

4. The method according to claim 3, wherein: Performing pairwise layer comparisons also includes: comparing, by one or more processors, compressed relative shapes of image blocks of adjacent layers; and An intersection of shapes is constructed by one or more processors based on the comparison of the compressed related shapes, thereby constructing N intermediate image layers, where N is one less than the number of hierarchically organized layers.

5. The method of claim 4, wherein performing hierarchical aggregation further comprises: A pixel-by-pixel union of all N intermediate image layers is constructed by one or more processors, thereby constructing a final image having a resolution equal to the resolution of the received high-resolution image, the high-resolution image including the polygonal surround of the detected object.

6. The method of claim 1 , wherein the result of the baseline image recognition algorithm comprises at least one item selected from the group consisting of: a category of the recognized item, a bounding box surrounding the identified object in the image patch, and a mask represented by a polygon surrounding the shape of the recognized object in the image patch.

7. The method of claim 1, wherein the baseline image recognition algorithm is a Masked Region-Based Convolutional Neural Network (R-CNN) algorithm or a Fast R-CNN algorithm.

8. The method of claim 1, wherein a resolution suitable for a baseline image recognition algorithm is selected from the group consisting of: 224x224 pixels, 512x512 pixels, 800x800 pixels, and 1024x800 pixels.

9. The method of claim 1, wherein the baseline image recognition algorithm is pre-trained such that a neural network model is constructed for inference tasks for object recognition.

10. The method according to claim 1, wherein the object to be inspected is a material defect.

11. A computer program product comprising: One or more computer-readable storage media and program instructions stored on the one or more computer-readable storage media, the program instructions comprising: Program instructions for executing the method according to any one of claims 1 to 10.

12. A computer system comprising: one or more computer processors; one or more computer-readable storage media; as well as Program instructions stored on the computer-readable storage medium for execution by at least one of the one or more processors, the program instructions comprising: Program instructions for executing the method according to any one of claims 1 to 10.

Images