Method, computer program product, and computer system for improving object detection in high resolution images

Abstract

To provide a method, a computer program and a system for improving object detection within a high-resolution image in inference.SOLUTION: A method includes receiving a high-resolution image by one or a plurality of processors. The method further includes decomposing the received image into hierarchically organized layers of images by the one or the plurality of processors. The layers each include at least one image tile of the received image. Image tiles each have corresponding resolution matching base-line image recognition algorithm. The method further includes applying the base-line algorithm to the respective image tiles of the respective layers by the one or the plurality of processors. The method furthermore includes executing result aggregation of results of application of the base-line algorithm to the image tiles of the layers by the one or the plurality of processors.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates generally to the field of object detection, and more particularly to object detection in high resolution images. [Background technology]

[0002] Research institutes and companies have devoted enormous efforts to implementing AI-driven applications to automate processes, achieve more human-friendly user interfaces, or help analyze large amounts of data. Deep learning methods have shown remarkable success, outperforming classical machine learning solutions. Two main factors have driven the success of deep learning techniques: primarily the availability of high-performance computing infrastructure and the availability of large, labeled datasets. Deep learning methods are often used in image classification, object detection, video analysis, text translation, and audio genre classification, to name a few. In particular, current, successful models operating on pixel data can utilize convolutional neural networks. Therefore, deep learning methods working with image data can be divided into three main tasks: a) classification, b) detection, and c) segmentation. All three tasks share a single input but define what the method must produce. In classification, a single class label is predicted (e.g., an image showing a dog), in detection, a bounding box is generated (e.g., a dog is located in a rectangle [X,Y,dX,dY]), and in segmentation, the pixels to which the intended object belongs are predicted (e.g., pixels p1,p2,p3,…,pN indicate a dog).

[0003] Automated defect detection defines a subset of the general task of object detection, where the goal is to identify (detect and / or segment) defects on industrial images. Applications can include use cases from a variety of fields, including the medical field (e.g., identifying human anatomical structures from X-ray scans), materials manufacturing (e.g., identifying defects in produced steel or other products), or defect detection in civil infrastructure (e.g., bridges or high-rise buildings). Summary of the Invention [Problem to be solved by the invention]

[0004] The present disclosure aims to provide methods, computer program products, and systems for improving object detection in high-resolution images during inference. [Means for solving the problem]

[0005] Aspects of the present invention disclose methods, computer programs, and systems for improving object detection in high-resolution images at inference time. The method includes one or more processors receiving a high-resolution image. The method further includes the one or more processors decomposing the received image into hierarchically organized layers of images. Each layer includes at least one image tile of the received image. Each image tile has a corresponding resolution suitable for a baseline image recognition algorithm. The method further includes the one or more processors applying a baseline algorithm to each of the image tiles of each layer. The method further includes the one or more processors performing result aggregation of results of applying the baseline algorithm to the image tiles of the layer.

[0006] In another embodiment, performing result aggregation of results of application of the baseline algorithm to the image tiles of the layer further includes one or more processors aggregating baseline algorithm results by layer, one or more processors performing pairwise layer comparison of the baseline algorithm results for adjacent pairwise layers, and one or more processors performing hierarchical aggregation of the baseline algorithm results responsive to the pairwise layer comparison.

[0007] It should be noted that embodiments of the invention are described with reference to different subject matters. In particular, some embodiments are described with reference to method-type claims, while other embodiments are described with reference to apparatus-type claims. However, those skilled in the art will understand from the above and following descriptions that, unless otherwise specified, any combination of features belonging to one type of subject matter, as well as any combination between features relating to different subject matters, in particular between features of a method-type claim and a device-type claim, is considered to be disclosed within the present application.

[0008] The above-defined and further aspects of the present invention will be apparent from and will be explained with reference to the example embodiments described hereinafter, without the invention being limited thereto. Preferred embodiments of the present invention will be described, by way of example only, with reference to the following drawings, in which: [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram of an embodiment of a method for improving object detection in high-resolution images during inference, according to an embodiment of the present invention. [Figure 2] FIG. 1 is a block diagram of an embodiment in which an original high-resolution image is used as the basis for the described processing, according to an embodiment of the present invention. [Figure 3]FIG. 1 is a block diagram detailing the feeding of a set of input tiles to an object recognition baseline algorithm, according to one embodiment of the present invention. [Figure 4] FIG. 10 is a block diagram detailing the steps of pairwise layer comparison and the final union aggregation step to produce the final result, according to one embodiment of the present invention. [Figure 5] FIG. 10 is an illustration of border region handling at the edge of a tile, according to one embodiment of the present invention. [Figure 6] FIG. 1 is a block diagram of an object recognition system for improving object detection in high-resolution images during inference, according to one embodiment of the present invention. [Figure 7] 1 is a block diagram of an embodiment of a computing system including the object recognition system of the present invention, according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0010] Embodiments of the present invention recognize that many object detection use cases present extremely challenging problem instances, even using state-of-the-art deep learning methods. The causes are manifold and can be explained by variations in light, including, among other factors, but not limited to, different lighting conditions, different image resolutions, different cameras used to capture the images (e.g., different lens distortions, camera sensitivity (ISO), etc.), different viewpoints, different zoom levels, occlusions by obstacles (e.g., a tree in front of a bridge support), different backgrounds (e.g., two bridges that look different), and unexpected background obstructing objects (e.g., people, cars, and boats near a bridge when the bridge and its defects are the main focus). Additionally, many defects lack clear boundaries, which makes them more difficult to detect as objects.

[0011] Further embodiments of the present invention recognize that traditional deep learning methods generally operate on relatively small image sizes. For example, state-of-the-art image classification algorithms evaluated on the CIFAR-10 dataset use inputs with a geometry of 32 x 32 pixels. Furthermore, the widely used ImageNet provides a variety of image sizes, and most algorithms follow a uniform training and evaluation setup that resizes images to a fixed size of 224 x 224 pixels. Object detection algorithms such as Mask R-CNN (region-based convolutional neural network) operate at a fixed scale of 1024 pixels. However, in contrast, embodiments of the present invention recognize that high-resolution images are almost freely available today. Many cameras support 2K, 4K, and 8K modes, with premium cameras available supporting 16K to 64K resolutions. Capturing images in such a setup results in processing images with pixel widths that are 2 to 64 times larger than the expected size for the detector's original configuration.

[0012] Embodiments of the present invention recognize that defects are often small features located in only a few sparse locations within a high-resolution image. Simply resizing an image to a smaller resolution is problematic because doing so results in significant resolution loss. Additionally, tiling a single high-resolution image into smaller images and processing them separately helps preserve high resolution. However, tiling entails additional overhead required to process overlapping regions (e.g., defects partially visible in one tile), which scales the workload for a single image to the number of tiles extracted from that image, thereby increasing the workload. Embodiments of the present invention recognize that, as required for deep learning techniques, it is important for training and test images to follow the same statistics to obtain good generalization behavior of deep learning methods. Validating various tiling configurations and verifying that the statistics of one image tile in training match the statistics of image tiles in test scenarios becomes a nontrivial task. Furthermore, embodiments of the present invention recognize that changes applied in the training domain trigger retraining of the model, which is computationally very intensive and therefore ineffective and expensive.

[0013] Additional embodiments of the present invention recognize that commonly known object recognition algorithms are repeatedly reused to implement the first development cycle. The algorithms typically rely on a fixed image resolution. However, camera resolutions are rapidly increasing, and as a result, the image resolutions assumed by known object recognition algorithms may lag behind their development. Additionally, even if object recognition algorithms keep up with the availability of cameras with ever-increasing resolutions, embodiments of the present invention recognize that the computational burden of retraining existing neural networks and reconfiguring their hyper-parameters is enormous, which is considered a major drawback of conventional approaches. To overcome this type of impasse, embodiments of the present invention recognize the need to provide robust object recognition capabilities without the need to retrain existing image recognition algorithms.

[0014] In the context of this description, the following rules, terms or expressions, or combinations thereof, may be used:

[0015] The term "object detection" can refer to the activity of a system supported by a method for identifying one or more predefined items, samples, or patterns within a given digital image.

[0016] The term "high-resolution image" can refer to an image having a higher resolution than a given algorithm used as the input resolution for a typical image object detection process. Thus, the resolution of the input required by the baseline algorithm does not match the resolution of the high-resolution image. Therefore, there may be a need for a means to process a high-resolution image, or at least a portion thereof (e.g., an image tile), using an already pre-trained baseline algorithm without the need to retrain, reconfigure, or redesign the pre-trained baseline algorithm.

[0017] The term "inference time" can refer to the time when a trained machine learning system (e.g., a convolutional neural network) is able to predict the class or segmentation of an input image. In contrast to inference time is training time. While upfront training of a machine learning system can require a lot of computational power and time, the inference activity of a trained machine learning system can be optimized to function with very few computational resources.

[0018] The term "decomposing an image" can refer to the process of cutting a given digital image into parts (e.g., rectangular pieces) that can also be referred to as image tiles.

[0019] The term "hierarchically organized layers" can refer to multiple layers containing a predefined number of sub-images of a given original (i.e., received) digital image within a given layer, whereby the layers can be distinguished by different resolutions. The lowest layer can be the one with the highest resolution, i.e., the largest number of pixels available in a given digital image, and also provides the basis for global coordinates. The highest layer can be defined as the one with the smallest number of pixels for a given digital image, i.e., the one with the lowest resolution.

[0020] The term "suitable resolution" (e.g., a resolution suitable for a particular baseline algorithm) can refer to a digital image resolution optimized for an object recognition algorithm, such as Mask R-CNN or Fast R-CNN. For example, the algorithms operate at a resolution of 224 x 224 pixels. Therefore, a digital image with 1000 x 1000 pixels would not be suitable for a given baseline algorithm.

[0021] The term "trained baseline image recognition algorithm" can refer to an image detection algorithm or an image recognition algorithm, or both, that is further fully implementable in hardware (e.g., using a memristive device crossbar) and that has been trained such that hyper-parameters and weight coefficients (e.g., of a convolutional neural network) are defined and a neural network model is developed accordingly. The trained baseline algorithm can then be used for object recognition tasks at inference time.

[0022] The term "smart result aggregation" may refer to a multi-step process useful for the object recognition process proposed herein. Smart result aggregation may include at least the steps of (i) aggregating results of baseline algorithms layer by layer, (ii) performing pairwise layer comparisons, and (iii) performing hierarchical aggregation of the comparison results. Details of the multi-step process are defined by the dependent claims and are explained in more detail in the context of the drawings.

[0023] The term "overlap area" can refer to a portion of an image tile that may be part of two adjacent image tiles of the same digital image. The image portion can be part of a left image tile and a right image tile that are side-by-side.

[0024] The term "intermediate image layer" can refer to a layer constructed by layer-wise comparison of intermediate results as part of the smart result aggregation process. Thus, a given number of M layers (e.g., four layers) results in N intermediate image layers, where N=M-1. A practical example is described in further detail with respect to FIG. 4.

[0025] The term "pixel-wise union" can refer to the process of combining two shapes with a logical "OR" function. The "OR" function can be applied to a binary mask that encodes the shapes. Similarly, the same term can refer to the process of taking the "union" of two shapes that can be encoded as a polygon.

[0026] The term "pixel-wise intersection" can refer to the process of combining two shapes with a logical "AND" function. The "AND" function can be applied to a binary mask that encodes the shapes. Similarly, the same term can refer to the process of "intersecting" two shapes, which can then be encoded as a polygon.

[0027] The term "item to be recognized" can refer to an object in a digital image that has a predefined shape (i.e., shape or other characteristic features) that the learning system being used is trained to recognize or identify. In this sense, the item to be recognized can be identical to the object to be recognized.

[0028] The term "Mask R-CNN algorithm" may refer to a known convolutional neural network algorithm used for instance segmentation based on known proceeding architectures for object detection, whereby image inputs may be presented to a neural network. A selected search process may be performed on the received digital image, and the output regions from the selected search process may then be used for feature extraction and classification using a pre-trained convolutional neural network.

[0029] The term "Fast R-CNN algorithm" can refer to an extended version of the Mask R-CNN algorithm. The Fast R-CNN algorithm can still use the selection search algorithm to obtain region candidates, but can add a region of interest (ROI) pooling module. The Fast R-CNN algorithm can extract a fixed-size window from the feature map to obtain the final class label and bounding box for a given object in the received digital image. The advantage of this approach is that convolutional neural networks are now trainable end-to-end.

[0030] The term "pre-trained" can refer to an image or object recognition system that has been trained prior to use. In particular, a pre-trained object recognition system or method can be used as a tool to process digital images to be classified that are not immediately suitable (e.g., due to resolution mismatch) for the previously trained baseline algorithm used. In contrast (e.g., compared to conventional systems), the concepts proposed herein use decomposition of a given (i.e., received) digital image, along with smart result aggregation, to overcome mismatches between the resolution of the received digital image and the resolution required by the baseline object recognition algorithm.

[0031] The term "neural network model" can refer to the sum of all weights of a given neural network, along with the used logical configuration of the neural network (i.e., the hyper-parameters of the underlying machine learning system, here a convolutional neural network).

[0032] In the following, a detailed description of the figures is provided. All designations in the figures are schematic. First, a block diagram of one embodiment of the inventive method for improving object detection in high-resolution images at inference time is given. Afterwards, further embodiments are described, as well as embodiments of an object recognition system for improving object detection in high-resolution images at inference time.

[0033] 1 illustrates a block diagram of an embodiment of a method 100 for improving object detection (e.g., defect detection) in high-resolution images during inference, according to an embodiment of the present invention. In an exemplary embodiment, an object recognition system 600 (shown in FIG. 6) can perform the processing steps of method 100 (i.e., perform FIG. 1) according to embodiments of the present invention. In an additional exemplary aspect, object recognition system 600 (in combination with method 100) can perform the operations shown and described in further detail with respect to FIGS. 2-5, according to various embodiments of the present invention.

[0034] In step 102, method 100 receives a high-resolution image. In an example embodiment, method 100 receives a digital image with a resolution greater than the image resolution used by the underlying baseline image recognition algorithm (e.g., Mask R-CNN).

[0035] In step 104, the method decomposes the received image into hierarchically organized layers of images. In an example embodiment, each layer contains at least one image tile of the received image (only the image with f=max has only one, all other layers have more tiles). In an additional embodiment, each of the image tiles has a resolution suitable (e.g., required or recommended) for a pre-trained baseline image recognition algorithm.

[0036] In step 106, method 100 applies a baseline algorithm to each of the image tiles in each layer. In an example embodiment, method 100 may operate to identify regions of interest, object bounding boxes (i.e., rectangles enclosing defects and classifications), or alternatively or additionally, masks, polygons, shapes, or combinations thereof based on pre-training.

[0037] Additionally, in process 108, method 100 performs smart result aggregation. For example, method 100 performs smart result aggregation of the results of applying the baseline algorithm to the image tiles of the layer using a three-step approach, as described in steps 110-114.

[0038] In step 110, method 100 aggregates baseline algorithm results by layer. In step 112, method 100 performs pairwise layer comparisons of baseline algorithm results for adjacent pairs of layers. In step 114, method 100 performs hierarchical aggregation of baseline algorithm results according to pairwise layer comparisons. This allows method 100 to utilize a consistent scaling factor, which means that, depending on the resolution, one pixel at one resolution can be compared to four pixels at another resolution or 16 pixels at a higher resolution, or both. If the number of pixels that are white or black is equally distributed (i.e., 50 / 50) at resolutions higher than the lowest resolution (e.g., in black and white images, or one or multiple color channels), a random decision is made for one of the two color options.

[0039] FIG. 2 shows a block diagram 200 of one embodiment in which an original received high-resolution image 202 is used as the basis for the operation of an embodiment of the present invention (e.g., the process of method 100). Rectangle 204 represents the fixed working size of the working resolution used and / or required by the baseline image recognition algorithm. Therefore, the images of different layers, specifically Layer 1, Layer 2, and Layer 3, corresponding to f=1.0, f=2.0, and f=6.0, respectively, must be cut into tiles. The first layer image 206 is cut into 24 tiles, so that each tile has a number of pixels equal to the working size of the image of the baseline algorithm.

[0040] Correspondingly, the image 208 in Layer 2, which has a lower resolution than the image in Layer 1, requires only six tiles, while the image 210 (Layer 3) with the lowest resolution requires only a single tile because its corresponding resolution is consistent with the working size of the baseline algorithm. The number of layers is configurable and can depend on the image resolution of the received digital image.

[0041] As a result of the tiling step, embodiments of the present invention generate sets of tiles 212, 214, and 216, with the number of tiles per set increasing as the resolution of images 206, 208, and 210 increases. The sets of image tiles are then used as input to the baseline algorithm. As an example, tile 216 is the bottom right corner of image 206, and image tile 214 is the top center portion of image 208 in layer 2. The entire process is described in further detail with respect to FIG. 3.

[0042] FIG. 3 shows a block diagram 300 detailing the feeding of sets of input tiles 212, 214, and 216 (of FIG. 2) to an object recognition baseline algorithm 302, according to an embodiment of the present invention. In various embodiments, the object recognition baseline algorithm 302 can be used in a pre-trained form without the need for additional training. For example, the object recognition baseline algorithm 302 can be used as available. The sets of outputs 304, 306, and 308 of the baseline algorithm 302 can then be merged layer-by-layer, as represented by arrows 310 and 312. For the result set with the lowest resolution (304), no aggregation is required.

[0043] Layer results 314, layer results 316, and layer results 318 are then input into a smart result aggregation step 320 (described in more detail with respect to FIG. 4) to generate the final results of the object recognition process.

[0044] FIG. 4 shows a block diagram 400 detailing the steps of pairwise layer comparison and final merging and aggregation to generate final result 402, according to an embodiment of the present invention. In the illustrated example of the layer concept in FIG. 4, four different result sets 404, 406, 408, and 410 (logically corresponding to layer result 314, layer result 316, and layer result 318 in FIG. 3) are used as layer results. Adjacent instances (in the sense of different resolution layers) are compared pairwise, and an intersection (i.e., a logical "AND") is constructed in intermediate layer data sets 412, 414, and 416, as shown in FIG. 4. An embodiment of the present invention can then merge intermediate data sets 412, 414, and 416 with a logical "OR" operation 418 to construct final result 402 and complete the smart result aggregation.

[0045] Furthermore, as shown in Figure 4, the result set labeled with f = 4 (410) has the lowest resolution (corresponding to the highest layer), and the result set with f = 1 (404) has the highest resolution, which can also represent a global coordinate reference.

[0046] FIG. 5 shows a diagram 500 of a border region operation at the edge of a tile, according to an embodiment of the present invention. As shown in FIG. 5, a tile 502 and another tile 504 both have corresponding border regions. The baselines of the tile 502 and the tile 504 provide independent results for the edge / boundary areas 506 and 508 of each tile 502 and 504, respectively. The merged tiles 502 and 504 in the form of a rectangle 510 show the results in global coordinates (i.e., in the coordinates of the original image with the highest resolution). Thereby, two separate partial detections exist in the result entity. As a result of the border region operation (represented as a rectangle 512), a partial overlap is detected, and the two partial results are unified into one detection result 516.

[0047] The large rectangle (surrounding the original image 514) shows the corresponding process as part of a regular high definition / high resolution (i.e., high resolution) image, or better in global coordinates. As a result, defects (i.e., recognized objects) are shown as black scratches on the surface. The description of Figure 5 can be assumed that the original image 514 was an image showing the surface of an element to be visually inspected.

[0048] The proposed method for improving object detection in high-resolution images during inference can provide numerous advantages, technical effects, contributions, and / or improvements. In various scenarios, high-resolution images can be effectively and efficiently fed into a baseline object detection algorithm without the need to retrain a baseline algorithm trained on very low-resolution images. However, as images of ever-increasing resolution (i.e., greater numbers of pixels per area) become available, and as trained baseline algorithms are not adapted to handle the available resolutions (e.g., due to longer computational cycles during training and inference, and therefore greater required computational power), special efforts are proposed in various embodiments of the present invention to generate results for digital images with very high resolution.

[0049] Some embodiments of the present invention can be fully automated, relying only on simple settings, such as the number of hierarchical layers used, the sensitivity threshold of the baseline model, and a predefined tile size (number of tiles per digital image). Based on this, embodiments of the present invention can already function satisfactorily using the default parameters of the baseline algorithm. Various embodiments of the present invention also share the advantage of using deep learning techniques for defect detection, which do not require manual feature engineering and can often outperform traditional machine learning methods by a large margin. Further embodiments of the present invention were also validated using real data for a type of defect detection task using Mask R-CNN (region-based convolutional neural network) as the baseline algorithm.

[0050] In particular, being able to reuse a previously trained image recognition model that used lower-resolution training images as annotated training patterns can be a significant advantage when the actual data has a resolution corresponding to a 4K or 8K camera. Retraining of the baseline algorithm is not required, and as a result, embodiments of the present invention can be used during the inference phase. Thus, high-resolution images and their respective object recognition can be processed using a baseline algorithm originally designed for and trained using low-resolution images. Some embodiments of the present invention can also be adapted to high-resolution images of various resolutions by setting only a small set of configuration parameters.

[0051] In addition, embodiments of the present invention can also effectively address the challenges of precision (also expressed as positive predicted value) and recall (also known as the sensitivity of the algorithm) of pattern recognition in that they minimize the number of false-positive detections and the number of false-negative detections, respectively.

[0052] According to one advantageous embodiment of the present invention, the aggregation of the results of the baseline algorithms layer by layer (i.e., the first substep of smart result aggregation) can further include extracting polygons encoding the shapes of recognized objects and mapping the local polygon coordinates used in the image tiles to the global coordinates used in the image tiles with the highest resolution. This compresses the polygonal shapes of the results of the baseline algorithms with higher resolutions, so that the compressed shapes are equivalent to the shapes of the tiles with lower resolutions. In this regard, it should be noted that other scaling mechanisms can be used (i.e., abstract coordinates can be used) instead of the coordinates of the highest resolution. Additionally, embodiments of the present invention can also be operated to enlarge an image with a lower resolution to match an image with a higher resolution. For this purpose, three or more pixels can be generated in the lower resolution image from one pixel. Extracting polygons encoding the shapes of recognized objects can also be considered a much more precise image capture technique than simply using a minimally bounding or enclosing rectangle of the recognized image.

[0053] According to another advantageous embodiment of the present invention, the aggregation of the results of the baseline algorithm per layer (an activity also related to the first substep of smart result aggregation) can further include removing overlapping areas between adjacent image tile edges of each layer and merging detected subobjects of adjacent tiles into a single detected object. The edge region manipulation can be advantageous for reconstructing a seamless image. Note: The overlaps were created during decomposition.

[0054] According to another embodiment of the present invention, the results of the baseline algorithm may include at least one selected from the group consisting of a recognized item or object class, a bounding box surrounding the identified object in the image tile, and a mask represented by a polygon surrounding the shape of the recognized object in the image tile. In particular, the recognized item or object may be associated with a material defect or surface (e.g., rebar corrosion, cracks, rust, peeling, or algae, or a combination thereof). As a result, embodiments of the present invention may be advantageously used for inspecting infrastructure components such as bridges, buildings, masts, pipes, pipelines, or other industrial or infrastructure elements, or a combination thereof. Therefore, in an exemplary embodiment, the object to be detected may be a material defect in the object to be inspected.

[0055] According to another embodiment of the present invention, the baseline algorithm can be the Mask R-CNN algorithm or the Fast R-CNN algorithm. Both algorithms are known and often used in the context of object detection. An input image is presented to a neural network, a selective search is performed on the image, and the output region from the selective search is then used for feature extraction and classification using a pre-trained convolutional neural network. Furthermore, Fast R-CNN is based on a pre-trained convolutional neural network, with the last three classification layers replaced with new classification layers specific to the required object class.

[0056] According to additional embodiments of the present invention, a suitable resolution for the baseline image recognition algorithm can be selected from the group of 224 x 224 pixels, 512 x 512 pixels, 800 x 800 pixels, and 1024 x 800 pixels. Additionally, other resolutions can be used. However, commonly used baseline algorithms typically operate at a fixed resolution (e.g., 224 x 224), making it difficult to compare images of different resolutions and determine their relationship.

[0057] According to a further embodiment of the present invention, the baseline algorithm can be pre-trained, as can a neural network model built for the inference task of object recognition. Pre-training is typically performed when using the baseline algorithms Mask R-CNN or Fast R-CNN. Thus, training that may have been performed on low-resolution images or filters can be used with the proposed concept for high-resolution images.

[0058] For completeness, Figure 6 shows a block diagram of an object recognition system 600 for improving object detection in high-resolution images during inference. The object recognition system 600 includes a processor 604 communicatively coupled to a memory 602 that stores instructions for causing the system to receive (particularly via a receiver 606) a high-resolution image and decompose (particularly via a decomposition unit 608) the received image into hierarchically organized layers of images, whereby each layer includes at least one image tile of the received image, each of the image tiles having a resolution suitable (or particularly required, recommended, or efficient) for a baseline image recognition algorithm.

[0059] The instructions further include applying a baseline algorithm to each of the image tiles of each layer (particularly by the baseline calling module 610) and performing smart result aggregation of results of applying the baseline algorithm to the image tiles of the layer (particularly by the smart result aggregation unit). Certain functions are realized by a first aggregation module 614 configured to aggregate baseline algorithm results by layer, a layer comparison module 616 configured to perform pairwise layer comparisons of baseline algorithm results for adjacent pairs of layers, and a second aggregation module 618 configured to perform hierarchical aggregation of baseline algorithm results as a function of the pairwise layer comparisons.

[0060] In addition, the modules and units (particularly the memory 602, the processor 604, the receiver 606, the decomposition unit 608, the baseline calling module 610, the smart result aggregation unit 612, the first aggregation module 614, the layer comparison module 616, and the second aggregation module 618) can be in communicative contact with each other by direct connection or by exchange of data, signals, or information, or a combination thereof, over a system internal bus system 620.

[0061] Embodiments of the present invention may be implemented in conjunction with virtually any type of computer, regardless of the platform suitable for storing and / or executing program code. Figure 7 illustrates, by way of example, a computing system 700 suitable for executing program code associated with the proposed method.

[0062] Computing system 700 is merely one example of a suitable computer system, and whether computer system 700 is capable of implementing and / or performing any of the functions described above is not intended to suggest any limitation as to the scope of use or functionality of the embodiments of the invention described herein. Computer system 700 has components that operate in numerous other general-purpose or special-purpose computing system environments or configurations. Examples of well-known computing systems, environments, or configurations, or combinations thereof, that may be suitable for use with computer system / server 700 include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the above systems or devices. Computer system / server 700 may be described in the general context of computer system-executable instructions, such as program modules, executed by computer system 700. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. Computer system / server 700 may be practiced in a distributed cloud computing environment where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media, including memory storage devices.

[0063] As shown in FIG. 7 , computer system / server 700 is illustrated in the form of a general-purpose computing device. Components of computer system / server 700 may include, but are not limited to, one or more processors or processing units 702, a system memory 704, and a bus 706 that couples various system components, including the system memory 704, to the processing unit 702. Bus 706 represents any one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor bus or local bus using any of a variety of bus architectures. By way of example, and without limitation, such architectures include an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus. Computer system / server 700 typically includes a variety of computer system-readable media. Such media can be any available media that is accessible by computer system / server 700 and includes both volatile and non-volatile media, removable and non-removable media.

[0064] The system memory 704 may include computer system-readable media in the form of volatile memory, such as random access memory (RAM) 708 and / or cache memory 710. The computer system / server 700 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 712 may be provided for reading from and writing to a non-removable, non-volatile magnetic medium (not shown, commonly referred to as a "hard drive"). Although not shown, a magnetic disk drive may be provided for reading from and writing to a removable, non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive may be provided for reading from and writing to a removable, non-volatile optical disk, such as a CD-ROM, DVD-ROM, or other optical medium. In such an example, each may be connected to the bus 706 by one or more data media interfaces. As further illustrated and described below, the memory 704 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of embodiments of the present invention.

[0065] By way of example, and not limitation, a set of program modules 716 (at least one) having a program / utility, as well as an operating system, one or more application programs, other program modules, and program data, may be stored in memory 704. Each of the operating system, one or more application programs, other program modules, and program data, or a combination thereof, may include an implementation of a networking environment. The program modules 716 generally perform the functions and / or methods of embodiments of the present invention as described herein.

[0066] The computer system / server 700 may also communicate with one or more external devices 718, such as a keyboard, pointing device, display 720, etc., one or more devices that allow a user to interact with the computer system / server 700, or any device (e.g., a network card, modem, etc.) that allows the computer system / server 700 to communicate with one or more other computing devices, or a combination thereof. Such communication may occur via an input / output (I / O) interface 714. Furthermore, the computer system / server 700 may communicate with one or more networks, such as a local area network (LAN), a general wide area network (WAN), or a public network (e.g., the Internet), or a combination thereof, via a network adapter 722. As shown, the network adapter 722 may communicate with other components of the computer system / server 700 via a bus 706. It should be understood that other hardware and / or software components, not shown, may be used with the computer system / server 700. Examples include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archive storage systems.

[0067] Additionally, the object recognition system 600 for improving object detection in high resolution images during inference can be attached to a bus system (eg, bus 706).

[0068] The programs described herein are identified based on the application for which they are implemented in a particular embodiment of the invention. However, it should be understood that the nomenclature of specific programs herein is used merely for convenience, and thus the invention should not be limited to the particular application identified and / or implied merely by such nomenclature.

[0069] The present invention may be a system, method, or computer program product, or a combination thereof, integrated at any possible level of technical detail. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions for causing a processor to perform aspects of the present invention.

[0070] A computer-readable storage medium may be any tangible device capable of retaining and storing instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanical coding devices such as punch cards or ridge structures in grooves on which instructions are recorded, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as being ephemeral signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through fiber optic cable), or electrical signals transmitted through wires.

[0071] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to each computing / processing device or to an external computer or storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, fiber optic transmission cables, wireless transmission cables, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and transfers the computer-readable program instructions for storage on a computer-readable storage medium within the respective computing / processing device.

[0072] Computer-readable program instructions for carrying out operations of the present invention can be either source code or object code written in any combination of one or more programming languages, including assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for integrated circuits, or object-oriented programming languages ​​such as Smalltalk®, C++, etc., and procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer-readable program instructions can execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (e.g., through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) can execute computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry to perform aspects of the present invention.

[0073] Aspects of the present invention are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0074] These computer-readable program instructions may be provided to a processor of a computer or other programmable data processing apparatus to create a machine, such that the instructions, when executed by the processor of the computer or other programmable data processing apparatus, create means for performing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. These computer-readable program instructions may also be stored on a computer-readable storage medium, capable of instructing a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, such that the computer-readable storage medium on which the instructions are stored comprises an article of manufacture containing instructions implementing aspects of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0075] The computer-readable program instructions may also be loaded into a computer, other programmable data processing apparatus, or other device to generate a computer-implemented process and cause the computer, other programmable apparatus, or other device to perform a series of operational steps, such that the instructions, which execute on the computer, other programmable apparatus, or other device, perform the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0076] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function(s). In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may actually be accomplished as a single step, or may be executed concurrently, substantially concurrently, partially, or fully in a time-overlapping manner, or the blocks may sometimes be executed in reverse order, depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented in a dedicated hardware-based system that performs the specified functions or operations or executes a combination of dedicated hardware instructions and computer instructions.

[0077] The description of various embodiments of the present invention has been presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the present invention. The terms used herein have been chosen to best explain the principles of the embodiments, practical applications, or technical improvements over commercially available technologies, or to enable those skilled in the art to understand the embodiments disclosed herein.

[0078] The terms used herein are for the purpose of describing particular embodiments only and are not intended to limit the invention. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. It will be further understood that the terms "comprises" and / or "comprising" as used herein specify the presence of stated features, integers, steps, operations, elements, or components, or combinations thereof, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or groups thereof, or combinations thereof.

[0079] The corresponding structure, material, acts, and equivalents of all means or step-plus-function elements in the following claims are intended to include structure, material, or acts for performing the function in combination with other claimed elements as specifically claimed. The description of the present invention has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the invention in the form disclosed. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the invention. The embodiments have been chosen and described to best explain the principles and practical applications of the invention and to enable those skilled in the art to understand the invention in various embodiments with various modifications suited to the particular uses intended. [Explanation of symbols]

[0080] 100 ways 200 Block Diagram 202 high-resolution images 204 Rectangle 206 images 208 images 210 images 212 tiles 214 tiles 216 tiles 300 Block Diagram 302 Object Recognition Baseline Algorithm 304 Output 306 Output 308 Output 310 Arrow 312 Arrow 314 Layer Results 316 Layer Results 318 Layer Results 320 Smart Result Aggregation Steps 400 Block Diagram 402 Final result 404 Result Set 406 Result Set 408 Result Set 410 result set 412 Intermediate Layer Data Set 414 Intermediate Layer Data Set 416 Intermediate Layer Data Set 418 Logical "OR" Operation 502 tiles 504 different tiles 506 Periphery / Border Area 508 Periphery / Border Area 510 rectangle 512 Rectangle 514 original images 516 detection results 600 Object Recognition System 602 memory 604 processor 606 Receiver 608 Disassembly Unit 610 Baseline Call Module 612 Smart Results Aggregation Unit 614 First Aggregation Module 616 Layer Comparison Module 618 Second Aggregation Module 620 System Internal Bus System 700 Computer Systems / Servers, Computing Systems 702 processor or processing unit 704 System Memory, Memory 706 Bus 708 Random Access Memory (RAM) 710 Cache Memory 712 Storage Systems 714 Input / Output (I / O) Interface 716 program modules 718 External Devices 720 display 722 Network Adapter

Claims

1. receiving, by one or more processors, a high resolution image; decomposing, by one or more processors, the received high-resolution image into a plurality of hierarchically organized layers of images, each of the plurality of layers including at least one image tile that is at least a portion of the received high-resolution image, each of the image tile having a corresponding resolution suitable for a baseline image recognition algorithm, the plurality of layers retaining the entirety of the received high-resolution image at different resolutions; applying, by one or more processors, the baseline image recognition algorithm to each of the image tiles in each of the plurality of layers; performing, by one or more processors, a result aggregation of results of application of the baseline image recognition algorithm to each of the image tiles in each of the plurality of layers; aggregating, by one or more processors, results of the baseline image recognition algorithm for each layer of the plurality of layers; performing, by one or more processors, layer comparisons of the set of results of the baseline image recognition algorithm for adjacent sets of layers arranged in order of resolution of the plurality of layers; performing, by one or more processors, a hierarchical aggregation of results of the baseline image recognition algorithms in response to layer comparisons in the set; performing the result aggregation, A method comprising:

2. performing the result aggregation of results of application of the baseline image recognition algorithm to each of the image tiles of each of the plurality of layers; extracting, by one or more processors, polygons encoding the shape of the recognized object; mapping, by one or more processors, local polygon coordinates used in the image tiles to global coordinates used in the corresponding image tiles having the highest resolution, thereby compressing polygonal shapes resulting from the baseline image recognition algorithm having higher resolutions so that the compressed shapes are equivalent to shapes in image tiles having lower resolutions; The method of claim 1 further comprising:

3. removing, by one or more processors, areas of overlap between adjacent image tile edges in each of the plurality of layers; merging, by one or more processors, detected partial objects in the adjacent image tiles into a single detected object; The method of claim 2 further comprising:

4. performing the layer comparison on the set, comparing, by one or more processors, compressed associated shapes of image tiles of said adjacent pairs of layers; constructing, by one or more processors, a shape intersection based on the comparison of the compressed related shapes, thereby constructing N intermediate image layers, where N is one less than the number of layers in the plurality of hierarchically organized layers; and The method of claim 3 further comprising:

5. performing the hierarchical aggregation, constructing, by one or more processors, a pixel-wise union of all N intermediate image layers, thereby constructing a final image of a resolution equivalent to the resolution of the received high-resolution image, the final image including a polygon enclosing the detected object; The method of claim 4 further comprising:

6. 6. The method of claim 1, wherein the results of the baseline image recognition algorithm include at least one selected from the group consisting of a class of recognized item, a bounding box surrounding an identified object in the image tile, and a mask represented by a polygon surrounding the shape of the recognized object in the image tile.

7. The method of any one of claims 1 to 6, wherein the baseline image recognition algorithm is a Mask R-CNN (Region-based Convolutional Neural Network) algorithm or a Fast R-CNN algorithm.

8. A method described in any one of claims 1 to 7, wherein the resolution suitable for the baseline image recognition algorithm is selected from the group consisting of 224 x 224 pixels, 512 x 512 pixels, 800 x 800 pixels, and 1024 x 800.

9. The method of any one of claims 1 to 8, wherein the baseline image recognition algorithm is pre-trained such that a neural network model is built for an inference task for object recognition.

10. The method according to any one of claims 2 to 5, wherein the object to be detected is a material defect.

11. A computer program comprising: receiving a high resolution image; decomposing the received high-resolution image into a plurality of hierarchically organized layers of images, each of the plurality of layers including at least one image tile that is at least a portion of the received high-resolution image, each of the image tile having a corresponding resolution suitable for a baseline image recognition algorithm, the plurality of layers retaining the entirety of the received high-resolution image at different resolutions; applying the baseline image recognition algorithm to each of the image tiles in each of the plurality of layers; performing a result aggregation of results of application of the baseline image recognition algorithm to each of the image tiles of each of the plurality of layers; aggregating results of the baseline image recognition algorithm for each layer of the plurality of layers; performing a layer comparison of the results of the baseline image recognition algorithm on adjacent pairs of layers arranged in order of resolution of the plurality of layers; performing a hierarchical aggregation of results of the baseline image recognition algorithm in response to layer comparisons in the set; performing the result aggregation, A computer program for executing the above.

12. performing the result aggregation of results of application of the baseline image recognition algorithm to each of the image tiles of each of the plurality of layers; Extracting polygons that encode the shape of the recognized object; mapping local polygon coordinates used in the image tiles to global coordinates used in the image tiles having a corresponding highest resolution, thereby compressing polygonal shapes resulting from the baseline image recognition algorithm having a higher resolution, such that the compressed shapes are equivalent to shapes in image tiles having a lower resolution; 12. The computer program of claim 11, further comprising:

13. 13. The computer program product of claim 11 or 12, wherein the baseline image recognition algorithm is a Mask R-CNN (Region-based Convolutional Neural Network) algorithm or a Fast R-CNN algorithm.

14. one or more computer processors; one or more computer-readable storage media; program instructions stored on the computer-readable storage medium for execution by at least one of the one or more processors; wherein the program instructions include: program instructions for receiving a high resolution image; program instructions for decomposing the received high-resolution image into a plurality of hierarchically organized layers of images, each of the plurality of layers including at least one image tile that is at least a portion of the received high-resolution image, each of the image tiles having a corresponding resolution suitable for a baseline image recognition algorithm, the plurality of layers holding the entire received high-resolution image at different resolutions; program instructions for applying the baseline image recognition algorithm to each of the image tiles in each of the plurality of layers; and program instructions for performing result aggregation of results of applying the baseline image recognition algorithm to each of the image tiles in each of the plurality of layers, the program instructions including: aggregating results of the baseline image recognition algorithm for each layer in the plurality of layers; performing pairwise layer comparisons of results of the baseline image recognition algorithm for adjacent pairs of layers in order of resolution of the plurality of layers; and performing hierarchical aggregation of results of the baseline image recognition algorithm in response to the pairwise layer comparisons.

1. A computer system comprising:

15. the program instructions for performing the result aggregation of results of application of the baseline image recognition algorithm to each of the image tiles of each of the plurality of layers, Extracting polygons that encode the shape of the recognized object; mapping local polygon coordinates used in the image tiles to global coordinates used in the corresponding image tiles having the highest resolution, thereby compressing polygonal shapes resulting from the baseline image recognition algorithm having higher resolutions so that the compressed shapes are equivalent to shapes in image tiles having lower resolutions; 15. The computer system of claim 14, further comprising program instructions for:

16. 16. The computer system of claim 14 or 15, wherein the baseline image recognition algorithm is a Mask R-CNN (Region-based Convolutional Neural Network) algorithm or a Fast R-CNN algorithm.

17. 16. The computer system of claim 15, wherein the object to be detected is a material defect.

Images