Occlusion anomaly detection method using image data augmentation and attention

Abstract

The method for detecting occlusion anomalies using image data augmentation and attention of the human motion recognition device of the present invention generates occlusion data considering direction, increases the amount of input data using various image data augmentation techniques, focuses on specific parts within the image through the attention mechanism of grad-CAM, and detects occlusion and anomaly objects through Mask R-CNN. It makes it possible to explain where the anomaly objects are located through a heatmap generated from the attention map output from grad-CAM.

Description

Technology Field

[0001] The present invention relates to computer vision technology, and more specifically, to a technology for detecting occlusion anomalies based on Mask R-CNN in manufacturing process image data. Background Technology

[0002] Object detection identifies the locations and classes of multiple objects by distinguishing instances through bounding boxes. Object detection is possible for image data through classification, localization, and detection information. Localization identifies the location of objects within an image and classifies objects by detecting them through detection. Object detection using deep learning is classified into 1-stage detectors and 2-stage detectors.

[0003] A representative example of a 1-stage detector is YOLO (You Only Look Once). It divides the input image into grid regions to identify stored anchors and predicts anchor boxes. An anchor box is a boundary that defines the likely location of an object within the input image. Based on this, a confidence score is obtained, and objects are classified into classes according to this confidence score. YOLO has the advantage of high speed because it performs classification and localization simultaneously. As an existing object detection technique using YOLO, Liang Tianjiao proposed a model based on YOLOv3 to detect welds on vehicle wheels. This improves the loss function of YOLOv3 by utilizing the original model structure and layer-based Distance-IoU (DIOU) loss. Additionally, by using DIOU-NMS Distance-IoU to remove duplicate candidate bounding boxes, the loss function further accelerates convergence speed and improves accuracy. However, it has the limitation of detecting only a single object during the vehicle wheel welding process.

[0004] Representative algorithms of 2-stage detectors include Fast R-CNN, Faster R-CNN, and Mask R-CNN. Traditional object detection uses rectangular bounding boxes to distinguish individual instances, confirm the presence of multiple objects, and locate their positions and classes. In contrast, Mask R-CNN utilizes instance segmentation, a concept that combines object detection and semantic segmentation. While traditional semantic segmentation divides objects into the same region without distinguishing between them if they belong to the same class, instance segmentation distinguishes different instances even within the same class.

[0005] Anomaly detection is a task that identifies and distinguishes the characteristics of normal and abnormal samples within a dataset. Depending on the underlying dataset and the configuration of the model used, this can be categorized into supervised learning and unsupervised learning methods. In the case of supervised learning, a dataset containing information specific to each image is utilized, and the model learns and infers features that may belong to the abnormal class. Improved model performance can be expected by leveraging data-appropriate metadata, such as labels and bounding boxes, along with large volumes of collected data. However, due to the nature of anomaly detection, where the probability of occurrence of normal and abnormal data differs, the class distribution of the collected data is imbalanced. This class imbalance induces biased learning in the model inferring the data, leading to a decline in classification performance. Furthermore, because the occurrence patterns of abnormal features vary, appropriate metadata or domain knowledge is required. Proper inference by supervised learning models requires well-described datasets, which presents a limitation as it necessitates significant time and the assistance of specialized personnel. To address these issues, various studies on anomaly detection using semi-supervised learning are currently underway. These methods include the application of pseudo-labels, one-class anomaly detection, and unsupervised learning. Insufficient annotation can be addressed through explicit modeling for anomalous data lacking labels, or by applying pseudo-labels based on data features learned by the model. However, this requires more time and analysis procedures than actual annotation, and the application of incorrect labels can negatively impact the model's inference process. The problem to be solved

[0006] The present invention aims to provide a detection method that reduces false positive and false negative rates in manufacturing process image data including occluded regions. means of solving the problem

[0007] A method for detecting occlusion abnormalities using image data augmentation and attention according to one aspect of the present invention comprises a preprocessing step, an input data generation step, an attention map generation step, a new input image generation step, and a detection step.

[0008] The preprocessing step generates occlusion data by adjusting the angle and direction from images of the unoccluded manufacturing process.

[0009] The input data generation step generates various input image data from occlusion data generated through image data augmentation techniques.

[0010] The attention map generation step uses an attention mechanism on the generated input image data to generate an attention map to be used for detecting occlusions and abnormal objects.

[0011] The new input image generation step generates a new input image using an attention map.

[0012] The detection step inputs a new input image into a Mask R-CNN model to detect occlusions or abnormal objects.

[0013] At this time, the image data enhancement techniques used in the input data generation step may include Shearing, Translate, Rotate, AutoContrast, Invert, Equalize, Solarize, Posterize, Contrast, Color, Blightless, Sharpness, Cutout, and Sample Pairing.

[0014] The detection step may include a step of acquiring feature pyramids from a new input image, a step of outputting region proposals for each feature pyramid, a step of selecting the optimal RoI among the region proposals and marking an anchor box, and a step of generating and resizing a mask for each instance for semantic segmentation.

[0015] At this time, the mask is generated by three mask generation branches consisting of a classification branch that performs class prediction, a box regression branch that performs box coordinate prediction, and a mask branch that generates an easy mask.

[0016] According to an additional aspect of the present invention, a method for detecting occlusion abnormalities using image data augmentation and attention may further include a heatmap display step.

[0017] The heatmap display step displays a heatmap that visualizes the locations of objects detected in the detection step. Effects of the invention

[0018] According to the present invention, the false positive and false negative rates can be reduced in object detection for manufacturing process image data including occluded regions. Brief explanation of the drawing

[0019] FIG. 1 is a flowchart of a method for detecting occlusion abnormalities using image data augmentation and attention according to one aspect of the present invention. Figure 2 conceptually illustrates a Mask R-CNN-based occlusion anomaly detection model with image data augmentation and attention applied. Figure 3 illustrates an example of a preprocessing process for adjusting the orientation of input data for detecting blockage abnormalities. Figure 4 is an example of an image data augmentation technique used by the present invention. Figure 5 conceptually illustrates the grad-CAM and Mask R-CNN models for attention map visualization. Figure 6 conceptually illustrates a Mask R-CNN-based occlusion anomaly detection model. Specific details for implementing the invention

[0020] The foregoing and additional aspects are embodied in the embodiments described with reference to the attached drawings. It is understood that the components of each embodiment may be combined in various ways within the embodiment unless otherwise stated or contradictory. Each block in the block diagram may represent a physical part in some cases, but in others, it may be a logical representation of a part of the function of a single physical part or a function spanning multiple physical parts. Sometimes, the entity of a block or part thereof may be a set of program instructions. These blocks may be implemented in whole or in part by hardware, software, or a combination thereof.

[0021] FIG. 1 is a flowchart of a method for detecting occlusion abnormalities using image data augmentation and attention according to one aspect of the present invention, and FIG. 2 conceptually illustrates a Mask R-CNN-based occlusion abnormality detection model with image data augmentation and attention applied.

[0022] A method for detecting occlusion abnormalities using image data augmentation and attention according to one aspect of the present invention is a method performed in a computing device comprising one or more processors and a memory connected to the processors and storing program instructions executable by the processors.

[0023] A computing device that performs an obstruction anomaly detection method using image data augmentation and attention is defined as an obstruction anomaly detection device, which includes a processor and, in addition to memory, a storage device, a display, an input device, etc. The processor is a processor that executes program instructions implementing the obstruction anomaly detection method using image data augmentation and attention, and memory is connected to the processor and stores program instructions executable by the processor, data to be used by the processor for computation, and data processed by the processor.

[0024] The method for detecting occlusion anomalies using image data augmentation and attention according to the present invention aims to resolve issues such as labor costs incurred when humans manually detect unnecessary or incorrect items produced during manufacturing processes, or the inability to detect items when the item image is occluded. The present invention performs direction-aware occlusion anomaly detection based on Mask R-CNN on manufacturing process data. The present invention improves the accuracy of object recognition compared to existing Mask R-CNN-based algorithms. Data preprocessing is performed using OpenCV's rotation function to represent images of items being manufactured from various angles visible during the manufacturing process, followed by the generation of various input data using image data augmentation techniques. Subsequently, images output using grad-CAM are input into Mask R-CNN to determine whether pixels are objects, and occlusions and anomalies within the image are detected by extracting bounding boxes and masking information. A Mask R-CNN capable of region detection is applied and combined with occlusion region detection during image preprocessing. In addition, the final attention map output through grad-CAM is output as a heatmap, making it possible to explain where abnormal objects are located. This provides the advantage of being reliable and allowing for the interpretation of locations identified as defects. Therefore, according to the present invention, the defect detection capability is improved to reduce false positives and false negatives in the manufacturing process.

[0025] The obstruction detection model of the present invention is described with reference to FIG. 2. The Mask R-CNN-based obstruction anomaly detection model, which applies image data augmentation and attention, consists of a two-stage process. The image used in the example of FIG. 2 is an image of the cutlery generation process.

[0026] The first step of the process is a data preprocessing process that generates input images. For an image showing forks mixed in during the cutlery production process, an image containing obstructed items is generated using OpenCV's rotation function, and various input images are created by applying diverse image data augmentation techniques to this generated image.

[0027] The process of the second step involves inputting the input image generated in the first step into grad-CAM, generating a new input image using the attention map obtained through the attention mechanism, and then inputting it into Mask-CNN to detect forks that are obstructed by the spoon.

[0028] A method for detecting occlusion abnormalities using image data augmentation and attention according to one aspect of the present invention comprises a preprocessing step, an input data generation step, an attention map generation step, a new input image generation step, and a detection step.

[0029] In the preprocessing step, the blockage anomaly detection device generates blockage data (S1000) by adjusting the angle and direction from an image of a non-blocked manufacturing process. Since the probability of occurrence of training data capable of learning blockage anomalies is almost zero, there is a significant shortage of data to train. Therefore, in the preprocessing step, the raw input image data of the manufacturing process is preprocessed to generate blocked product image data.

[0030] Figure 3 illustrates an example of the preprocessing process for orientation adjustment of input data for detecting blockage anomalies. Figure 3 shows the preprocessing process of input data for detecting blockage anomalies. The manufacturing process data utilizes data from the spoon and fork classes, which are part of the COCO dataset. In tableware manufacturing processes, spoons and forks are often not evenly aligned but are blocked and emerge in various directions. Additionally, there is a problem where the amount of blocked data is generally insufficient compared to image datasets that are not blocked. To solve this problem, the present invention performs image preprocessing using the OpenCV library. Among various image data augmentation methods using OpenCV, the present invention utilizes an image data augmentation method that preprocesses in the order of Orientation, Compositing, and Paste. First, the present invention performs Orientation on the existing image dataset. At this time, the functions that can be utilized for Orientation are transition and random functions.

[0031] In the case of the transition function, angles cannot be considered because it moves the image up, down, left, and right. However, since the rotation function considers the rotation axis and rotation angle of the image, angles and directions can be considered. Accordingly, by generating new images rotated at 15° intervals from spoon and fork data, it is possible to represent the manufacturing process from various angles visible in the manufacturing process and to generate occluded images.

[0032] In the input data generation step, the obstruction anomaly detection device generates various input image data from obstruction data generated through an image data augmentation technique (S1020). In the input data generation step, the obstruction anomaly detection device generates various input image data by applying various image data augmentation techniques to image data including the obstruction image generated in the preprocessing step.

[0033] FIG. 4 is an example of an image data augmentation technique used by the present invention. The image data augmentation technique used in the input data generation step may include Shearing, Translate, Rotate, AutoContrast, Invert, Equalize, Solarize, Posterize, Contrast, Color, Blightless, Sharpness, Cutout, and Sample Pairing.

[0034] Figure 4 shows examples of input images generated using various image data augmentation techniques. Image data augmentation techniques are methods of augmenting data to improve performance by increasing the amount of limited image data available. In the present invention, a total of 16 image data augmentation techniques are used to increase the amount of input data. Shear specifies a value 'a' for the degree of shifting and new x and y coordinates according to each x and y coordinate. Translate is a method of specifying new coordinates based on the degree of translation, and Rotate is a method of specifying new coordinates using rotation. AutoContrast is a method of flattening the contrast histogram, Invert is an image inversion, and Equalize is an image smoothing method. Solarize inverts pixels exceeding a threshold, Posterize represents the image with a lower bit rate, Contrast is contrast, and Color is a method of balancing colors. Bitless adjusts brightness, Sharpness increases clarity, and Cutout is a method of erasing a part of the image and setting that part to 0. Sample Pairing is a method of averaging two unknowns at the pixel level and then using the result for training. Through this image data augmentation technique, the present invention can augment the amount of insufficient input data to construct a higher-performance model.

[0035] The attention map generation step generates an attention map to be used for occlusion and abnormal object detection using an attention mechanism on the generated input image data (S1040).

[0036] The new input image generation step generates a new input image using an attention map (S1060).

[0037] Figure 5 conceptually illustrates the grad-CAM and Mask R-CNN models for attention map visualization. The grad-CAM (gradient-Class Activation Map) illustrated in Figure 5 is one of the explainable AI techniques, which explains the parts visually attended by the model by visualizing the locations of pixels that influenced the model's results. The grad-CAM is broadly divided into an attention mechanism and a classification mechanism.

[0038] The attention mechanism converts the last convolutional layer into a fully connected layer, assigns importance values ​​using the gradient information generated through it, and then inputs a new input image created using the original image and the attention map into the classification mechanism to predict the class. To visualize the locations in the images output by the convolutional model, a feature map is created for each feature of the last convolutional layer, and a heatmap is generated by calculating the pixel-by-pixel sum for the same location. This allows for explaining which areas received more focus for detecting anomalous objects and where those objects are located.

[0039] The detection step inputs a new input image into a Mask R-CNN model to detect occlusion or abnormal objects (S1080).

[0040] The detection step may include a step of acquiring feature pyramids from a new input image, a step of outputting region proposals for each feature pyramid, a step of selecting the optimal RoI among the region proposals and marking an anchor box, and a step of generating and resizing a mask for each instance for semantic segmentation.

[0041] At this time, the mask is generated by three mask generation branches consisting of a classification branch that performs class prediction, a box regression branch that performs box coordinate prediction, and a mask branch that generates an easy mask.

[0042] Figure 6 conceptually illustrates a Mask R-CNN-based occlusion anomaly detection model. The Mask R-CNN model utilizes a model implemented based on the Feature Pyramid Network (FPN) and the ResNet101 backbone. Mask R-CNN generates bounding boxes and segmentation masks for each instance of an object in an image. Unlike the previous Fast R-CNN and Faster R-CNN, which performed object detection, the Segmentation Mask occlusion anomaly detection process is essentially a combination of object detection and semantic segmentation. First, images generated through preprocessing are input into the ResNet-FPN backbone network to obtain feature pyramids. A feature pyramid refers to a network that inputs images of arbitrary size into a convolutional network to output feature maps of various scales. Each of these obtained feature pyramids is input into the Region Proposal Network (RPN) to output region proposals with objectness scores and bbox regressors. Among the obtained region proposals, the optimal ROI is selected, and anchor boxes are marked at locations in the image where objects are likely to be located. In the subsequent step, the most optimal anchor box finally detected is displayed. Next, the model generates and resizes a mask for each instance for semantic segmentation. The mask generation branch of Mask R-CNN consists of three branches, which distinguishes it from the methods used in conventional object detection. A binary mask is generated through the Classification branch, which performs the role of class prediction; the Bbox regression branch, which performs the role of modifying box coordinates; and the final FCN-based mask branch.After rescaling the feature map generated through three branches, a mask segment is generated that can be compared with the actual mask. When the boxes and masks obtained from the two processes are placed at the correct positions on the input image, occlusions and abnormal regions in the image are detected.

[0043] According to an additional aspect of the present invention, a method for detecting occlusion abnormalities using image data augmentation and attention may further include a heatmap display step.

[0044] The heatmap display step displays a heatmap that visualizes the location of the object detected in the detection step. That is, the attention map output through grad-CAM is implemented as a heatmap to visualize where the abnormal object occurred, making it possible to explain the location.

[0045] The method of the present invention is a method for detecting anomalies of occluded objects during a manufacturing process. It involves increasing the amount of input data using an orientation adjustment method and image data augmentation techniques utilizing OpenCV, and then detecting anomalies hidden within normal objects using Mask R-CNN on the augmented data, which is preprocessed by considering orientation rather than alignment using Grad-CAM, which focuses on areas in the image where objects can be detected effectively. In the first step, the aligned data is converted into data that considers orientation; this is achieved by rotating images from the existing dataset using the rotation function of OpenCV to form new images that consider orientation. Additionally, the amount of input data is increased using various image data augmentation techniques. In the second step, an attention map generated by utilizing the attention mechanism of Grad-CAM is output, and occlusion anomalies are detected using Mask R-CNN based on this map. This involves detecting occlusions and anomalies using boxes and masks obtained through Mask R-CNN that contain the occluded anomalies. Subsequently, the attention map generated by Grad-CAM is visualized as a heatmap to identify the location of the occluded and anomaly objects. The method of the present invention significantly improves the accuracy of detecting abnormal objects in manufacturing processes by reliably detecting occluded objects compared to the existing object detection model, the YOLO detection model. Additionally, the performance of the model can be enhanced by increasing the amount of input data using image data augmentation techniques. Furthermore, adding a process for detecting occlusion and abnormalities using the attention mechanism of grad-CAM is more effective for detecting occlusion and abnormalities because it allows focusing on key areas of the image for object detection. Finally, the attention map output through grad-CAM is implemented as a heatmap to visualize the location where abnormal objects occurred, thereby explaining the situation.Accordingly, the method of the present invention corrects the problem of occlusion data, which degrades the performance of an object detection model during the manufacturing process, into directional data and augments the input data using various image data augmentation techniques. Subsequently, by using an attention mechanism and a Mask R-CNN mechanism to detect abnormal objects located between occlusion objects, it is possible to detect abnormal objects in the occlusion data during the manufacturing process, and the location where the abnormal object occurred can be explained through a heatmap output by visualizing the attention map output through grad-CAM.

[0046] Although the present invention has been described above with reference to embodiments with reference to the accompanying drawings, it is not limited thereto and should be interpreted to encompass various modifications that can be obviously derived from them by those skilled in the art. The claims are intended to encompass such modifications.

Claims

Claim 1 A method for detecting occlusion anomalies using image data augmentation and attention, comprising: a preprocessing step of generating occlusion data that reproduces an occlusion situation in which an abnormal item is occluded between normal items in a manufacturing process in which items included in an image of an unoccluded manufacturing process are rotated and arranged in various angles and directions; an input data generation step of generating various input image data by applying an image data augmentation technique to the occlusion data generated in the preprocessing step; an attention map generation step of generating an attention map that identifies an area to focus on for occlusion and abnormal object discrimination using an attention mechanism on the generated input image data; a new input image generation step of generating a new input image using the attention map; and a detection step of inputting the new input image into a Mask R-CNN model to detect occlusion or abnormal objects. Claim 2 A method for detecting occlusion anomalies using image data augmentation and attention, further comprising the step of displaying a heatmap that visualizes the location of an object detected in the detection step in claim 1. Claim 3 A method for detecting occlusion anomalies using image data enhancement and attention, wherein the image data enhancement technique used in the input data generation step includes Shearing, Translate, Rotate, AutoContrast, Invert, Equalize, Solarize, Posterize, Contrast, Color, Blightless, Sharpness, Cutout, and Sample Pairing. Claim 4 A method for detecting occlusion anomalies using image data augmentation and attention according to claim 1, wherein the detection step comprises: a step of acquiring a feature pyramid from a new input image; a step of outputting a Region Proposal for each feature pyramid; a step of selecting an optimal RoI among the region proposals and displaying an anchor box; and a step of generating and resizing a mask for each instance for semantic segmentation. Claim 5 In claim 4, the mask is generated by three mask generation branches consisting of a classification branch that performs class prediction, a box regression branch that performs box coordinate prediction, and a mask branch that generates an easy mask, in a method for detecting occlusion anomalies using image data augmentation and attention.

Images