Data augmentation device and method for background bias removing in case of weakly supervised sementic segmentation

Abstract

The present invention relates to a data augmentation device and method for removing background bias during weak map semantic segmentation. The present invention comprises the steps of: inputting a plurality of images into an encoder to extract features for each image; inputting the extracted features of each image into a pre-trained first aggregation unit and a second aggregation unit to separate features of an object part and a background part in each image; inputting the features of the object part and the features of the background part of each image into a shuffle unit and shuffling the features of the background part in the shuffle unit; synthesizing the shuffled features of the background part and the features of the object part in a synthesis unit to generate a synthesized feature; and generating a data augmented image based on the synthesized feature.

Description

Technology Field

[0001] The present invention relates to a data augmentation device and method for removing background bias during weak map semantic segmentation. Background Technology

[0003] Semantic segmentation is the process of classifying an image pixel by pixel. In other words, it classifies which class each pixel belongs to. For reference, the aforementioned semantic segmentation is also referred to as semantic segmentation.

[0004] Weakly Supervised Semantic Segmentation (WSSS), which uses weak forms of labeling, is being actively researched to mitigate the annotation costs associated with acquiring pixel-level labels.

[0005] In the case of the Weakly Mapped Semantic Segmentation (WSSS) described above, only image-level labels are used for training, and the class of each pixel in the image is predicted.

[0006] However, there was a problem in that classifiers trained on biased datasets tended to utilize shortcut functions and make predictions based on incorrect correlations between specific backgrounds and objects, resulting in degraded generalization performance.

[0007] In addition, conventional technology does not consider the context or background in which an object appears, so deep learning models trained with augmented data are affected by background bias, where a specific object and background frequently appear together, and thus the accuracy of pixel-level labels inferred by mixing the object and background is limited.

[0008] Furthermore, there was a problem in that the number of augmented data was limited due to the limited diversity of augmented data, specifically when extracting object masks from images, as the number and size of the extracted masks were restricted.

[0009] Therefore, the development of Shortcut Mitigating Augmentation (SMA) training classifiers for Weakly Supervised Semantic Segmentation (WSSS), which generate synthetic representations of object-background combinations not visible in the training data, has been significantly highlighted.

[0010] In addition, there was a problem in that deep learning models trained on augmented data without considering the context or background in which objects appear were affected by background bias, where specific objects and backgrounds frequently appear together, and thus the accuracy at the pixel level of inference where objects and backgrounds are mixed was limited.

[0011] In addition, in the case of conventional techniques for extracting object masks from images, there was also a problem in that the number of augmented data was limited as the number and size of the extracted masks were limited.

[0012] Referring to Figure 1, a localization map generated by applying conventional augmentation is shown. Classifiers trained in the conventional method often roughly skip over and utilize the "sky" region (especially the flight path) that appears with the object "airplane," resulting in the generation of inaccurate Class Activation Maps (CAMs) and incorrect Pseudo-masks. Prior art literature

[0014] Republic of Korea Patent Publication No. 2022-0115757 Republic of Korea Patent Publication No. 2023-0035297 The problem to be solved

[0015] Accordingly, the present invention was devised to solve these problems and aims to provide a high level of pixel-level label acquisition that is not biased toward the background by relying solely on category information about objects in an image or video.

[0016] In addition, the purpose is to provide a data augmentation device and method for acquiring high-quality images by removing background bias during weak map semantic segmentation. means of solving the problem

[0018] To achieve this objective, the present invention provides a data augmentation method for removing background bias during weak map semantic segmentation, comprising: a step of inputting a plurality of images into an encoder to extract features for each image; a step of inputting the extracted features of each image into a pre-trained first aggregation unit and a second aggregation unit to separate features of an object part and a background part in each image; a step of inputting the features of the object part and the features of the background part of each image into a shuffle unit and shuffling the features of the background part in the shuffle unit; a step of synthesizing the shuffled features of the background part and the features of the object part in a synthesis unit to generate a synthesized feature; and a step of generating a data augmented image based on the synthesized feature.

[0019] The above data augmentation method further comprises a step of learning the first aggregation unit and the second aggregation unit, wherein the learning step comprises a step of inputting an image into an encoder to extract features of the image; a step of inputting the features of the image into the first aggregation unit to aggregate features of the object portion from the features of the image; a step of inputting the features of the image into the second aggregation unit to aggregate features of the background portion from the features of the image; and a step of performing contrast learning to reduce the similarity between the aggregated features of the object portion and the aggregated features of the background portion.

[0020] The above data augmentation method is characterized by further comprising: a step of measuring an activation value for object inference for each pixel in the data-augmented image; and a step of calculating the degree of background bias of the data-augmented image based on the measured activation value.

[0021] The step of calculating the degree of background bias is characterized by including the step of measuring the contribution of the object part and the contribution of the background part in the data-enhanced image, respectively, and the step of calculating the degree of background bias based on the ratio of the contribution of the object part and the contribution of the background part.

[0022] The step of measuring the contribution of the object portion and the contribution of the background portion includes the step of measuring the integrated gradient of each pixel of the data-enhanced image, wherein the integrated gradient of the pixel is calculated by the following mathematical formula.

[0023] (Mathematical formula)

[0024]

[0025] The object image received in the shuffle section can be shuffled, and the shuffled object image can be combined with the background image in the composite section.

[0026] A data augmentation device for removing background bias during semantic segmentation of a map comprises: an encoder that receives a plurality of images and extracts features for each of the images; a first aggregation unit and a second aggregation unit that receive features of the images extracted by the encoder and separate an object part and a background part from each of the images; a shuffle unit that receives features of the object part and features of the background part of each image from the first aggregation unit and the second aggregation unit and shuffles the features of the background part; a synthesis unit that synthesizes the features of the background part shuffled by the shuffle unit and the features of the object part to generate a synthesized feature; and a data augmented image based on the synthesized feature of the synthesis unit.

[0027] The method for augmenting the data described above further comprises learning the first aggregation unit and the second aggregation unit, wherein the learning is characterized by inputting an image into an encoder to extract features of the image, inputting the features of the image into the first aggregation unit to aggregate features of the object portion from the features of the image, inputting the features of the image into the second aggregation unit to aggregate features of the background portion from the features of the image, and performing contrast learning to reduce the similarity between the aggregated features of the object portion and the aggregated features of the background portion.

[0028] The augmentation of the above data is characterized by measuring an activation value for object inference for each pixel in the augmented image of the above data, and calculating the degree of background bias of the augmented image of the above data based on the measured activation value.

[0029] The calculation of the degree of background bias is characterized by measuring the contribution of the object portion and the contribution of the background portion in the data-enhanced image, respectively, and calculating the degree of background bias based on the ratio of the contribution of the object portion and the contribution of the background portion.

[0030] The measurement of the contribution of the object portion and the contribution of the background portion is characterized by measuring the integrated gradient of each pixel of the data-enhanced image, wherein the integrated gradient of the pixel is calculated by the following mathematical formula.

[0031] [Mathematical Formula]

[0032]

[0033] (Here, I(xi): integrated gradient of the i-th pixel of the data-augmented image, xi: data-augmented image, xbase: pre-set black image, m: number of gradients, k: k-th gradient)

[0034] The object image received in the shuffle section can be shuffled, and the shuffled object image can be combined with the background image in the composite section. Effects of the invention

[0035] Therefore, the present invention reduces the influence of background bias in categorical classification models by performing data augmentation based on the separation of object and background features during weak map semantic segmentation, and has the effect of quantitatively measuring the degree of background bias through evaluation indicators. Brief explanation of the drawing

[0037] Figure 1 is an example drawing showing a background biased based only on categorical information about an object in an image. FIG. 2 is a drawing illustrating the process of separating features of an object and a background in one embodiment of the present invention. FIG. 3 is a diagram illustrating a data augmentation process that randomly combines an object and a background in one embodiment of the present invention. FIG. 4 is a configuration diagram of a data augmentation device for removing background bias during weak map semantic segmentation according to an embodiment of the present invention. Figure 5 is a flowchart of a data augmentation method for removing background bias during weak map semantic segmentation. Figure 6 is a photograph comparing pixel-unit labels generated through a weak map semantic segmentation method. Figure 7 is a photograph comparing the visualization of a category activation map by the data augmentation method according to an embodiment of the present invention. FIG. 8 is a block diagram illustrating a computing environment including a computing device suitable for use in an exemplary embodiment. Specific details for implementing the invention

[0038] Hereinafter, specific embodiments of the present invention will be described with reference to the drawings. The following detailed description is provided to facilitate a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, this is merely illustrative and the present invention is not limited thereto.

[0039] In describing the embodiments of the present invention, detailed descriptions of known technologies related to the present invention are omitted if it is determined that such detailed descriptions may unnecessarily obscure the essence of the present invention. Furthermore, the terms described below are defined in consideration of their functions within the present invention, and these may vary depending on the intentions or practices of the user or operator. Therefore, such definitions should be based on the content throughout this specification. Terms used in the detailed description are intended merely to describe the embodiments of the present invention and should not be limiting in any way. Unless explicitly stated otherwise, expressions in the singular form include the meaning of the plural form. In this description, expressions such as "include" or "comprise" are intended to refer to certain characteristics, numbers, steps, actions, elements, parts thereof, or combinations thereof, and should not be interpreted to exclude the existence or possibility of one or more other characteristics, numbers, steps, actions, elements, parts thereof, or combinations thereof other than those described.

[0041] Before describing the present invention, semantic segmentation is a process of dividing a digital image into multiple sets of pixels. Through this segmentation, the representation of the image is simplified and transformed into a form that is easy to interpret. Semantic segmentation is widely utilized in the field of computer vision, along with object detection.

[0042] FIG. 2 is a diagram showing the process of separating features of an object and a background in one embodiment of the present invention, FIG. 3 is a diagram showing the process of randomly combining an object and a background in one embodiment of the present invention, FIG. 4 is a configuration diagram of a data augmentation device for removing background bias during weak map semantic segmentation according to the present invention, FIG. 5 is a flowchart of a data augmentation method for removing background bias during weak map semantic segmentation, FIG. 6 is a photograph comparing pixel-unit labels generated through the weak map semantic segmentation method, FIG. 7 is a photograph comparing category activation map visualizations by the data augmentation method according to an embodiment of the present invention, and FIG. 8 is a block diagram for illustrating and explaining a computing environment including a computing device suitable for use in an exemplary embodiment.

[0043] First, referring to Fig. 2, we will explain the process of learning to separate the features of the object part and the background part in the input image.

[0044] As illustrated in FIG. 2, F is an encoder, and two aggregation units are formed, Mo is a first aggregation unit and Mb is a second aggregation unit, wherein the first aggregation unit (Mo) and the second aggregation unit (Mb) use the feature obtained from the encoder (F) as an object feature z o and background feature z b Each is separated as follows. That is, when an image is input to the encoder (F), the background (z) of the image is separated through the first aggregation unit (Mo) and the second aggregation unit (Mb). b ) and object(z o ) are separated individually.

[0045] Next, the image separated into background and object undergoes a differentiation process through contrastive learning. The aforementioned contrastive learning refers to the background image (z b ) and object image(z o The purpose is to make them move further apart so that they do not become similar to each other. As a result, COS(z o , z bCalculates the similarity. In other words, it separates the background and the object so that they do not become similar through contrast learning, thereby reducing the COS (cosine) similarity between the two features.

[0046] Next, the two classification scores f(z) obtained from the classifier (f) o ), f(z b ) are supervised by label y and label 0, respectively (for convenience, the background part is set to the correct answer value of 0). It is verified whether the background and the object have been properly separated by comparing the y value and the 0 value obtained through the above classifier (f). Object-related feature z o and background feature z b To further distinguish them, contrastive loss is additionally used.

[0047] As such, synthesizing distinct representations of different samples is very important because the optimal classifier (f) must consistently identify target objects regardless of background. Here, z o (Object Image) and z b (Background images) are separated exclusively.

[0048] The above object image z o While is highly related to class label prediction, the above background image z b It is correlated with the object, but there is no need to predict the class label. That is, if the image is set as x, the background representation z b Another expression z b* It is assumed that the prediction will not be affected even if it is replaced by.

[0049] Therefore, the optimal classifier f* must provide consistent predictions without being affected by bias. The hypothesis can be expressed as [Equation 1] below.

[0050]

[0051] [. , .] represent channel-specific connections. In other words, shuffling individual representations is used to achieve these invariant predictions. First, the entangled background representations are randomly permuted. Obtains. Then, ul z o Create a new expression z by connecting with.

[0052] z sb represents a fixed object-related representation combined with a background representation swapped from another image in a mini-batch (a small data sample randomly selected from the entire dataset).

[0053] In addition, to provide more diverse representations to the classifier (f), augmentation is also performed in the opposite direction. That is, is randomly shuffled zo, then z b Connect with z so = [ , z b ] obtain. Then, z sb classifier(f s Supply to ) and supervise classification with target label y. z so In this case, the target object is shuffled in the mini-batch, and therefore, the target label y also depends on the permutation index It is rearranged into.

[0054]

[0055] The objective function for training an augmented classifier using shuffle representation can be expressed as [Equation 2] above.

[0056]

[0057] Therefore, the total loss function can be explained by the above [Equation 3], where λ represents the equilibrium scalar.

[0059] FIG. 3 is a diagram illustrating a data augmentation process that randomly combines an object and a background in one embodiment of the present invention.

[0060] Through the sequence of FIG. 2 above, neural network-trained images are input (Input x), and the encoder (F) is divided into image objects in the first aggregation unit Mo and image backgrounds in the second aggregation unit Mb. Referring to the figure, the feature map z is an object-related representation z by the aggregation units Mo and Mb. o and background expression z b It is disentangled.

[0061] Background expressions are permuted in the mini-batch Perform background shuffling to obtain . Then, z o and Connecting to create a new expression z sb Composes. Here, fs(z sb ) represents the prediction by fs, and represents the per-channel connection operation.

[0062] The classifier (fs) combines the features of the background image and the features of a random object image to perform bidirectional data augmentation.

[0063] As a result, the classifier (fs) learns representations that rarely appear in the biased dataset, thereby enabling it to represent improved image backgrounds and objects that rely less on shortcut functions. The aforementioned shortcut function means to omit or skip over intermediate parts.

[0064] For example, when using images of an "airplane" with a "sky" and "cows and sheep" with a "grass landscape" as input images, the classifier (fs) can learn representations related to "airplanes with grass landscapes" and "cows and sheep appearing in the sky" in the representation space. However, since the background and objects may differ, it fails to recognize this and "skips" them, resulting in images of such scenes being excluded from the training dataset. Additionally, since the representations of the mini-batch are randomly combined in every iteration, the diversity of representations is guaranteed.

[0066] Hereinafter, a description of a data augmentation device (D) according to an embodiment of the present invention will be provided.

[0067] Referring to FIG. 4, the data augmentation device (D) is largely composed of an encoder (100) that receives a plurality of images and extracts features for each of the images, an aggregation unit (200) composed of a first aggregation unit (Mo) and a second aggregation unit (Mb) that receive features of the images extracted by the encoder and separate the object part and the background part from each of the images, a shuffle unit (300) that receives features of the object part and the background part of each image from the first aggregation unit (Mo) and the second aggregation unit (Mb) and shuffles the features of the background part, and a synthesis unit (400) that synthesizes the features of the background part shuffled by the shuffle unit (300) and the features of the object part to generate a composite feature. For reference, the detailed description of the encoder (100) is a conventional encoder with widely known functions and configurations, and since such a detailed description is far removed from the purpose of the present invention, it will be omitted.

[0068] As illustrated in FIG. 4 above, a video or image (x) is received by the aggregation unit (200) through a network, and the aggregation unit (200) is composed of a pair of a first aggregation unit (Mo) and a second aggregation unit (Mb), wherein the first aggregation unit (Mo) uses an object and the second aggregation unit uses Mb (background) to aggregate object-related and background features.

[0069] Here, the aggregation unit (200) is at least one pair and is learned through different labels, and uses a contrastive learning method to separate the features of the object image and the background image.

[0070] If, N images Given, the feature map is z i ∈R c×h×w It is obtained through the network. Here, C, h, and w represent the channel size, height, and width of the feature map, respectively.

[0071] The shuffle unit (300) is characterized by mixing and synthesizing the background image in a mini-batch, where a mini-batch refers to a small data sample randomly selected from the entire dataset. That is, the shuffle unit (300) randomly mixes entangled expressions and combines them to generate a synthetic expression.

[0072] Referring again to FIG. 3, it is shown that an object image received in the shuffle unit (300) and a random background image can be randomly combined in a 1:1 ratio, but conversely, it is possible to randomly combine the random object image received in the shuffle unit (300). In this way, data augmentation is performed by combining the background features and object features of the image.

[0073] The synthesis unit (400) serves to synthesize the object image shuffled from the shuffle unit (300) and the shuffled background image, respectively. The synthesis unit (400) synthesizes the object image and the randomly shuffled background image, respectively, so that they match each other. Conversely, the shuffled object image and the randomly mixed background image can be combined in the synthesis unit (400) to regenerate a pair. In this way, bidirectional data augmentation is possible.

[0074] An augmented image of the data is generated based on the synthetic characteristics of the above-mentioned synthesis unit (400).

[0075] The augmentation of the above data further includes the step of learning the first aggregation unit (Mo) and the second aggregation unit (Mb). The learning method inputs an image into an encoder (100) to extract features of the image, inputs the features of the image into the first aggregation unit to aggregate the features of the object part from the features of the image, inputs the features of the image into the second aggregation unit (Mb) to aggregate the features of the background part from the features of the image, and performs contrast learning so that the similarity between the aggregated features of the object part and the aggregated features of the background part is reduced.

[0076] The augmentation of the above data measures activation values ​​for object inference for each pixel in the augmented image of the above data, and calculates the degree of background bias of the augmented image of the above data based on the measured activation values.

[0077] The calculation of the above degree of background bias is performed by measuring the contribution of the object portion and the contribution of the background portion in the data-enhanced image, respectively, and calculating the degree of background bias based on the ratio of the contribution of the object portion and the contribution of the background portion.

[0078] The contribution of the object part and the contribution of the background part are measured by measuring the integrated gradient of each pixel of the data-enhanced image, and the integrated gradient of the pixel is calculated by the following [Equation 4].

[0079]

[0080] (Here, I(x i ): Integrated gradient of the i-th pixel of the data-augmented image, x i : Data-enhanced image, x base : default black image, m: number of gradients, k: k-th gradient)

[0081] That is, when augmenting data, the calculation of the integrated gradient to quantitatively measure background bias in an image combining a background image and an object image is generated through the above [Equation 4].

[0082] The SUR (activation ratio value) of the image is the background area (R b )'s I G and object area (R o )'s I G It is defined as a ratio. The above SUR (activation ratio value) indicates how much the information of the object area is utilized compared to the background area. The above SUR can be expressed mathematically as [Equation 5] below.

[0083]

[0084] Furthermore, to directly evaluate the degree of Skip A Relaxation (SMA), the Background A Ratio (BAR) was calculated between the attributes of the background region and the sum of all attributes when predicting the target class.

[0085] The above shorthand is an unintended decision rule during prediction, and the background indicates this. Therefore, the extent to which shorthand is used when predicting class labels can be measured by 1) object-related attributes and 2) background attributes, which can be measured as SUR and BAR, respectively.

[0087] Hereinafter, a data augmentation method for removing background bias during weak map semantic segmentation according to an embodiment of the present invention will be described with reference to FIG. 5. Descriptions that overlap with the data augmentation device (D) described in the above embodiment will be omitted to some extent.

[0088] First, multiple images are input into an encoder (100) to extract features from each image (S100). The learned images are input through the input section (not shown) of the encoder (100) to extract features from the images. The images are general images, and images that include backgrounds and objects.

[0089] Next, the features of each image extracted in the above S100 are input into the pre-learned first aggregation unit (Mo) and second aggregation unit (Mb) to separate the features of the object part and the background part in each image (S200).

[0090] The first aggregator (Mo) and the second aggregator (Mb: aggregators) above are composed of a convolution and an attention layer (not shown) to distinguish between the image input to the encoder (100) as object and background features, and at least one pair of two is preferred so as to be able to separate the background and the object.

[0091] Herein, the method further includes a step of learning the first aggregation unit (Mo) and the second aggregation unit (Mb), wherein the learning step includes a step of inputting an image into an encoder (100) to extract features of the image, a step of inputting the features of the image into the first aggregation unit (Mo) to aggregate features of the object part from the features of the image, a step of inputting the features of the image into the second aggregation unit (Mb) to aggregate features of the background part from the features of the image, and a step of performing contrast learning so as to reduce the similarity between the aggregated features of the object part and the aggregated features of the background part.

[0092] Next, the characteristics of the object part and the background part of each image are input into the shuffle unit (300), and the characteristics of the background part are shuffled in the shuffle unit (300) (S300).

[0093] The shuffle unit (300) learns using a mini-batch method in which the entire dataset is divided into smaller units and trained on those units. The learned data is shuffled to prevent bias in the model. Therefore, for this function, it would be desirable for the shuffle unit (300) to be separately equipped with a data-loader (not shown). In this way, data augmentation is performed by combining the background features and object features of the image.

[0094] Here, it should be noted that, conversely, the object images received in the shuffle unit (300) can be shuffled, and the shuffled object images can be combined with the background image in a 1:1 manner in the composite unit (400).

[0095] The features of the shuffled background part and the features of the object part are combined in the synthesis part (400) to generate a composite feature (S400).

[0096] The above synthesis unit (400) serves to regenerate a pair by combining the background image shuffled in the previous stage shuffle unit (300) and the randomly mixed object image, respectively. That is, to enhance the diversity of data that can be generated, background features and random image object features are combined. Conversely, the shuffled object image and the randomly mixed background image can be combined in the above synthesis unit (400) to regenerate a pair. In this form, bidirectional data augmentation is possible.

[0097] The above data augmentation method further includes the step of measuring an activation value for object inference for each pixel in the data-enhanced image; and the step of calculating the degree of background bias of the data-enhanced image based on the previously measured activation value.

[0098] Here, the step of calculating the degree of background bias includes the step of measuring the contribution of the object part and the contribution of the background part in the data-enhanced image, respectively, and the step of calculating the degree of background bias based on the ratio of the contribution of the object part and the contribution of the background part.

[0099] Here, the step of measuring the contribution of the object part and the contribution of the background part includes the step of measuring the integrated gradient of each pixel of the data-enhanced image, and the integrated gradient of the pixel is calculated by the following [Equation 6].

[0100]

[0101] (Here, I(x i ) is the integrated gradient of the i-th pixel of the data-augmented image, x i is a data-enhanced image, x base is the default black image, m is the number of gradients, and k is the k-th gradient)

[0103] A data-enhanced image is generated based on the synthetic features in the above S400 (S500).

[0104] The weak map semantic segmentation model obtained through the data augmentation methods of each step (S100 ~ S500) described above is characterized by being evaluated by the evaluation metric mIoU (mean Intersection over Union). To verify the performance of the generated category activation map and pixel-unit labels, a comparative experiment was conducted with the existing weak map semantic segmentation methodology on the PASCAL VOC 2012 dataset.

[0105] To briefly explain the evaluation metric mIoU (mean Intersection over Union) used in the present invention, mIoU refers to the average value of IoU values. As an evaluation method for semantic partitioning models, mIoU is used by calculating the average of the classes after calculating the IoU (Intersection over Union) for each class, and IoU is characterized by being calculated as true positive / (true positive + false positive + false negative).

[0106] During the evaluation experiment, mean Intersection over Union (mIoU) was evaluated by performing data augmentation using the data augmentation method according to an embodiment of the present invention on the existing weak map semantic segmentation method.

[0107] As a result of the experiment, the augmentation method according to the present invention showed improved performance compared to existing research.

[0108]

[0109] [Table 1] Comparison of mIou performance for category activation maps and pixel-unit labels with existing weak map semantic segmentation methods.

[0111] In addition to [Table 1] above, a deep learning-based semantic segmentation model was trained using pixel-unit labels generated by applying the proposed data augmentation method to the existing weak map semantic method.

[0112]

[0113] [Table 2] Performance Evaluation of the Generated Pixel-Level Label-Based Semantic Segmentation Model

[0114] As shown in [Table 2] above, experimental results confirmed that when the method according to the embodiment of the present invention is applied, semantic segmentation accuracy is improved, and high-quality pixel-unit labels are generated compared to the existing method.

[0115] Below, a qualitative evaluation was performed with existing methods to visually verify the pixel-unit labels generated through the data augmentation method according to an embodiment of the present invention.

[0116] Referring to FIG. 6, pixel-unit labels generated through the conventional weak map semantic method capture the background area as an object or capture only a part of the object, whereas the image (SMA Ours in FIG. 6) generated by the data augmentation method according to the embodiment of the present invention effectively generates labels for object areas while being relatively less affected by the background.

[0118] Referring to Figure 7, a qualitative comparison was performed on category activation maps generated according to a data augmentation method applicable to the weak map semantic segmentation method.

[0119] As a result of the experiment, it can be confirmed that the image (SMA Ours in FIG. 7) produced by the data augmentation method according to the embodiment of the present invention captures the object region more accurately compared to the existing data augmentation method.

[0121] FIG. 8 is a block diagram illustrating a computing environment (10) including a computing device suitable for use in an embodiment of the present invention. In the illustrated embodiment, each component may have different functions and capabilities in addition to those described below, and may include additional components in addition to those described below.

[0122] The illustrated computing environment (10) includes a computing device (12). In one embodiment, the computing device (12) may be a data augmentation device (D).

[0123] The computing device (12) includes at least one processor (14), a computer-readable storage medium (16), and a communication bus (18). The processor (14) can cause the computing device (12) to operate according to the exemplary embodiment described above. For example, the processor (14) can execute one or more programs stored in the computer-readable storage medium (16). The one or more programs may include one or more computer-executable instructions, and the computer-executable instructions may be configured to cause the computing device (12) to perform operations according to the exemplary embodiment when executed by the processor (14).

[0124] A computer-readable storage medium (16) is configured to store computer-executable instructions or program code, program data and / or other suitable forms of information. A program (20) stored in the computer-readable storage medium (16) includes a set of instructions executable by a processor (14). In one embodiment, the computer-readable storage medium (16) may be memory (volatile memory such as random access memory, non-volatile memory, or a suitable combination thereof), one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, other forms of storage media that are accessed by a computing device (12) and capable of storing desired information, or a suitable combination thereof.

[0125] The communication bus (18) interconnects various other components of the computing device (12), including the processor (14) and the computer-readable storage medium (16).

[0126] The computing device (12) may also include one or more input / output interfaces (22) and one or more network communication interfaces (26) that provide interfaces for one or more input / output devices (24). The input / output interfaces (22) and network communication interfaces (26) are connected to a communication bus (18). The input / output devices (24) may be connected to other components of the computing device (12) through the input / output interfaces (22). An exemplary input / output device (24) may include an input device such as a pointing device (such as a mouse or trackpad), a keyboard, a touch input device (such as a touchpad or touchscreen), a voice or sound input device, various types of sensor devices and / or imaging devices, and / or an output device such as a display device, a printer, a speaker and / or a network card. An exemplary input / output device (24) may be included inside the computing device (12) as a component constituting the computing device (12), or it may be connected to the computing device (12) as a separate device distinct from the computing device (12).

[0128] Although representative embodiments of the present invention have been described in detail above, those skilled in the art will understand that various modifications can be made to the above-described embodiments without departing from the scope of the present invention. Therefore, the scope of the present invention should not be limited to the described embodiments, but should be defined by the claims set forth below as well as equivalents thereof. Explanation of the symbols

[0131] D: Augmentation device 100 : Encoder 200 : Accounting Department 300 : Shuffle section 400 : Composite part

Claims

Claim 1 A data augmentation method for removing background bias during semantic segmentation of a weak map, comprising: a step of inputting a plurality of images into an encoder to extract features for each image; a step of inputting the extracted features of each image into a pre-trained first aggregation unit and a second aggregation unit to separate features of an object part and a background part in each image; a step of inputting the features of the object part and the features of the background part of each image into a shuffle unit and shuffling the features of the background part in the shuffle unit; a step of synthesizing the shuffled features of the background part and the features of the object part in a synthesis unit to generate synthetic features; a step of generating a data-augmented image based on the synthetic features; a step of measuring activation values ​​for object inference for each pixel in the data-augmented image; and a step of calculating the degree of background bias of the data-augmented image based on the measured activation values. Claim 2 In claim 1, the data augmentation method further comprises the step of learning the first aggregation unit and the second aggregation unit, wherein the learning step comprises: the step of inputting an image into an encoder to extract features of the image; the step of inputting the features of the image into the first aggregation unit to aggregate the features of the object part from the features of the image; the step of inputting the features of the image into the second aggregation unit to aggregate the features of the background part from the features of the image; and the step of performing contrast learning so as to reduce the similarity between the aggregated features of the object part and the aggregated features of the background part. Claim 3 delete Claim 4 A data augmentation method according to claim 1, wherein the step of calculating the degree of background bias comprises: a step of measuring the contribution of an object part and the contribution of a background part in the data-enhanced image, respectively; and a step of calculating the degree of background bias based on the ratio of the contribution of the object part and the contribution of the background part. Claim 5 In claim 4, the step of measuring the contribution of the object portion and the contribution of the background portion includes the step of measuring the integrated gradient of each pixel of the data-enhanced image, wherein the integrated gradient of the pixel is calculated by the following mathematical formula. (Mathematical formula) (Here, I(xi): integrated gradient of the i-th pixel of the data-augmented image, xi: data-augmented image, xbase: pre-set black image, m: number of gradients, k: k-th gradient) Claim 6 A data augmentation method for removing background bias during semantic segmentation of a weak map, comprising: a step of inputting a plurality of images into an encoder to extract features for each image; a step of inputting the extracted features of each image into a pre-trained first aggregation unit and a second aggregation unit to separate features of an object part and a background part in each image; a step of inputting the features of the object part and the features of the background part of each image into a shuffle unit and shuffling the features of the object part in the shuffle unit; a step of synthesizing the shuffled features of the object part and the features of the background part in a synthesis unit to generate a synthesized feature; a step of generating a data augmented image based on the synthesized feature; a step of measuring an activation value for object inference for each pixel in the data augmented image; and a step of calculating the degree of background bias of the data augmented image based on the measured activation value. Claim 7 A data augmentation device for removing background bias during semantic segmentation of a map, comprising: an encoder that receives a plurality of images and extracts features for each of the images; a first aggregation unit and a second aggregation unit that receive features of the images extracted by the encoder and separate an object part and a background part from each of the images; a shuffle unit that receives features of the object part and features of the background part of each image from the first aggregation unit and the second aggregation unit and shuffles the features of the background part; and a synthesis unit that synthesizes the features of the background part shuffled by the shuffle unit and the features of the object part to generate a synthesized feature; and generating a data augmented image based on the synthesized feature of the synthesis unit, wherein the data augmentation measures an activation value for object inference for each pixel in the data augmented image and calculates the degree of background bias of the data augmented image based on the measured activation value. Claim 8 In claim 7, the data augmentation device is characterized by training the first aggregation unit and the second aggregation unit, inputting an image into an encoder to extract features of the image, inputting the features of the image into the first aggregation unit to aggregate features of the object part from the features of the image, inputting the features of the image into the second aggregation unit to aggregate features of the background part from the features of the image, and performing contrast learning so as to reduce the similarity between the aggregated features of the object part and the aggregated features of the background part. Claim 9 delete Claim 10 A data augmentation device according to claim 7, characterized in that the calculation of the degree of background bias is based on the ratio of the contribution of the object part and the contribution of the background part in the data-augmented image, by measuring the contribution of the object part and the contribution of the background part, respectively. Claim 11 In claim 10, the measurement of the contribution of the object part and the contribution of the background part is the measurement of the integrated gradient of each pixel of the data-enhanced image, wherein the integrated gradient of the pixel is calculated by the following mathematical formula, a data enhancement device.[Mathematical Formula] (Here, I(xi): integrated gradient of the i-th pixel of the data-augmented image, xi: data-augmented image, xbase: pre-set black image, m: number of gradients, k: k-th gradient) Claim 12 A computer program stored on a non-transitory computer-readable storage medium, wherein the computer program comprises one or more instructions, and when the instructions are executed by a computing device having one or more processors, the computing device performs the steps of: inputting a plurality of images into an encoder to extract features for each image; inputting the extracted features of each image into a pre-learned first aggregation unit and a second aggregation unit to separate features of an object part and a background part in each image; inputting the features of the object part and the background part of each image into a shuffle unit and shuffling the features of the background part in the shuffle unit; synthesizing the shuffled features of the background part and the features of the object part in a synthesis unit to generate synthetic features; generating a data-enhanced image based on the synthetic features; measuring an activation value for object inference for each pixel in the data-enhanced image; and calculating the degree of background bias of the data-enhanced image based on the measured activation value.

Images