Method and apparatus for generating training data for artificial intelligence image analysis models.

Abstract

A method and device for generating training data for an artificial intelligence image analysis model are provided. [Solution] A method for generating training data for a deep learning model includes the steps of: (a) inputting a background image and object-related information; (b) generating a target object image that meets certain conditions based on the object-related information; (c) applying the background image to a depth estimation model to estimate the depth of each pixel position and generate a depth map; (d) determining an insertion position and scale at which the target object image is inserted based on the depth map and generating a regional insertion object image and a mask image, respectively; (e) simply combining the regional insertion object image and the background image, and then generating a composite image that reflects the scale characteristics of the background image through a learned multi-scale image analysis model; and (f) generating a final training image using the background image, the composite image, and the mask image.

Description

Technical Field

[0001] The present invention relates to a method for generating learning data for an artificial intelligence image analysis model and an apparatus therefor.

Background Art

[0002] Typical artificial intelligence-based object detection technologies include R-CNN (Region-based Convolutional neural network), Faster R-CNN, YOLO (You Only Look Once), etc. R-CNN consists of a region proposal step for estimating object positions in an image and a classification step for classifying object types. Faster R-CNN integrates the region proposal and classification steps into one network to improve the processing speed. Also, YOLO, published in 2016, is a model that processes the entire image at once to detect objects and has contributed to improving the processing speed and efficiency in implementation.

[0003] Although YOLO has been improved in performance through continuous version updates, the complexity of the network has increased due to the performance improvement. To learn a complex network, a large amount of learning data is required. However, in actual application fields, it is difficult to obtain learning data, and a simple and inexpensive network is often needed.

[0004] For example, in an autonomous driving vehicle, when using YOLO to detect vehicles, pedestrians, and traffic lights, since the vehicle travels in various environments, a model that can detect various types of vehicles, pedestrians, and traffic lights is required. However, to detect various types of objects, a large amount of learning data is required, and due to the condition that objects must be detected in real time, a complex network cannot be used. The application environments of image analysis models such as object tracking and segmentation are similar to those of such deep learning-based object detection models.

[0005] Therefore, in order to effectively perform image analysis using artificial intelligence models in diverse application fields, it is extremely important to be able to easily generate training data for each field. [Overview of the project] [Problems that the invention aims to solve]

[0006] The present invention has been made in view of the above-mentioned prior art, and the object of the present invention is to provide a method and apparatus for generating training data for artificial intelligence image analysis models. [Means for solving the problem]

[0007] A method for generating training data for an artificial intelligence image analysis model according to one aspect of the present invention, made to achieve the above objective, comprises: (a) inputting a background image and object-related information; (b) generating a target object image that meets certain conditions based on the object-related information; (c) estimating the depth of each pixel position by applying the background image to a depth estimation model and generating a depth map; (d) determining the insertion position and scale in which the target object image is inserted based on the depth map and generating a regional insertion object image and a mask image, respectively; (e) simply combining the regional insertion object image and the background image, and then generating a composite image that reflects the scale features of the background image through a trained multi-scale image analysis model; and (f) generating a final training image using the background image, the composite image, and the mask image.

[0008] The aforementioned object-related information includes an object image and a description of the object, the description may include a descriptive prompt and at least one condition for the object. The depth estimation model may, after extracting a background feature map from the background image, apply a normalization layer to normalize it, then apply a channel splitting operation to separate the channels, apply a multi-perceptron neural network model to the channel-separated background feature map to estimate a pixel-by-pixel depth feature map, and then apply a channel shuffling operation to the estimated pixel-by-pixel depth feature map and the channel-separated background feature map to mix them and then combine them to generate a final depth map. Step (d) above may include: determining an insertion position in which the target object image is inserted into the space corresponding to the background image; scaling the target object image by a depth value corresponding to the insertion position based on the depth map; inserting the scaled target object image into the insertion position in the space corresponding to the background image to generate a regional object insertion image; and generating the mask image using the regional object insertion image. The mask image may contain information regarding the position and size at which the target object image will be inserted into the background image. Step (d) above may be repeated with different insertion positions and different scales. The multiscale image analysis model includes a plurality of scale image generation blocks that, after inputting an image in which a generated image generated by a lower-scale image generation block has been upsampled by a factor of r, divide it into a crowd without pixel superposition, apply self-attentional operations to extract a crowd unit feature map, and apply a random mask to generate a scale generated image; and a plurality of discrimination blocks that adjust the weight values ​​of the generator blocks located in the scale image generation block so as to distinguish between a background image downsampled by a factor of r and the scale generated image, wherein the multiscale image analysis model can be pre-trained using the background image. The aforementioned plurality of scale image generation blocks include image generation blocks and discrimination blocks for each different scale, and may be composed of pairs of image generation blocks and discrimination blocks of the same scale. The lowest resolution Nth-order scale image generation block among the multiple scale image generation blocks is an Nth-order random noise image. After TIFF0007839578000001.tif9128 is input, an Nth-order generated image similar to the background image is generated. TIFF0007839578000002.tif9128 is generated, and the zero-order discriminant block with the highest resolution among the multiple discriminant blocks is used to generate the background image and zero-order scaled image without downsampling. The weight values ​​of the generator blocks within the zero-order scale image generation block can be adjusted using TIFF0007839578000003.tif9128. The aforementioned multiple scale image generation blocks (for example, Nth order block, Each of the TIFF0007839578000004.tif10128 files represents a scaled-down image by one level. A context attention block is formed by first inputting TIFF0007839578000005.tif9128, then applying a simple linear iterative crowding algorithm to divide it into crowds without pixel overlap, applying a convolution operation within each crowd to embed it in the feature space, then applying a K-proximity algorithm to crowdify it, setting the center vector of the t-th superpixel, performing a feature aggregation using the number of crowded samples to calculate the aggregated center vector and performing a self-attention operation, then using a dispatch operation to add the features of the aggregated center vector to each crowd element by element to extract a crowd-unit feature map; a random mask block is formed at the end of the context attention block, and after the crowd-unit feature map, which is the output of the context attention block, is input, a random mask block is formed by patch embedding and random masking to extract a randomly masked feature map; and after the image obtained by summing the masked feature map, which is the output of the random mask block, and a target-scale random noise image is input, a target-scale feature map is extracted, weights are applied to each channel and regionally important feature, and then the scale image of the first lower stage is formed. It may include a target scale generator block that learns target scale features by adding TIFF0007839578000006.tif9128. Step (e) is performed by having the simply synthesized image downsampled to a predetermined minimum scale, the Nth order scale, and inputting it into the trained multiscale image analysis model, after the downsampled Nth order image is input, dividing it into a crowd without pixel superposition, then applying self-attention operations to extract a target scale crowd unit feature map, and then generating a target scale synthesized image that reflects the target scale features pre-trained for the background image, and the multiscale image analysis model repeats the process of generating a synthesized image by upsampling the target scale synthesized image by a factor of r until the scale of the target scale synthesized image is equal to that of the background image.

[0009] To achieve the above objective, a computing device for generating training data for an artificial intelligence image analysis model according to one aspect of the present invention comprises a memory for storing at least one instruction word and a processor for executing the instruction word stored in the memory, wherein each instruction word executed by the processor performs the following steps: (a) inputting a background image and object-related information; (b) generating a target object image that meets certain conditions based on the object-related information; (c) estimating the depth of each pixel position by applying the background image to a depth estimation model and generating a depth map; (d) determining the insertion position and scale in which the target object image is inserted based on the depth map and generating a regional insertion object image and a mask image, respectively; (e) simply combining the regional insertion object image and the background image and then generating a composite image that reflects the scale features of the background image through a trained multi-scale image analysis model; and (f) generating a final training image using the background image, the composite image, and the mask image. [Effects of the Invention]

[0010] According to the method and apparatus for generating training data for artificial intelligence image analysis models of the present invention, data for training artificial intelligence image analysis models can be generated even in hard-to-reach environments such as county facilities and hazardous areas.

[0011] Furthermore, according to the present invention, the model can be made lightweight for training an artificial intelligence image analysis model that is used restrictively in a special environment, and it can also be applied to embedded systems. [Brief explanation of the drawing]

[0012] [Figure 1] This flowchart shows a method for generating training data for an artificial intelligence image analysis model according to one embodiment of the present invention. [Figure 2] This figure shows the configuration of an object generation model according to one embodiment of the present invention. [Figure 3]It is a diagram showing the configuration of a depth estimation model according to an embodiment of the present invention. [Figure 4] It is a flowchart showing a method for generating a regional object insertion image and a mask image according to an embodiment of the present invention. [Figure 5] It is a diagram illustrating a target object image according to an embodiment of the present invention. [Figure 6] It is a diagram illustrating a depth map, a composite image, and a mask image of a background image according to an embodiment of the present invention. [Figure 7] It is a diagram shown to explain the learning process of a multi-scale image analysis model according to an embodiment of the present invention. [Figure 8] It is a diagram showing the detailed structure of a scale image generation block according to an embodiment of the present invention. [Figure 9] It is a diagram shown to explain the generation process of a composite image of a multi-scale image analysis model in which features of various sizes are analyzed according to an embodiment of the present invention. [Figure 10] It is a diagram schematically showing the internal configuration of an arithmetic device for generating learning data of an artificial intelligence image analysis model according to an embodiment of the present invention.

Embodiments for Carrying Out the Invention

[0013] As used in this specification, the singular expressions include plural expressions unless otherwise clearly indicated in the context. In this specification, terms such as "configured to" or "including" should not necessarily be construed as including all of the many components or many steps described in the specification, and some of the components or some of the steps may not be included, or it should be construed that additional components or steps may be further included. Also, terms such as "… part" and "module" described in the specification mean a unit that processes at least one feature or operation, which may be implemented by hardware or software, or implemented by a combination of hardware and software.

[0014] Hereinafter, specific examples of embodiments for carrying out the present invention will be described in detail while referring to the drawings.

[0015] FIG. 1 is a flowchart showing a method for generating learning data of an artificial intelligence image analysis model according to an embodiment of the present invention, FIG. 2 is a diagram showing a configuration of an object generation model according to an embodiment of the present invention, FIG. 3 is a diagram showing a configuration of a depth estimation model according to an embodiment of the present invention, FIG. 4 is a flowchart showing a method for generating a regional object insertion image and a mask image according to an embodiment of the present invention, FIG. 5 is a diagram illustrating a target object image according to an embodiment of the present invention, FIG. 6 is a diagram illustrating a depth map, a composite image, and a mask image of a background image according to an embodiment of the present invention, FIG. 7 is a diagram shown for explaining a learning process of a multi-scale image analysis model according to an embodiment of the present invention, FIG. 8 is a diagram showing a detailed structure of a scale image generation block according to an embodiment of the present invention, and FIG. 9 is a diagram shown for explaining a generation process of a composite image of a multi-scale image analysis model in which features of various sizes are analyzed according to an embodiment of the present invention.

[0016] In step 110 of FIG. 1, the arithmetic device 100 shown in FIG. 10 inputs a background image and object-related information.

[0017] Here, the object-related information is at least one of an object image and a description of the object. Here, the description of the object includes a prompt (text) for explaining the object to be generated and one or more conditions (for example, the pose of the object, etc.).

[0018] In step 115, the arithmetic device 100 applies the object-related information to the object generation model to generate a target object image. FIG. 2 shows an example of the object generation model. Referring to FIG. 2, the object generation model includes a first encoder 210, a second encoder 220, and a generation decoder 230.

[0019] Either the first encoder 210 or the second encoder 220 is an encoder that processes images, and the other is an encoder that processes descriptions. In one embodiment of the present invention, we will assume that the first encoder 210 is an encoder that processes images and the second encoder 220 is an encoder that processes descriptions, and will explain the invention primarily based on this assumption.

[0020] The first encoder 210 is a means for extracting an object feature map after an object image from the object-related information has been input. Here, the object feature map is a feature map based on a synthetic neural network.

[0021] The second encoder 220 is a means for extracting a condition feature map after a description has been input. The second encoder 220 has a feature extraction network for each condition domain for multiple conditions, and extracts a condition domain feature map through the feature extraction network for each condition domain.

[0022] The generating decoder 230 generates a target object image that meets the conditions using at least one of the object feature map and the condition domain feature map.

[0023] In this way, the computing device 100 uses object-related information to extract at least one of the object feature map and the condition domain feature map through at least one of the multiple encoders, and uses these to generate a target object image suitable for the conditions.

[0024] In step 120, the computing unit 100 applies the background image to a depth estimation model to estimate the depth of each pixel position and generate a depth map.

[0025] This will be explained in more detail by referring to Figure 3.

[0026] The depth estimation model consists of a feature extraction module 310 and a feature synthesis module 320. The feature extraction module 310 is a DenseNet-169-based model. As a DenseNet-169-based model, the feature extraction module 310 enables the construction of a deep network by connecting all layers with dense blocks, and can extract enhanced feature maps by combining low-level and high-level feature maps. In one embodiment of the present invention, we assume that the feature extraction module 310 is a DenseNet-169-based model and will explain it primarily based on this, but other models (e.g., encoders) may also be applied.

[0027] The feature extraction module 310, after receiving a background image as input, extracts a feature map as cognitive information through the DenseNet-169 underlying encoder.

[0028] The feature synthesis module 320 separates the feature map input from the feature extraction module 310 and then generates a depth map for each pixel.

[0029] The feature synthesis module 320 applies a normalization layer to the feature map input from the feature extraction module 310 to normalize it, and then applies a channel separation operation to separate the channels of the feature map. Next, the feature synthesis module 320 applies a multi-layer perceptron neural network model to the channel-separated feature map to estimate a depth feature map for each pixel. Here, the multi-layer perceptron neural network model is PointMLP. The feature synthesis module 320 applies a channel shuffling operation to the estimated pixel-by-pixel depth feature map and the channel-separated feature map, mixes them, and then combines them to output the final depth map.

[0030] In step 125, the computing unit 100 determines the position and scale of the target object based on the target object image and depth map, and then generates a regional object insertion image and a mask image, respectively.

[0031] Figure 4 shows the process of generating regional object insertion images and mask images.

[0032] In step 410, the arithmetic unit 100, after receiving the target object image and depth map as input, determines random positions (referred to as insertion positions for convenience) where the object will be inserted.

[0033] Next, in step 415, the arithmetic unit 100 scales the size of the target object image (i.e., the target object) by the depth value (depth value of the depth map) corresponding to the predetermined insertion position.

[0034] In step 420, the computing unit 100 generates a regional object insertion image based on the scaled object and insertion position. Here, the size of the regional object insertion image is equal to that of the background image or depth map. That is, the scaled target object image is inserted into the insertion position in the space corresponding to the background image to generate a regional object insertion image in which the target object is localized and scaled.

[0035] In step 425, the arithmetic unit 100 generates a mask image corresponding to the regional object insertion image. Here, the mask image is an image in which the object and background are binaryly separated in the regional object insertion image, for example, the pixel values ​​of the object region have a first value and the pixel values ​​of the background region have a second value.

[0036] As a result, the mask image contains information about where and at what size the target object image was inserted (combined) within the background image, and is used as a training label.

[0037] For example, let's assume the target object image is as shown in Figure 5. In this case, 610 in Figure 6 shows the depth map generated based on the background image. The regional object insertion image generated by scaling the target object image to reflect the depth value in which the target object image in Figure 5 is inserted is as shown in 620, and a mask image indicating the insertion position and size of the target object image in the regional object insertion image is generated as shown in 630.

[0038] Figure 4 illustrates the process assuming that the target object image is inserted at a single location; however, the process in Figure 4 is repeated at multiple insertion locations. Furthermore, it goes without saying that the scaling of the target object at a single location is not limited to a single size, but can be scaled to a variety of sizes.

[0039] In other words, the computing device 100 generates multiple regional object insertion images and mask images by determining the insertion position and scaling them, so that target object images can be synthesized at various positions within the background image.

[0040] In step 130, the computing unit 100 generates a simple composite image by simply combining the regional object insertion image and the background image.

[0041] In step 135, the computing unit 100 downsamples the simple composite image to a predetermined Nth-order scale, then applies it to the learned multi-scale image analysis model to generate a composite image that reflects the scale features relative to the background image.

[0042] The multiscale image analysis model is pre-trained, but for the sake of understanding and explanation, we will first describe the training process of the multiscale image analysis model with reference to Figure 7.

[0043] The multiscale image analysis model is based on a generative adversarial model.

[0044] Referring to Figure 7, the multi-scale image analysis model consists of multiple scale image generation blocks (710a to 710n) and multiple discrimination blocks (720a to 720n). Here, among the scale image generation blocks and discrimination blocks, those that process images of the same scale are configured in pairs. Unless otherwise explained below, the first scale image generation block 710a and the first discrimination block 720a operate as a pair, and the nth scale image generation block 710n and the nth discrimination block 720n operate as a pair.

[0045] In the following, lower-scale representations should be understood as representing a scale (resolution) one level lower than the image scale (resolution) currently processed by the scale image generation block or discrimination block, and higher-scale representations should be understood as representing a scale (resolution) one level higher.

[0046] Furthermore, the resolution (scale) processed by the currently processing block (e.g., the scale image generation block or the discrimination block) will be referred to as the target scale to distinguish it from lower and higher-level representations. Therefore, the target scale representation must be understood as the specified scale (resolution) of the currently processing block (e.g., the scale image generation block or the discrimination block).

[0047] As shown in Figure 7, a multi-scale image analysis model according to one embodiment of the present invention includes a plurality of scale image generation blocks (710a to 710n) and a plurality of discrimination blocks (720a to 720n) that process images at different scales.

[0048] The following describes how a multi-scale image analysis model uses a single image (e.g., a background image) to enable each scale image generation block to learn target scale features, with reference to Figure 7.

[0049] According to one embodiment of the present invention, the multiple scale image generation blocks operate starting from the lowest scale image generation block, followed sequentially by the higher scale image generation blocks, and this process is repeated up to the highest scale image generation block.

[0050] For the sake of understanding and explanation, the smallest scale image generation block will be referred to as the Nth-order scale image generation block 710a, and the highest-order scale image generation block will be referred to as the 0th-order scale image generation block 710n.

[0051] For training a multiscale image analysis model, the Nth-order scale image generation block 710a generates an Nth-order scale random noise image. After TIFF0007839578000007.tif9128 is input, an Nth-order generated image similar to the background image is generated. The arithmetic unit 100 generates TIFF0007839578000008.tif9128. Next, the arithmetic unit 100 generates the Nth generation image. TIFF0007839578000009.tif9128 is upsampled by a factor of r and transferred to a higher-scale image generation block (for example, the N-1 scale image generation block 710b).

[0052] The remaining scale image generation blocks (710b to 710n), excluding the Nth-order scale image generation block, consist of a context attention block 712, a random mask block 714, and a generator block 716, respectively, as shown in Figure 8.

[0053] Context attention block 712 takes an image as input, which is an r-fold upsampled output from a lower-scale image generation block, then divides it into a crowd without pixel superposition, and applies a self-attention operation to extract a crowd-unit feature map.

[0054] More specifically, context attention block 712 converts the output of the lower-scale image generation block into an r-fold upsampled image in a normalized coordinate space. After converting to TIFF0007839578000010.tif17128, each pixel becomes a data point Convert to TIFF0007839578000011.tif12128. Here, TIFF0007839578000012.tif9128 represents the feature dimensions of the input, and 2 represents the pixel coordinates.

[0055] Next, context attention block 712 applies a Simple Linear Iterative Clustering (SLIC) algorithm to divide the pixels into non-overlapping clusters. Then, context attention block 712 applies a convolutional layer within each cluster to embed them in the feature space, and then clusters them using the K-Nearest Neighbors (KNN) algorithm, and then the t-th superpixel Center vector of TIFF0007839578000013.tif9128 After setting TIFF0007839578000014.tif8128, feature aggregation is performed on M crowd samples, and the aggregated center vector is obtained. The TIFF0007839578000015.tif9128 is calculated. Furthermore, the context attention block 712 applies a self-attention operation to perform attention operations between each group, and then uses a dispatch operation to transmit the features of the representative vector to each group as an element-wise sum, outputting a group-unit feature map.

[0056] The random mask block 714 is located at the trailing end of the context attention block 712 and takes the crowd unit feature map output from the context attention block 712 as input to extract a randomly masked feature map by patch embedding and random masking.

[0057] The output of the random mask block 714 and the target-scale random noise image are combined and transmitted to the generator block 716. The generator block 716 receives the combined image of the output of the random mask block 714 and the target-scale random noise image as input, extracts a target-scale feature map, and learns from it.

[0058] The target-scale feature map is weighted through SE blocks and spatial-attention blocks to focus on channel and geographically important features of the target-scale feature map. Here, the SE block includes two processes: Squeeze (compression) and Excitation (re-adjustment). The feature map that has passed through the SE block and spatial-attention block is then combined with an image that has been upsampled by a factor of r from the output of a lower-scale image generation block to learn features that can be learned at the target scale.

[0059] This process is repeated from the n-1 scale image generation block down to the 0 scale image generation block. This allows the generator blocks at each scale to learn the overall structural features of the image at the smallest size, and to learn more detailed features as the scale approaches 0.

[0060] A multi-scale image analysis model according to one embodiment of the present invention has image generation blocks for each different scale, thereby pre-training features for each different scale using a single background image.

[0061] To summarize again, the scale image generation blocks (710b and 710n) receive an image generated by the lower scale image generation block, which has been upsampled by a factor of r, as input. After dividing this into a crowd without pixel superposition, they apply self-attentional operations to extract crowd-unit feature maps. Then, they generate a target scale image similar to the input background image using randomly masked feature maps through a random mask block. This process is repeated until the highest scale image (0th scale image) is generated.

[0062] Also, the background image input to train the 0th scale generation block 710n TIFF0007839578000016.tif8128 and 0th scale image By comparing it with TIFF0007839578000017.tif10128, the weight of the 0th-order generator block 716n is optimized so that the discrimination block 720n constituting the pair cannot distinguish between the input background image and the generated image. Similarly, the weights of the generator blocks within each target-scale image generation block forming each pair are optimized so that the remaining discrimination blocks (720n-1, 720n-2, ..., 720a) cannot distinguish between the downsampled input background image and the generated image.

[0063] Referring to Figures 7 and 8, the process of training a multi-scale image analysis model based on the input background image is explained.

[0064] Figures 7 and 8 show the operation when training a multiscale image analysis model using a single image (background image). Figure 9 describes how to generate a synthesized image using the trained multiscale image analysis model. As shown in Figure 9, a trained multiscale image analysis model may not function if some components used during the training process are deactivated. For example, the lowest scale image generation block 710a, the remaining random mask blocks (714a-714n) within the scale image generation block, and the lowest scale discrimination block 720a are used only during the training process and are removed or deactivated after training is complete, and are not used to generate a synthesized image.

[0065] The simple composite image is downsampled to the smallest predetermined scale (for convenience, the Nth order scale) and then passed to the target scale image generation block. In one embodiment of the present invention, it is assumed that the simple composite image is downsampled to the Nth order scale, and this is the basis for the explanation, but the simple composite image is scaled randomly from the 0th to the Nth order scale.

[0066] Assuming that the simple composite image is downsampled to an Nth-order scale, the image is input to a higher-scale image generation block (for example, an N-1-order scale image generation block) than the downsampled Nth-order image. As a result, the context attention block located within the target-scale image generation block 710b divides the input image into clusters without pixel superposition, then applies a self-attention operation to transmit the features of the representative vector to each cluster as an element-wise sum using a dispatch operation, extracts a cluster-unit feature map, and transmits it to the generator block.

[0067] The generator block generates a target-scale composite image by reflecting the target-scale features learned from the background image into a crowd-unit feature map. This target-scale composite image is upsampled by a factor of r and transmitted to the higher-scale image generation block, and the above-described series of processes is repeated until a zero-order scale composite image is generated.

[0068] Also, referring to Figure 1, in step 140, the computing unit 100 generates the final training image using the background image, the final composite image, and the mask image.

[0069] Figure 10 is a schematic diagram showing the internal configuration of a computing device for generating training data for a deep learning model according to one embodiment of the present invention.

[0070] Referring to Figure 10, a computing device 100 for generating training data for a deep learning model according to one embodiment of the present invention is configured to include a memory 1010 and a processor 1020.

[0071] Memory 1010 stores at least one instruction word.

[0072] The processor 1020 works in conjunction with the memory 1010 to execute instructions stored in the memory 1010. The instructions executed by the processor 1020 take the background image and object-related information as input, generate a target object image that meets the conditions based on the object-related information, estimate the depth of each pixel position by applying the background image to a depth estimation model to generate a depth map, determine the insertion position and scale where the target object image will be inserted based on the depth map to generate a regional insertion object image and a mask image, respectively, simply combine the regional insertion object image and the background image, generate a composite image that reflects the scale features of the background image through a trained multi-scale image analysis model, and then generate the final training image using the background image, composite image, and mask image. This is the same as the part explained with reference to Figures 1 to 9, so redundant explanations will be omitted.

[0073] The apparatus and methods according to embodiments of the present invention are embodied in the form of program instructions performed by various computer means and recorded on a computer-readable recording medium. The computer-readable recording medium includes program instructions, data files, data structures, etc., individually or in combination. The program instructions recorded on the computer-readable recording medium are either specifically designed and configured for the present invention or are publicly known and usable by those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, Proppy® disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magnetic-optical media such as floppy disks, and hardware devices specifically configured to store and execute program instructions, such as ROMs, RAMs, and flash memory. Examples of program instructions include not only machine code produced by compilers, but also high-level language code executed by computers using interpreters, etc.

[0074] The hardware device described above is configured to operate with at least one software module in order to perform the operation of the present invention, and vice versa.

[0075] To date, the present invention has been described primarily in terms of its embodiments. Those skilled in the art will understand that the present invention can be embodied in modified forms without departing from its essential characteristics. Therefore, the disclosed embodiments should be considered in an explanatory rather than restrictive manner. The scope of the present invention is defined in the claims, not in the above description, and all differences within an equivalent scope should be interpreted as being included in the present invention. [Explanation of Symbols]

[0076] 100 Computing equipment 210, 220 First and second encoders 230 Generating Decoder 310 Feature Extraction Module (DenseNet-169-based model) 320 Feature Synthesis Module 610 Depth Map 620 Regional object insertion images 630 Mask Images 710a First-order scale image generation block (Nth-order scale image generation block) 710n nth scale image generation block (0th scale image generation block) 712 Contextual Attention Block 714 Random Mask Block 716 Generator Block 720a First discrimination block 720n nth Discriminant Block 1010 memory 1020 processor

Claims

1. A method for generating training data for an artificial intelligence image analysis model performed by a computing device, (a) A step of inputting a background image and object-related information, (b) A step of generating a target object image that meets the conditions using at least one of the object feature map and the condition domain feature map based on the object-related information, (c) A step of generating a depth map by applying the background image to a depth estimation model to estimate the depth of each pixel position, (d) A step of inserting the scaled target object image into the insertion position in the space corresponding to the background image based on the depth map to generate a localized and scaled localized object insertion image and a mask image in which the object and background are binary separated in the localized object insertion image, (e) The step of simply combining the regional object insertion image and the background image, and then generating a composite image that reflects the scale features of the background image through a trained multiscale image analysis model, The aforementioned object-related information includes an image of the object and a description of the object. The description includes an explanatory prompt for the object and at least one condition, The multi-scale image analysis model has image generation blocks for each different scale, and pre-trains features for each different scale using a single background image generated for each block. The depth estimation model is characterized by extracting a background feature map from the background image, applying a normalization layer to normalize it, then applying a channel splitting operation to separate the channels, applying a multi-perceptron neural network model to the channel-separated background feature map to estimate a pixel-by-pixel depth feature map, applying a channel shuffling operation to the estimated pixel-by-pixel depth feature map and the channel-separated background feature map to mix them, and then combining them to generate a final depth map.

2. Step (d) above is, The steps include determining the insertion position in which the target object image is inserted into the space corresponding to the background image, The steps include scaling the target object image by the depth value corresponding to the insertion position based on the depth map, The steps include: generating a regional object insertion image by inserting the scaled target object image into the insertion position in the space corresponding to the background image; The method according to claim 1, characterized by comprising the step of generating the mask image using the aforementioned regional object insertion image.

3. The method according to claim 2, characterized in that the mask image includes information on at what position and of what size the target object image is inserted into the background image.

4. The method according to claim 1, characterized in that step (d) is repeated with different insertion positions and different scales.

5. The aforementioned multi-scale image analysis model is The system generates multiple scaled image generation blocks that, after receiving an image that has been upsampled by r times from a lower-scale image generation block, divide it into a crowd without pixel superposition, apply self-attentional operations to extract a crowd-unit feature map, and then apply a random mask to generate a scaled image. The method according to claim 1, further comprising: a plurality of discrimination blocks that adjust the weight values ​​of generator blocks located in the scale image generation block so as to distinguish between a background image downsampled by a factor of r and a scale generation image.

6. The method according to 5, characterized in that the plurality of scale image generation blocks include image generation blocks and discrimination blocks for each different scale, and the same scale image generation block and discrimination block are configured as a pair.

7. The Nth-order scale image generation block with the lowest resolution among the multiple scale image generation blocks is an Nth-order random noise image. After the input, an Nth-order generated image similar to the background image is generated. Generate, The zero-order discrimination block with the highest resolution among the multiple discrimination blocks is used to generate the background image and the zero-order scaled image without downsampling. The method according to 5, characterized in that the weight values ​​of the generator blocks in the zero-order scale image generation block are adjusted using

8. Each of the aforementioned multiple scale image generation blocks is: Image generated at a lower scale level After the input, a simple linear iterative crowding algorithm is applied to divide the data into crowds without pixel overlap, a convolution operation is applied within each crowd to embed it in the feature space, a K-proximity algorithm is applied to crowdify the data, the center vector of the t-th superpixel is set, feature aggregation is performed using the number of crowded samples to calculate the aggregated center vector, a self-attention operation is performed, and then the features of the aggregated center vector are added element by element to each crowd by a dispatch operation to extract a crowd-unit feature map. A random mask block located at the rear end of the context attention block, which, after receiving the crowd unit feature map that is the output of the context attention block, extracts a feature map that has been randomly masked by patch embedding and random masking, After receiving an image obtained by summing the masked feature map, which is the output of the random mask block, and the target-scale random noise image, the target-scale feature map is extracted, and then weights are applied to each channel and regionally important feature, followed by the lower-scale image from the previous step. The method according to 7, characterized by comprising a target scale generator block that learns target scale features by adding to

9. Step (e) above is, The aforementioned simply synthesized image is downsampled to a predetermined minimum scale, the Nth order scale, and input into the learned multi-scale image analysis model. The trained multiscale image analysis model, after receiving an image downsampled to the Nth order scale, divides it into a crowd without pixel superposition, then applies self-attention calculations to extract a target-scale crowd unit feature map, and then generates a target-scale composite image that reflects the pre-trained target-scale features on the background image. The method according to claim 1, characterized in that the multiscale image analysis model repeats the process of generating a composite image in which the target scale composite image is upsampled by a factor of r until the scale of the target scale composite image becomes equal to that of the background image.

10. A computer-readable recording medium having a program code for performing the method described in any one of claims 1 to 9.

11. Memory to store at least one instruction word, The system comprises a processor that executes instruction words stored in the memory, Each instruction word executed by the aforementioned processor is: (a) A step of inputting a background image and object-related information, (b) A step of generating a target object image that meets the conditions using at least one of the object feature map and the condition domain feature map based on the object-related information, (c) A step of generating a depth map by applying the background image to a depth estimation model to estimate the depth of each pixel position, (d) A step of inserting the scaled target object image into the insertion position in the space corresponding to the background image based on the depth map to generate a localized and scaled localized object insertion image and a mask image in which the object and background are binary separated in the localized object insertion image, (e) After simply combining the regional object insertion image and the background image, a composite image is generated that reflects the scale features of the background image through a trained multiscale image analysis model. The aforementioned object-related information includes an image of the object and a description of the object. The description includes an explanatory prompt for the object and at least one condition, The multi-scale image analysis model has image generation blocks for each different scale, and pre-trains features for each different scale using a single background image generated for each block. The depth estimation model is a computing device characterized by extracting a background feature map from the background image, applying a normalization layer to normalize it, applying a channel splitting operation to separate the channels, applying a multi-perceptron neural network model to the channel-separated background feature map to estimate a pixel-by-pixel depth feature map, applying a channel shuffling operation to the estimated pixel-by-pixel depth feature map and the channel-separated background feature map to mix them, and then combining them to generate a final depth map.

12. Step (d) above is, The steps include determining the insertion position in which the target object image is inserted into the space corresponding to the background image, The steps include scaling the target object image by the depth value corresponding to the insertion position based on the depth map, The steps include: generating a regional object insertion image by inserting the scaled target object image into the insertion position in the space corresponding to the background image; The computing device according to claim 11, characterized by performing the step of generating the mask image using the aforementioned regional object insertion image.

13. The aforementioned multi-scale image analysis model is The system generates multiple scaled image generation blocks that, after receiving an image that has been upsampled by r times from a lower-scale image generation block, divide it into a crowd without pixel superposition, apply self-attentional operations to extract a crowd-unit feature map, and then apply a random mask to generate a scaled image. It includes a plurality of discrimination blocks that adjust the weight values ​​of generator blocks located in the scale image generation block so as to distinguish between a background image downsampled by a factor of r and a scale generation image, The arithmetic device according to claim 11, characterized in that the plurality of scale image generation blocks include image generation blocks and discrimination blocks for each different scale, and the same scale image generation block and discrimination block are configured as a pair.

14. The Nth-order scale image generation block with the lowest resolution among the multiple scale image generation blocks is an Nth-order random noise image. After the input, an Nth-order generated image similar to the background image is generated. Generate, The zero-order discrimination block with the highest resolution among the multiple discrimination blocks is used to generate the background image and the zero-order scaled image without downsampling. The arithmetic device according to claim 13, characterized in that it adjusts the weight values ​​of the generator blocks in the zero-order scale image generation block using the method described above.

15. Each of the aforementioned multiple scale image generation blocks is: Image generated at a lower scale level After the input, a simple linear iterative crowding algorithm is applied to divide the data into crowds without pixel overlap, a convolution operation is applied within each crowd to embed it in the feature space, a K-proximity algorithm is applied to crowdify the data, the center vector of the t-th superpixel is set, feature aggregation is performed using the number of crowded samples to calculate the aggregated center vector, a self-attention operation is performed, and then the features of the aggregated center vector are added element by element to each crowd by a dispatch operation to extract a crowd-unit feature map. A random mask block located at the rear end of the context attention block, which, after receiving the crowd unit feature map that is the output of the context attention block, extracts a feature map that has been randomly masked by patch embedding and random masking, After receiving an image obtained by summing the masked feature map, which is the output of the random mask block, and the target-scale random noise image, the target-scale feature map is extracted, and then weights are applied to each channel and regionally important feature, followed by the lower-scale image from the previous step. The computing device according to claim 14, comprising a target scale generator block that learns target scale features by adding a target scale generator block.

16. Step (e) above is, The aforementioned simply synthesized image is downsampled to a predetermined minimum scale, the Nth order scale, and input into the learned multi-scale image analysis model. The trained multiscale image analysis model, after receiving an image downsampled to the Nth order scale, divides it into a crowd without pixel superposition, then applies self-attention calculations to extract a target-scale crowd unit feature map, and then generates a target-scale composite image that reflects the pre-trained target-scale features on the background image. The computing device according to claim 11, characterized in that the multiscale image analysis model repeats the process of generating a composite image in which the target scale composite image is upsampled by a factor of r until the scale of the target scale composite image becomes equal to that of the background image.

Images