Method and device for generating learning data for artificial intelligence image analysis model

Abstract

A method and device for generating training data for an artificial intelligence image analysis model are provided. [Solution] A method for generating training data for a deep learning model includes the steps of: (a) inputting a background image and object-related information; (b) generating a target object image that meets certain conditions based on the object-related information; (c) applying the background image to a depth estimation model to estimate the depth of each pixel position and generate a depth map; (d) determining an insertion position and scale at which the target object image is inserted based on the depth map and generating a regional insertion object image and a mask image, respectively; (e) simply combining the regional insertion object image and the background image, and then generating a composite image that reflects the scale characteristics of the background image through a learned multi-scale image analysis model; and (f) generating a final training image using the background image, the composite image, and the mask image.

Description

[Technical Field]

[0001] The present invention relates to a method and apparatus for generating training data for an artificial intelligence image analysis model. [Background technology]

[0002] Representative AI-based object detection technologies include R-CNN (Region-based Convolutional neural network), Faster R-CNN, and YOLO (You Only Look Once). R-CNN consists of a region proposal step that estimates the location of objects in an image and a classification step that classifies object types. Faster R-CNN improves processing speed by integrating the region proposal and classification steps into a single network. In addition, YOLO, which was announced in 2016, is a model that detects objects by processing the entire image at once, contributing to improving processing speed and implementation efficiency.

[0003] YOLO's performance has improved through continuous version updates, but the performance improvements have also led to increased network complexity. Training a complex network requires a large amount of training data, but in real-world applications, training data is often difficult to obtain, and simple, inexpensive networks are often required.

[0004] For example, when using YOLO to detect vehicles, pedestrians, and traffic lights in an autonomous vehicle, a model that can detect various types of vehicles, pedestrians, and traffic lights is required because the vehicle drives in a variety of environments. However, detecting various types of objects requires a large amount of training data, and the requirement of detecting objects in real time precludes the use of complex networks. In addition to such deep learning-based object detection models, the application environment for image analysis models such as object tracking and segmentation is similar.

[0005] Therefore, in order to effectively perform image analysis using artificial intelligence models in various application fields, it is very important to easily generate learning data for each field. Summary of the Invention [Problem to be solved by the invention]

[0006] The present invention has been made in view of the above-mentioned conventional techniques, and an object of the present invention is to provide a method and an apparatus for generating training data for an artificial intelligence image analysis model. [Means for solving the problem]

[0007] In order to achieve the above object, one aspect of the present invention provides a method for generating training data for an artificial intelligence image analysis model, comprising: (a) inputting a background image and object-related information; (b) generating a target object image that meets certain conditions based on the object-related information; (c) applying the background image to a depth estimation model to estimate the depth of each pixel position and generate a depth map; (d) determining an insertion position and scale at which the target object image is to be inserted based on the depth map and generating a regional insertion object image and a mask image, respectively; (e) simply combining the regional insertion object image and the background image, and then generating a composite image that reflects the scale characteristics of the background image through a trained multi-scale image analysis model; and (f) generating a final training image using the background image, the composite image, and the mask image.

[0008] The object-related information may include an image of the object and a description of the object, and the description may include an explanatory prompt and at least one condition for the object. The depth estimation model may extract a background feature map for the background image, apply a normalization layer to normalize it, and then apply a channel splitting operation to separate each channel. Then, apply a multi-perceptron neural network model to the channel-separated background feature map to estimate a depth feature map for each pixel. Then, apply a channel shuffling operation to the estimated depth feature map for each pixel and the channel-separated background feature map to mix them and combine them to generate a final depth map. The step (d) may include the steps of: determining an insertion position where the target object image is to be inserted into a space corresponding to the background image; scaling the target object image according to a depth value corresponding to the insertion position based on the depth map; inserting the scaled target object image into the insertion position in the space corresponding to the background image to generate a regional object insertion image; and generating the mask image using the regional object insertion image. The mask image may include information regarding the position and size of the target object image to be inserted into the background image. The step (d) can be repeated with different insertion positions and different scales. The multi-scale image analysis model includes a plurality of scale image generation blocks that input an image in which a generated image generated by a lower-scale image generation block is upsampled r times, divide the image into crowds without pixel overlap, apply a self-attention operation to extract a crowd-based feature map, and apply a random mask to generate a scale generated image; and a plurality of discrimination blocks that adjust weight values ​​of generator child blocks located in the scale image generation blocks so as to distinguish between a background image downsampled r times and the scale generated image, and the multi-scale image analysis model can be pre-trained using the background image. The plurality of scale image generating blocks may include image generating blocks and discrimination blocks for different scales, and image generating blocks and discrimination blocks of the same scale may be paired. The N-th order scale image generation block having the lowest resolution among the plurality of scale image generation blocks is an N-th order random noise image Nth order generated image similar to the background image after TIFF2025538325000002.tif9128 is input TIFF2025538325000003.tif9128 is generated, and the 0th-order discrimination block having the highest resolution among the plurality of discrimination blocks is used to generate the background image and the 0th-order scale generated image without downsampling. TIFF2025538325000004.tif9128 can be used to adjust the weight value of the generator block in the 0th scale image generator block. The plurality of scale image generation blocks (e.g., N-order blocks, Each of TIFF2025538325000005.tif10128 is a scaled-down image. A context attention block is configured to input TIFF2025538325000006.tif9128, divide it into clusters without pixel overlap by applying a simple linear iterative clustering algorithm, embed it in a feature space by applying a convolution operation within each cluster, cluster it by applying a K-nearest neighbor algorithm, set the center vector of the t-th superpixel, perform feature aggregation by the number of clustered samples to calculate the aggregated center vector, perform self-attention operation, and add the features of the aggregated center vector to each cluster element by a dispatch operation to extract a cluster-based feature map. The context attention block is located at the rear of the context attention block, inputs the cluster-based feature map output from the context attention block, and extracts a randomly masked feature map by patch embedding and random masking. The random mask block is configured to input an image obtained by combining the masked feature map output from the random mask block and a target-scale random noise image, extracts a target-scale feature map, and assigns weights to important features for each channel and region, and then extracts the target-scale feature map from the image at a lower scale. and a target scale generator block that adds TIFF2025538325000007.tif9128 to learn target scale features. In step (e), the simply synthesized image is downsampled to a predetermined lowest scale, that is, an N-th scale, and input to the trained multi-scale image analysis model. After the image downsampled to the N-th scale is input, the trained multi-scale image analysis model divides the image into crowds without pixel overlap, applies a self-attention operation to extract a target-scale crowd-unit feature map, and generates a target-scale synthesized image that reflects pre-trained target-scale features for the background image. The multi-scale image analysis model repeats the process of generating a synthesized image by upsampling the target-scale synthesized image by r times until the scale of the target-scale synthesized image becomes equal to that of the background image.

[0009] In order to achieve the above object, according to one aspect of the present invention, a computing device for generating training data for an artificial intelligence image analysis model includes a memory for storing at least one instruction, and a processor for executing the instruction stored in the memory. Each of the instructions executed by the processor performs the following steps: (a) inputting a background image and object-related information; (b) generating a target object image that meets certain conditions based on the object-related information; (c) applying the background image to a depth estimation model to estimate the depth of each pixel position and generate a depth map; (d) determining an insertion position and scale at which the target object image is inserted based on the depth map and generating a regional insertion object image and a mask image, respectively; (e) simply combining the regional insertion object image and the background image, and then generating a composite image that reflects the scale characteristics of the background image through a trained multi-scale image analysis model; and (f) generating a final training image using the background image, the composite image, and the mask image. [Effects of the Invention]

[0010] According to the method and device for generating artificial intelligence image analysis model training data of the present invention, data for artificial intelligence image analysis model training can be generated even in inaccessible environments such as county facilities and dangerous areas.

[0011] Furthermore, according to the present invention, since an artificial intelligence image analysis model that is used in a limited manner in a special environment is trained, the model can be made lightweight and can be applied to embedded systems. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a flowchart illustrating a method for generating training data for an artificial intelligence image analysis model according to an embodiment of the present invention. [Figure 2] FIG. 2 illustrates a configuration of an object generation model according to an embodiment of the present invention. [Figure 3]FIG. 2 illustrates the configuration of a depth estimation model according to an embodiment of the present invention. [Figure 4] 1 is a flowchart illustrating a method for generating a regional object insertion image and a mask image according to an embodiment of the present invention. [Figure 5] 1 is a diagram illustrating an example of a target object image according to an embodiment of the present invention; [Figure 6] 3A-3C are diagrams illustrating a depth map of a background image, a composite image, and a mask image according to one embodiment of the present invention. [Figure 7] 1 is a diagram illustrating a learning process of a multi-scale image analysis model according to an embodiment of the present invention. [Figure 8] FIG. 2 is a diagram illustrating a detailed structure of a scale image generation block according to an embodiment of the present invention. [Figure 9] 10A and 10B are diagrams illustrating a process of generating a synthetic image of a multi-scale image analysis model in which features of various sizes are analyzed according to an embodiment of the present invention; [Figure 10] 1 is a diagram illustrating an internal configuration of a computing device for generating learning data for an artificial intelligence image analysis model according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0013] As used herein, singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as "comprise" or "include" should not be interpreted as including all of the components or steps described in the specification, but should be interpreted as including some of the components or steps, or including additional components or steps. Furthermore, terms such as "unit," "module," and the like used in the specification refer to a unit that processes at least one feature or operation, and may be implemented in hardware or software, or a combination of hardware and software.

[0014] Hereinafter, specific examples of embodiments of the present invention will be described in detail with reference to the drawings.

[0015] FIG. 1 is a flowchart showing a method for generating training data for an artificial intelligence image analysis model according to one embodiment of the present invention, FIG. 2 is a diagram showing the configuration of an object generation model according to one embodiment of the present invention, FIG. 3 is a diagram showing the configuration of a depth estimation model according to one embodiment of the present invention, FIG. 4 is a flowchart showing a method for generating a regional object insertion image and a mask image according to one embodiment of the present invention, FIG. 5 is a diagram showing an example of a target object image according to one embodiment of the present invention, FIG. 6 is a diagram showing an example of a depth map, a composite image, and a mask image of a background image according to one embodiment of the present invention, FIG. 7 is a diagram shown to explain the learning process of a multi-scale image analysis model according to one embodiment of the present invention, FIG. 8 is a diagram showing a detailed structure of a scale image generation block according to one embodiment of the present invention, and FIG. 9 is a diagram shown to explain the process of generating a composite image of a multi-scale image analysis model in which features of various sizes are analyzed according to one embodiment of the present invention.

[0016] In step 110 of FIG. 1, the arithmetic device 100 shown in FIG. 10 receives a background image and object-related information.

[0017] Here, the object-related information is at least one of an object image and a description of the object, where the description of the object includes a prompt (text) describing the object to be generated and one or more conditions (e.g., the posture of the object).

[0018] In step 115, the computing device 100 applies the object-related information to an object generation model to generate a target object image. Figure 2 shows an example of the object generation model. Referring to Figure 2, the object generation model includes a first encoder 210, a second encoder 220, and a generator / decoder 230.

[0019] One of the first encoder 210 and the second encoder 220 is an encoder that processes images, and the other is an encoder that processes descriptions. In one embodiment of the present invention, the first encoder 210 is an encoder that processes images, and the second encoder 220 is an encoder that processes descriptions, and the following description will be centered on this assumption.

[0020] The first encoder 210 is a means for extracting an object feature map after an object image of the object-related information is input, where the object feature map is a feature map based on a synthetic neural network.

[0021] The second encoder 220 is a means for extracting a condition feature map after a description is input. The second encoder 220 has a feature extraction network for each condition domain for multiple conditions, and extracts a condition domain feature map through the feature extraction network for each condition domain.

[0022] The generator / decoder 230 generates a target object image that meets the condition using at least one of the object feature map and the condition domain feature map.

[0023] In this manner, the computing device 100 extracts at least one of an object feature map and a condition domain feature map through at least one of a plurality of encoders using object-related information, and generates a target object image suitable for the conditions using the object feature map and the condition domain feature map.

[0024] In step 120, the computing device 100 applies the background image to a depth estimation model to estimate the depth at each pixel location and generate a depth map.

[0025] This is explained in more detail with reference to FIG.

[0026] The depth estimation model includes a feature extraction module 310 and a feature synthesis module 320. The feature extraction module 310 is a DenseNet-169-based model. The feature extraction module 310, which is a DenseNet-169-based model, is capable of constructing a deep network by connecting all layers in a dense block, and is capable of extracting an enhanced feature map by combining a low-level feature map with a high-level feature map. In one embodiment of the present invention, it is assumed that the feature extraction module 310 is a DenseNet-169-based model, and the description will focus on this, however, other models (e.g., encoders) may also be applied.

[0027] The feature extraction module 310 receives a background image and then extracts a feature map as cognitive information through a DenseNet-169 based encoder.

[0028] The feature synthesis module 320 separates the input feature maps from the feature extraction module 310 and then generates a depth map for each pixel.

[0029] The feature synthesis module 320 applies a normalization layer to the feature maps input from the feature extraction module 310 to normalize them, and then applies a channel separation operation to separate each channel of the feature map. The feature synthesis module 320 then applies a multi-layer perceptron neural network model to the channel-separated feature maps to estimate a depth feature map for each pixel. Here, the multi-layer perceptron neural network model is a PointMLP. The feature synthesis module 320 applies a channel shuffling operation to the estimated depth feature map for each pixel and the channel-separated feature maps to mix them, and then combines them to output a final depth map.

[0030] In step 125, the computing device 100 determines the position and scale of the target object based on the target object image and the depth map, and then generates a regional object insertion image and a mask image, respectively.

[0031] FIG. 4 shows the process of generating a local object insertion image and a mask image.

[0032] In step 410, the calculation device 100 receives the target object image and the depth map, and then randomly determines a position (for convenience, referred to as an insertion position) where the object is to be inserted.

[0033] Next, in step 415, the calculation device 100 scales the size of the target object image (i.e., the target object) according to the depth value (depth value of the depth map) corresponding to the determined insertion position.

[0034] In step 420, the computing device 100 generates a regional object insertion image based on the scaled object and the insertion position. Here, the size of the regional object insertion image is equal to the background image or depth map. That is, the scaled target object image is inserted at the insertion position in the space corresponding to the background image to generate a regional object insertion image in which the target object is regionalized and scaled.

[0035] In step 425, the computing device 100 generates a mask image corresponding to the regional object-inserted image, where the mask image is an image in which the object and background in the regional object-inserted image are separated by a binary code, e.g., pixel values ​​in the object region have a first value and pixel values ​​in the background region have a second value.

[0036] Thus, the mask image contains information about the position and size at which the target object image is inserted (combined) in the background image, and is used as a learning label.

[0037] For example, assume that the target object image is as shown in Fig. 5. In this case, 610 in Fig. 6 indicates a depth map generated based on a background image. A regional object insertion image is generated by scaling the target object image to reflect the depth value into which the target object image of Fig. 5 is inserted, as shown in 620, and a mask image indicating the insertion position and size of the target object image in the regional object insertion image is generated as shown in 630.

[0038] 4, the target object image is assumed to be inserted at one position, but the process of FIG. 4 can be repeated at multiple insertion positions. Furthermore, it goes without saying that the target object at one position can be scaled to various sizes, not just one size.

[0039] That is, the calculation device 100 generates a plurality of local object insertion images and mask images by determining insertion positions and scaling so that target object images can be synthesized at various positions within a background image.

[0040] In step 130, the computing device 100 generates a simple composite image by simply combining the local object insertion image and the background image.

[0041] In step 135, the computing device 100 downsamples the simple composite image to a predetermined N-th scale, and then applies the learned multi-scale image analysis model to generate a composite image that reflects the scale characteristics of the background image.

[0042] The multi-scale image analysis model is trained in advance, but for ease of understanding and explanation, the training process of the multi-scale image analysis model will be first described with reference to FIG.

[0043] The multi-scale image analysis model is based on a generative adversarial model.

[0044] 7, the multi-scale image analysis model includes a plurality of scale image generation blocks (710a to 710n) and a plurality of discrimination blocks (720a to 720n). Here, among the scale image generation blocks and discrimination blocks, the scale image generation blocks and discrimination blocks that process images of the same scale are configured in pairs. Unless otherwise described below, the first scale image generation block 710a and the first discrimination block 720a operate as a pair, and the nth scale image generation block 710n and the nth discrimination block 720n operate as a pair.

[0045] In the following, a lower scale representation should be understood as a scale (resolution) one step lower than the image scale (resolution) processed in the current scale image generation block or discrimination block, and a higher scale representation should be understood as a scale (resolution) one step higher.

[0046] In addition, the resolution (scale) processed in the current processing block (e.g., scale image generation block or discrimination block) is referred to as the target scale in order to distinguish it from the lower and upper representations. Therefore, the target scale representation should be understood as the designated scale (resolution) of the block (e.g., scale image generation block or discrimination block) currently being processed.

[0047] As shown in FIG. 7, a multi-scale image analysis model according to an embodiment of the present invention includes a plurality of scale image generation blocks (710a to 710n) and a plurality of discrimination blocks (720a to 720n) that process images at different scales.

[0048] In the following, with reference to FIG. 7, we will explain how the multi-scale image analysis model uses a single image (eg, a background image) to allow each scale image generation block to learn target scale features.

[0049] According to one embodiment of the present invention, the plurality of scale image generating blocks operate starting from the lowest scale image generating block, then the higher scale image generating blocks operate in sequence, and so on up to the highest scale image generating block.

[0050] For ease of understanding and explanation, the smallest scale image generation block is referred to as an N-order scale image generation block 710a, and the highest scale image generation block is referred to as a 0-order scale image generation block 710n.

[0051] For training the multi-scale image analysis model, the N-th scale image generation block 710a generates an N-th scale random noise image After TIFF2025538325000008.tif9128 is input, the Nth generated image is similar to the background image. TIFF2025538325000009.tif9128 is generated. Next, the calculation device 100 generates an Nth generated image TIFF2025538325000010.tif9128 is upsampled by r times and transmitted to a higher scale image generation block (for example, the N-1 order scale image generation block 710b).

[0052] Each of the remaining scale image generation blocks (710b to 710n) other than the N-th scale image generation block includes a context attention block 712, a random mask block 714, and a generator block 716, as shown in FIG.

[0053] The context attention block 712 receives the image that is upsampled r times as the output result of the lower scale image generation block, divides it into clusters without pixel overlap, and then applies self-attention calculation to extract cluster-based feature maps.

[0054] More specifically, the context attention block 712 converts the output of the subscale image generation block into an r-times upsampled image in the normalized coordinate space After converting to TIFF2025538325000011.tif17128, each pixel becomes a data point Convert to TIFF2025538325000012.tif12128, where TIFF2025538325000013.tif9128 indicates the input feature dimension, and 2 indicates the pixel coordinate.

[0055] The context attention block 712 then applies a Simple Linear Iterative Clustering (SLIC) algorithm to divide the pixels into non-overlapping clusters. Then, the context attention block 712 applies a convolutional layer within each cluster to embed the pixels into a feature space, and then clusters them using the K-Nearest Neighbor (KNN) algorithm, after which the t-th superpixel Center vector of TIFF2025538325000014.tif9128 After setting TIFF2025538325000015.tif8128, feature aggregation is performed on M clustered samples, and the aggregated center vectors are TIFF2025538325000016.tif9128 is calculated. In addition, the context attention block 712 applies self-attention calculation to perform attention calculation between each crowd, and then uses dispatch calculation to transmit the features of the representative vector to each crowd as an element-wise sum and output a crowd-based feature map.

[0056] The random mask block 714 is located at the end of the context attention block 712. It inputs the crowd-based feature map output from the context attention block 712 and extracts a randomly masked feature map through patch embedding and random masking.

[0057] The output of the random mask block 714 and the target scale random noise image are combined and transmitted to the generator block 716. The generator block 716 receives the combined image of the output of the random mask block 714 and the target scale random noise image, extracts a target scale feature map, and performs training.

[0058] The target-scale feature map is weighted through the SE block and spatial-attention block to focus on the channel and regionally important features of the target-scale feature map. Here, the SE block includes two processes: Squeeze (compression) and Excitation (realignment). The feature map that has passed through the SE block and spatial-attention block is added to the image that is upsampled r times from the output of the lower-scale image generation block to learn learnable features at the target scale.

[0059] This method is repeated from the n-1th scale image generation block to the 0th scale image generation block, so that the generator block at each scale can learn the global structural features of the image at the smallest size, and the closer to the 0th scale, the more detailed the features can be learned.

[0060] A multi-scale image analysis model according to one embodiment of the present invention has image generation blocks for different scales, whereby features for different scales are pre-trained using a single background image.

[0061] To recap, the scale image generation blocks (710b and 710n) receive the image generated by the lower scale image generation block after upsampling it by r times, divide it into clusters without pixel overlap, apply self-attention calculation to extract cluster-based feature maps, and then generate a target scale image similar to the input background image using the feature maps randomly masked through the random mask block. This is repeated until the highest scale image (0th scale image) is generated.

[0062] In addition, the background image input for training the zero-order scale generation block 710n is TIFF2025538325000017.tif8128 and 0th order scale image TIFF2025538325000018.tif10128, and optimizes the weighting of the zeroth-order generator block 716n so that the paired discrimination block 720n cannot distinguish between the input background image and the generated image. Similarly, the remaining discrimination blocks (720n-1, 720n-2, ..., 720a) optimize the weighting of the generator blocks in each paired target scale image generation block so that they cannot distinguish between the downsampled input background image and the generated image.

[0063] The process of training a multi-scale image analysis model based on an input background image has been described with reference to FIGS.

[0064] 7 and 8 show the operation of training a multi-scale image analysis model using a single image (background image), and a method of generating a synthesized image using the trained multi-scale image analysis model will be described with reference to FIG. 9. In a multi-scale image analysis model that has completed training, some components used in the training process may be deactivated and not operate, as shown in FIG. 9. For example, the lowest scale image generation block 710a, the random mask blocks 714a to 714n in the remaining scale image generation blocks, and the lowest scale discrimination block 720a are used only in the training process, and after training is completed, they are removed or deactivated and are not used to generate a synthesized image.

[0065] The simple composite image is downsampled to a predetermined smallest scale (for convenience, N-th scale) and then transmitted to the target scale image generation block. In one embodiment of the present invention, it is assumed that the simple composite image is downsampled to N-th scale, and the description will focus on this, but the simple composite image is scaled randomly from 0th to N-th scales.

[0066] Assuming that the simple composite image is downsampled to the Nth scale, the image is input to a higher scale image generation block (e.g., an N-1th scale image generation block) than the image downsampled to the Nth scale. As a result, the context attention block located in the target scale image generation block 710b divides the input image into clusters without pixel overlap, then applies self-attention operation, transmits the features of the representative vector to each cluster by element-by-element sum through dispatch operation, extracts cluster-based feature maps, and transmits them to the generator child block.

[0067] The generator block reflects the target scale features learned for the background image in the crowd-based feature map to generate a target scale composite image, which is then upsampled r times and transmitted to the higher scale image generation block, where the above-mentioned series of processes are repeated until a zero-order scale composite image is generated.

[0068] Also referring to FIG. 1, at step 140, the computing device 100 generates a final training image using the background image, the final composite image, and the mask image.

[0069] FIG. 10 is a diagram illustrating an internal configuration of a computing device for generating deep learning model training data according to an embodiment of the present invention.

[0070] Referring to FIG. 10, a computing device 100 for generating deep learning model training data according to an embodiment of the present invention includes a memory 1010 and a processor 1020.

[0071] The memory 1010 stores at least one instruction word.

[0072] The processor 1020, in conjunction with the memory 1010, executes commands stored in the memory 1010. The commands executed by the processor 1020 input a background image and object-related information, generate a target object image that meets conditions based on the object-related information, generate a depth map by estimating the depth of each pixel position by applying the background image to a depth estimation model, generate a regional insertion object image and a mask image by determining an insertion position and scale at which the target object image is inserted based on the depth map, simply combine the regional insertion object image and the background image, generate a composite image that reflects the scale characteristics of the background image through a learned multi-scale image analysis model, and generate a final learning image using the background image, composite image, and mask image. This is the same as the process described with reference to FIGS. 1 to 9, so repeated description will be omitted.

[0073] Apparatus and methods according to embodiments of the present invention may be embodied in the form of program instructions executed by various computer means and recorded on a computer-readable recording medium. Computer-readable recording media may include, alone or in combination, program instructions, data files, data structures, and the like. The program instructions recorded on the computer-readable recording medium may be specially designed and constructed for the present invention, or may be well known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include not only machine language code, such as produced by a compiler, but also high-level language code executed by a computer using an interpreter, for example.

[0074] The hardware devices described above are configured to operate with at least one software module to perform the operations of the present invention, and vice versa.

[0075] The present invention has been described above with reference to its preferred embodiments. Those skilled in the art will recognize that the present invention can be embodied in various modified forms without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments should be considered in an illustrative rather than a restrictive sense. The scope of the present invention is defined by the claims, not the above description, and all variations within the scope of the claims should be construed as being within the scope of the present invention. [Explanation of symbols]

[0076] 100 Computing equipment 210, 220 1st and 2nd encoders 230 Generative Decoder 310 Feature Extraction Module (DenseNet-169 based model) 320 Feature Synthesis Module 610 Depth Map 620 Regional Object Insertion Image 630 Mask Images 710a First scale image generation block (Nth scale image generation block) 710n nth scale image generation block (0th scale image generation block) 712 Contextual Attention Block 714 Random Mask Block 716 Generator Blocks 720a First discrimination block 720n nth discrimination block 1010 memory 1020 processor

Claims

1. A method for generating artificial intelligence image analysis model learning data, comprising: (a) inputting background image and object-related information; (b) generating a target object image that meets a condition based on the object-related information; (c) applying the background image to a depth estimation model to estimate depth at each pixel location to generate a depth map; (d) determining an insertion position and a scale at which the target object image is inserted based on the depth map, and generating a regional insertion object image and a mask image, respectively; (e) simply combining the regionally inserted object image and the background image, and then generating a combined image that reflects the scale characteristics of the background image through a trained multi-scale image analysis model; (f) generating a final training image using the background image, the composite image, and the mask image.

2. The object-related information includes an object image and a description of the object, The method of claim 1 , wherein the description includes a descriptive prompt and at least one condition for the object.

3. 2. The method of claim 1, wherein the depth estimation model extracts a background feature map for the background image, normalizes the background feature map by applying a normalization layer, and then separates the channels by applying a channel splitting operation.

3. The method of claim 1, wherein the depth estimation model estimates a per-pixel depth feature map by applying a multi-perceptron neural network model to the channel-separated background feature map.

4. The method of claim 2, wherein the depth estimation model generates a final depth map by combining the estimated per-pixel depth feature map and the channel-separated background feature map by applying a channel shuffling operation to the combined feature map.

4. The step (d) determining an insertion position where the target object image is inserted into a space corresponding to the background image; scaling the target object image according to a depth value corresponding to the insertion position based on the depth map; generating a local object insertion image by inserting the scaled target object image at the insertion position in the space corresponding to the background image; and generating the mask image using the local object inset image.

5. The method of claim 4 , wherein the mask image includes information about the size and position of the target object image to be inserted into the background image.

6. 2. The method of claim 1, wherein step (d) is repeated with different insertion positions and different scales.

7. The multi-scale image analysis model comprises: A plurality of scale image generation blocks, which input an image that has been upsampled r times as the generated image generated by the lower scale image generation block, divide the image into groups without pixel overlap, apply self-attention operation to extract group-based feature maps, and apply a random mask to generate scale generated images; a plurality of discrimination blocks for adjusting weight values ​​of generator blocks located in the scaled image generation block so as to distinguish between a background image downsampled r times and a scaled generated image; The method of claim 1 , wherein the multi-scale image analysis model is pre-trained using the background image.

8. 8. The method according to claim 7, wherein the plurality of scale image generating blocks include image generating blocks and discrimination blocks for different scales, and an image generating block and a discrimination block for the same scale are paired.

9. The N-th order scale image generation block having the lowest resolution among the plurality of scale image generation blocks is an N-th order random noise image After inputting, the Nth generated image similar to the background image Generate The 0th-order discrimination block having the highest resolution among the plurality of discrimination blocks is used to obtain the background image and the 0th-order scale generated image without downsampling.

8. The method of claim 7, wherein the weights of the generator blocks in the zeroth scale image generator block are adjusted using:

10. Each of the plurality of scale image generation blocks One step lower scale generated image After inputting, a simple linear iterative clustering algorithm is applied to divide the image into clusters without pixel overlap, a convolution operation is applied within each cluster to embed the image into a feature space, and a K-nearest neighbor algorithm is applied to cluster the image, and then a central vector of the t-th superpixel is set. Features are aggregated by the number of clustered samples, and the aggregated central vector is calculated. A self-attention operation is performed, and the features of the aggregated central vector are added to each cluster element by a dispatch operation to extract a cluster unit feature map. A context attention block; a random mask block located at the rear end of the context attention block, which receives the crowd-based feature map output from the context attention block and extracts a randomly masked feature map by patch embedding and random masking; The output of the random mask block is a sum of the masked feature map and the target scale random noise image, and then the target scale feature map is extracted. After that, weights are assigned to the important features of each channel and region, and the next lower scale image is generated. and a target scale generator block that adds:

11. The step (e) comprises: The simply synthesized image is downsampled to a predetermined lowest scale, that is, an N-th scale, and input to the trained multi-scale image analysis model; The trained multi-scale image analysis model receives the image downsampled to the N-th scale, divides the image into crowds without pixel overlap, applies a self-attention operation to extract a target-scale crowd-unit feature map, and then generates a target-scale synthetic image that reflects the target-scale features pre-trained on the background image; 2. The method of claim 1, wherein the multi-scale image analysis model repeats the process of generating a synthetic image in which the target scale synthetic image is upsampled by a factor r until the scale of the target scale synthetic image is equal to that of the background image.

12. A computer-readable recording medium having recorded thereon program code for carrying out the method according to any one of claims 1 to 11.

13. a memory for storing at least one instruction word; a processor for executing instructions stored in the memory; Each of the instructions executed by the processor: (a) inputting background image and object-related information; (b) generating a target object image that meets a condition based on the object-related information; (c) applying the background image to a depth estimation model to estimate depth at each pixel location to generate a depth map; (d) determining an insertion position and a scale at which the target object image is inserted based on the depth map, and generating a regional insertion object image and a mask image, respectively; (e) simply combining the regionally inserted object image and the background image, and then generating a combined image that reflects the scale characteristics of the background image through a trained multi-scale image analysis model; (f) generating a final training image using the background image, the composite image, and the mask image.

14. 14. The computing device of claim 13, wherein the depth estimation model extracts a background feature map for the background image, normalizes the background feature map by applying a normalization layer, and then separates the channels by applying a channel splitting operation, estimates a per-pixel depth feature map by applying a multi-perceptron neural network model to the channel-separated background feature map, and generates a final depth map by combining the estimated per-pixel depth feature map and the channel-separated background feature map by applying a channel shuffling operation.

15. The step (d) determining an insertion position where the target object image is inserted into a space corresponding to the background image; scaling the target object image according to a depth value corresponding to the insertion position based on the depth map; generating a local object insertion image by inserting the scaled target object image at the insertion position in the space corresponding to the background image; and generating the mask image using the local object insertion image.

16. The multi-scale image analysis model comprises: A plurality of scale image generation blocks, which input an image that has been upsampled r times as the generated image generated by the lower scale image generation block, divide the image into groups without pixel overlap, apply self-attention operation to extract group-based feature maps, and apply a random mask to generate scale generated images; a plurality of discrimination blocks for adjusting weight values ​​of generator blocks located in the scaled image generation block so as to distinguish between a background image downsampled r times and a scaled generated image; the multi-scale image analysis model is pre-trained using the background image; 14. The arithmetic device according to claim 13, wherein the plurality of scale image generating blocks include image generating blocks and discrimination blocks for different scales, and an image generating block and a discrimination block for the same scale are configured as a pair.

17. The N-th order scale image generation block having the lowest resolution among the plurality of scale image generation blocks is an N-th order random noise image After inputting, the Nth generated image similar to the background image Generate The 0th-order discrimination block having the highest resolution among the plurality of discrimination blocks is used to obtain the background image and the 0th-order scale generated image without downsampling.

17. The arithmetic unit according to claim 16, wherein the weights of the generator blocks in the 0th-order scale image generator block are adjusted using:

18. Each of the plurality of scale image generation blocks One step lower scale generated image After inputting, a simple linear iterative clustering algorithm is applied to divide the image into clusters without pixel overlap, a convolution operation is applied within each cluster to embed the image into a feature space, and a K-nearest neighbor algorithm is applied to cluster the image, and then a central vector of the t-th superpixel is set. Features are aggregated by the number of clustered samples, and the aggregated central vector is calculated. A self-attention operation is performed, and the features of the aggregated central vector are added to each cluster element by a dispatch operation to extract a cluster unit feature map. A context attention block; a random mask block located at the rear end of the context attention block, which receives the crowd-based feature map output from the context attention block and extracts a randomly masked feature map by patch embedding and random masking; The output of the random mask block is a sum of the masked feature map and the target scale random noise image, and then the target scale feature map is extracted. After that, weights are assigned to the important features of each channel and region, and the next lower scale image is generated. and a target scale generator block that adds:

19. The step (e) comprises: The simply synthesized image is downsampled to a predetermined lowest scale, that is, an N-th scale, and input to the trained multi-scale image analysis model; The trained multi-scale image analysis model receives the image downsampled to the N-th scale, divides the image into crowds without pixel overlap, applies a self-attention operation to extract a target-scale crowd-unit feature map, and then generates a target-scale synthetic image that reflects the target-scale features pre-trained on the background image; The computing device of claim 13, wherein the multi-scale image analysis model repeatedly performs a process of generating a synthetic image in which the target scale synthetic image is upsampled r times until the scale of the target scale synthetic image becomes equal to that of the background image.

Images