Method for generating training data using generative model

Description

How to generate training data using generative models

[0001] The present disclosure relates to a method for generating learning data, and more particularly, to a method for generating learning data using a generative model.

[0002] Facial recognition technology automatically detects faces in images and analyzes them to identify individuals. Typically, the facial recognition process involves extracting a face from the entire image and cropping it. The cropped face image is then adjusted for size, angle, and other factors, and feature information is extracted from the aligned face image.

[0003] As artificial intelligence technology advances, face detection and feature extraction are performed using deep learning-based algorithms. Examples include face detection models like AlexNet and a fine-tuned version of YOLO, while feature extraction models include RetinaFace, FaceNet, and GroupFace.

[0004] Meanwhile, deep learning-based models prioritize the quantity and quality of training data. However, images are susceptible to the effects of lighting and the surrounding environment, which can result in images with high contrast or blurry images. Features extracted from these images may be inaccurate.

[0005] In particular, facial images may include side images captured from various angles in addition to frontal images. Because key facial features may be partially exposed or distorted in side images, features extracted from side images may also be inaccurate.

[0006] Korean Patent No. 10-2353837 (registration date: January 17, 2022) discloses a method for restoring high-resolution facial images.

[0007] The present disclosure relates to a method for generating learning data capable of accurate feature extraction by utilizing a generative model.

[0008] Meanwhile, the technical task to be achieved by the present disclosure is not limited to the technical task mentioned above, and may include various technical tasks within a scope obvious to a person skilled in the art from the contents described below.

[0009] To address the aforementioned challenges, a method for modeling the location of an item by analyzing a plurality of images, performed by a computing device, is disclosed. The method may include the steps of: acquiring a first type of image; detecting a plurality of feature points from the first type of image; acquiring context information based on the plurality of feature points; and utilizing a generative model to generate a second type of image based on the context information.

[0010] In one embodiment, the step of detecting a plurality of feature points from the first type image may include the steps of detecting a plurality of landmarks from the first type image, calculating a rotation angle of a face from the first type image, adjusting a position of at least one landmark using the rotation angle of the face, and detecting the plurality of feature points based on at least one landmark whose position has been adjusted.

[0011] In one embodiment, the method may further include a step of labeling the second type of image based on information labeled in the first type of image.

[0012] In one embodiment, the step of generating the second type of image includes the step of generating the second type of image based on the context information and text information by utilizing the generation model, wherein the text information may include information for changing a class labeled in the first type of image.

[0013] In one embodiment, the step of labeling the second type of image may include the step of labeling the second type of image based on information labeled in the first type of image and the text information.

[0014] In one embodiment, the step of detecting a plurality of feature points from the first type image may include the steps of detecting a plurality of landmarks from the first type image, the step of adjusting the position of at least one landmark corresponding to an area of ​​interest among the plurality of landmarks to obtain a plurality of transformed landmarks, and the step of outputting the plurality of transformed landmarks as the plurality of feature points.

[0015] In one embodiment, the step of obtaining the plurality of transformed landmarks may include the step of obtaining the plurality of landmarks classified into each of the plurality of face regions from the first type of image, and the step of adjusting the position of at least one landmark of a selected region among the plurality of face regions.

[0016] In one embodiment, the step of labeling the second type of image may include the step of classifying a facial expression based on the plurality of transformed landmarks using a first classification model, and the step of labeling the second type of image based on information labeled in the first type of image and the classification result.

[0017] In one embodiment, the step of labeling the second type of image may further include the steps of classifying facial expressions based on the second type of image using a second classification model, comparing a first classification result, which is a result of classifying facial expressions based on the transformed landmarks, with a second classification result, which is a result of classifying facial expressions based on the second type of image, selecting one of the first classification result and the second classification result based on a result of the comparison, and labeling the second type of image based on the selected classification result.

[0018] In one embodiment, the step of selecting one of the first classification result and the second classification result based on the result of the comparison may include the step of comparing the reliability of the first classification result and the reliability of the second classification result, and the step of selecting one of the first classification result and the second classification result based on the result of the comparison.

[0019] A computer program stored in a computer-readable storage medium is disclosed to solve the above-described problem. When the computer program is executed by one or more processors, the one or more processors perform the following operations, which may include: acquiring a first type of image; detecting a plurality of feature points from the first type of image; acquiring context information based on the plurality of feature points; and generating a second type of image based on the context information using a generative model.

[0020] In addition, a computing device is disclosed for solving the aforementioned problem. The computing device includes at least one processor and a memory, and the at least one processor may be configured to acquire a first type of image, detect a plurality of feature points from the first type of image, acquire context information based on the plurality of feature points, and generate a second type of image based on the context information using a generation model.

[0021] The present disclosure has the effect of improving the quantity and quality of learning data by generating learning data capable of accurate feature extraction using a generative model.

[0022] Meanwhile, the effects of the present disclosure are not limited to the effects mentioned above, and various effects may be included within a range apparent to those skilled in the art from the contents described below.

[0023] FIG. 1 is a block diagram of a computing device performing operations according to one embodiment of the present disclosure.

[0024] FIG. 2 illustrates an exemplary structure of an artificial intelligence-based model according to one embodiment of the present disclosure.

[0025] FIG. 3 is a block diagram of a learning data generation device according to an embodiment of the present disclosure.

[0026] FIG. 4 is a drawing for explaining a method for detecting feature points according to an embodiment of the present disclosure.

[0027] FIG. 5 is a diagram for explaining a method for detecting feature points according to another embodiment of the present disclosure.

[0028] FIG. 6 is a block diagram of a learning data generation device according to an additional embodiment of the present disclosure.

[0029] FIG. 7 is a flowchart illustrating a method for generating learning data according to an embodiment of the present disclosure.

[0030] FIG. 8 is a simplified, general schematic diagram of an exemplary computing environment in which embodiments of the present disclosure may be implemented.

[0031] Various embodiments are now described with reference to the drawings. In this disclosure, various descriptions are provided to facilitate understanding of the present disclosure. However, it will be apparent that these embodiments can be practiced without these specific descriptions.

[0032] The terms "component," "module," "system," and the like, as used herein, refer to computer-related entities, hardware, firmware, software, a combination of software and hardware, or an execution of software. For example, a component may be, but is not limited to, a procedure running on a processor, a processor, an object, a thread of execution, a program, and / or a computer. For example, both an application running on a computing device and the computing device may be a component. One or more components may reside within a processor and / or a thread of execution. A component may be localized within a single computer. A component may be distributed between two or more computers. Furthermore, these components may execute from various computer-readable media having various data structures stored therein. Components may communicate via local and / or remote processes, for example, by signals comprising one or more data packets (e.g., data from one component interacting with another component in a local system, a distributed system, and / or data transmitted to another system via a network such as the Internet via signals).

[0033] Furthermore, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless otherwise specified or clear from context, "X utilizes A or B" is intended to mean one of its natural inclusive permutations. That is, if X utilizes A; X utilizes B; or X utilizes both A and B, "X utilizes A or B" can apply to any of these cases.

[0034] Additionally, the terms "comprises" and / or "comprising" should be understood to imply the presence of the features and / or components. However, it should be understood that the terms "comprises" and / or "comprising" do not exclude the presence or addition of one or more other features, components, and / or groups thereof. Furthermore, unless otherwise specified or clear from context to refer to the singular form, the singular in the present disclosure and claims should generally be construed to mean "one or more."

[0035] And, the term "at least one of A or B" should be interpreted to mean "if it includes only A", "if it includes only B", or "if it is combined in the composition of A and B".

[0036] Those skilled in the art should further appreciate that the various illustrative logical blocks, configurations, modules, circuits, means, logics, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate the interchangeability of hardware and software, various illustrative components, blocks, configurations, means, logics, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application. However, such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

[0037] The description of the disclosed embodiments is provided to enable a person skilled in the art to make or use the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art. The general principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Thus, the present disclosure is not limited to the embodiments set forth herein. The present disclosure is to be construed in the widest scope consistent with the principles and novel features disclosed herein.

[0038]

[0039] FIG. 1 is a block diagram of a computing device performing operations according to one embodiment of the present disclosure.

[0040] The configuration of the computing device (100) illustrated in FIG. 1 is merely a simplified example. In one embodiment of the present disclosure, the computing device (100) may include other configurations for performing the computing environment of the computing device (100), and only some of the disclosed configurations may constitute the computing device (100).

[0041] A computing device (100) may include a processor (110), memory (130), and network unit (150).

[0042] The processor (110) may be configured with one or more cores, and may include a processor for data analysis and deep learning, such as a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), and a tensor processing unit (TPU) of a computing device. The processor (110) may read a computer program stored in the memory (130) and perform data processing for machine learning according to an embodiment of the present disclosure. According to an embodiment of the present disclosure, the processor (110) may perform operations for learning a neural network model. The processor (110) may perform calculations for learning a neural network model, such as processing input data for learning in deep learning (DL), extracting features from input data, calculating errors, and updating weights of a neural network model using backpropagation. At least one of the CPU, GPGPU, and TPU of the processor (110) may process learning of the neural network model. For example, a CPU and a GPGPU can work together to train a neural network model and classify data using the neural network model. Furthermore, in one embodiment of the present disclosure, processors of multiple computing devices can be used together to train a neural network model and classify data using the neural network model. Furthermore, a computer program executed on a computing device according to one embodiment of the present disclosure may be a CPU, GPGPU, or TPU executable program.

[0043]

[0044] A computing device (100) according to one embodiment of the present disclosure may be a learning data generation device. The learning data generation device can receive low-quality images and generate high-quality images. Furthermore, the learning data generation device can automatically label the generated images. Therefore, the learning data generation device can effectively improve the quantity and quality of learning data.

[0045]

[0046] According to one embodiment of the present disclosure, the memory (130) can store any form of information generated or determined by the processor (110) and any form of information received by the network unit (150).

[0047] According to one embodiment of the present disclosure, the memory (130) may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, and an optical disk. The computing device (100) may also operate in relation to web storage that performs the storage function of the memory (130) on the internet. The description of the above-described memory is merely an example, and the present disclosure is not limited thereto.

[0048] The network unit (150) according to one embodiment of the present disclosure can use various wired communication systems such as a public switched telephone network (PSTN), xDSL (x Digital Subscriber Line), RADSL (Rate Adaptive DSL), MDSL (Multi Rate DSL), VDSL (Very High Speed ​​DSL), UADSL (Universal Asymmetric DSL), HDSL (High Bit Rate DSL), and a local area network (LAN).

[0049] In addition, the proposed network unit (150) according to one embodiment of the present disclosure can use various wireless communication systems such as CDMA (Code Division Multi Access), TDMA (Time Division Multi Access), FDMA (Frequency Division Multi Access), OFDMA (Orthogonal Frequency Division Multi Access), SC-FDMA (Single Carrier-FDMA) and other systems.

[0050] In one embodiment, the network unit (150) may be configured regardless of the communication mode, such as wired or wireless, and may be configured as various communication networks, such as a personal area network (PAN) and a wide area network (WAN). In addition, the network may be the well-known World Wide Web (WWW), and may also utilize a wireless transmission technology used for short-distance communication, such as infrared (IrDA) or Bluetooth. The technologies described in the present disclosure may be used not only in the networks mentioned above but also in other networks.

[0051]

[0052] FIG. 2 illustrates an exemplary structure of an artificial intelligence-based model according to one embodiment of the present disclosure.

[0053] Throughout this specification, the terms artificial intelligence model, artificial intelligence-based model, computational model, neural network, network function, and neural network may be used interchangeably.

[0054] A neural network can be composed of a set of interconnected computational units, generally referred to as nodes. These nodes can also be referred to as neurons. A neural network consists of at least one node. The nodes (or neurons) that make up a neural network can be interconnected by one or more links.

[0055] Within a neural network, one or more nodes connected via links can form a relationship between input nodes and output nodes. The concept of input nodes and output nodes is relative, meaning that any node that is in an output node relationship with one node can also be in an input node relationship with another node, and vice versa. As described above, the relationship between input nodes and output nodes can be created based on links. One input node can be connected to one or more output nodes via links, and vice versa.

[0056] In a relationship between input nodes and output nodes connected through a single link, the data of the output node can have its value determined based on the data input to the input node. Here, the link interconnecting the input nodes and output nodes can have a weight. The weight can be variable and can be varied by the user or an algorithm so that the neural network can perform a desired function. For example, when one or more input nodes are interconnected to one output node through each link, the output node can determine the output node value based on the values ​​input to the input nodes connected to the output node and the weight set on the link corresponding to each input node.

[0057] As described above, a neural network is a network in which one or more nodes are interconnected through one or more links, forming input and output node relationships within the network. The characteristics of a neural network can be determined based on the number of nodes and links within the network, the relationships between the nodes and links, and the weights assigned to each link. For example, if two neural networks have the same number of nodes and links but different weight values ​​for the links, the two neural networks can be perceived as different from each other.

[0058] A neural network can be composed of a set of one or more nodes. A subset of the nodes comprising the neural network can form a layer. Some of the nodes comprising the neural network can form a layer based on their distances from the initial input node. For example, a set of nodes that are n distances from the initial input node can form n layers. The distance from the initial input node can be defined by the minimum number of links required to reach the node from the initial input node. However, this definition of a layer is arbitrary for illustrative purposes, and the degree of a layer within a neural network can be defined in a different way than described above. For example, a layer of nodes can be defined by its distance from the final output node.

[0059] In one embodiment of the present disclosure, a set of neurons or nodes may be defined as a layer.

[0060] An initial input node may refer to one or more nodes within a neural network into which data is directly input without going through links with other nodes. Alternatively, within a neural network, it may refer to nodes that do not have other input nodes connected by links in the relationship between nodes based on links. Similarly, a final output node may refer to one or more nodes within a neural network that do not have output nodes in their relationship with other nodes. Furthermore, a hidden node may refer to nodes that constitute a neural network other than the initial input node and the final output node.

[0061] A neural network according to one embodiment of the present disclosure may be a neural network in which the number of nodes in an input layer may be the same as the number of nodes in an output layer, and the number of nodes decreases and then increases as it progresses from the input layer to the hidden layer. In addition, a neural network according to another embodiment of the present disclosure may be a neural network in which the number of nodes in an input layer may be less than the number of nodes in an output layer, and the number of nodes decreases as it progresses from the input layer to the hidden layer. In addition, a neural network according to another embodiment of the present disclosure may be a neural network in which the number of nodes in an input layer may be greater than the number of nodes in an output layer, and the number of nodes increases as it progresses from the input layer to the hidden layer. A neural network according to another embodiment of the present disclosure may be a neural network in a combined form of the neural networks described above.

[0062] An AI-based model according to one embodiment of the present disclosure may include a deep neural network (DNN). A DNN may refer to a neural network that includes multiple hidden layers in addition to an input layer and an output layer. Using a DNN, it is possible to identify latent structures in data. That is, the latent structures of a photo, text, video, voice, protein sequence structure, gene sequence structure, peptide sequence structure, music (e.g., what objects are in a photo, what the content and emotion of a text are, what the content and emotion of a voice are, etc.), and / or the binding affinity between a peptide and MHC can be identified. Deep neural networks may include convolutional neural networks (CNNs), recurrent neural networks (RNNs), autoencoders (AEs), variational autoencoders (VAEs), restricted boltzmann machines (RBMs), deep belief networks (DBNs), Q-networks, U-networks, Siamese networks, generative adversarial networks (GANs), transformers, and the like. The description of the above-described deep neural networks is merely an example and the present disclosure is not limited thereto.

[0063] The artificial intelligence-based model of the present disclosure can be represented by a network structure of any structure described above, including an input layer, a hidden layer, and an output layer.

[0064] The neural network that can be used in the artificial intelligence-based model of the present disclosure may be trained using at least one of supervised learning, unsupervised learning, semi-supervised learning, transfer learning, active learning, or reinforcement learning. Training of the neural network may be a process of applying knowledge to the neural network to perform a specific action.

[0065] Neural networks can be trained to minimize output errors. This process involves repeatedly inputting training data into the neural network, calculating the neural network output and target error for the training data, and backpropagating the neural network error from the output layer to the input layer to update the weights of each node in the neural network to reduce the error. In supervised learning, training data with the correct answer for each training data is used (i.e., labeled training data). In unsupervised learning, the correct answer may not be labeled for each training data. For example, in supervised learning for data classification, the training data may be data with each category labeled. Labeled training data is input to the neural network, and the error can be calculated by comparing the output (category) of the neural network with the training data labels. Alternatively, in unsupervised learning for data classification, the error can be calculated by comparing the input training data with the neural network output. The calculated error is backpropagated in the neural network in the backward direction (i.e., from the output layer to the input layer), and the connection weights of each node in each layer of the neural network can be updated according to the backpropagation. The amount of change in the connection weights of each node to be updated can be determined by the learning rate. The neural network's calculation of the input data and the backpropagation of the error can constitute a learning cycle (epoch). The learning rate can be applied differently depending on the number of iterations of the neural network's learning cycle. For example, a high learning rate can be used in the early stages of neural network training to quickly achieve a certain level of performance, thereby increasing efficiency. A lower learning rate can be used in the later stages of training to increase accuracy.

[0066] In neural network training, training data can typically be a subset of real-world data (i.e., the data to be processed using the trained neural network). Therefore, there can be a learning cycle where errors on the training data decrease but errors on the real-world data increase. Overfitting is a phenomenon where excessive training on the training data leads to increased errors on the real-world data. For example, a neural network trained on yellow cats may fail to recognize cats when shown non-yellow colors, a type of overfitting. Overfitting can increase errors in machine learning algorithms. Various optimization methods can be used to prevent overfitting. These methods include increasing the training data, regularization, dropout, which disables some nodes in the network during the learning process, and the use of batch normalization layers.

[0067]

[0068] FIG. 3 is a block diagram of a learning data generation device according to an embodiment of the present disclosure.

[0069] Referring to FIG. 3, the learning data generation device (100A) may be a device that receives a first type of image and generates a second type of image. The first type of image may be a low-quality image. A low-quality image may refer to an image from which accurate feature extraction is difficult. In other words, the accuracy of a model performing a specific task may be reduced.

[0070] For example, low-quality images may include images with high contrast, blurry images, images with significant distortion, and images of the side of a face. A second type of image may refer to a high-quality image. A high-quality image may refer to an image from which relatively accurate feature extraction is possible. In other words, the accuracy of a model performing a specific task may be relatively high.

[0071] Additionally, the learning data generation device (100A) can automatically label the second type of image. Therefore, the learning data generation device (100A) has the effect of improving the quantity and quality of learning data.

[0072] A learning data generation device (100A) may include a feature point detection model (110), a first encoder (120), a neural network model (130), a decoder (140), and a second encoder (170). The neural network model (130), the decoder (140), and the second encoder (170) may be an example for implementing a generation model.

[0073] The feature point detection model (110) can acquire a first type of image (IMG1). As illustrated in FIG. 3, the first type of image may be a side view image of a face or a blurred image. According to embodiments, the first type of image may be an image obtained by detecting a face region from an entire image, cropping the face region from the entire image, and then resizing the image.

[0074] The feature point detection model (110) can detect a plurality of feature points from a first type image using a landmark detection algorithm. The landmark detection algorithm can be a statistical model such as an Active Contour Model (ACM), an Active Shape Model (ASM), or a deep learning-based model such as a Cascade-based CNN, an AutoEncoder, or a Retina-Face.

[0075] The plurality of feature points output from the feature point detection model (110) may be reconstructed from the plurality of landmarks detected from the first type of image (IMG1). For example, if the first type of image (IMG1) is a side image of a face, the plurality of feature points output from the feature point detection model (110) may be for a frontal image of the face corresponding to the side image of the face.

[0076] The first encoder (120) can receive a plurality of feature points from the feature point detection model (110). The encoder (120) can obtain context information based on the plurality of feature points.

[0077] The generative model can generate a second type of image (IMG2) based on the context information. For example, the generative model can be a model in which a ControlNet module is applied to a Stable Diffusion model. Specifically, the neural network model (130) can be a model in which a ControlNet module is applied to a Diffusion model. The ControlNet module can be a module for learning new conditions while preserving the semantics of a pre-trained Stable Diffusion model. Accordingly, the ControlNet module can adjust the generative model so that the generative model can utilize additional input conditions. According to one embodiment of the present disclosure, the generative model can generate an image by utilizing the context information output from the first encoder (120) as an additional input condition.

[0078] In addition, generative models can be, but are not limited to, Conditional GAN-based models and StyleGAN-based models.

[0079] Additionally, the generative model can acquire text information as additional conditional information. The second encoder (170) may be a CLIP (Contrastive Language-Image Pre-training) model. The text information may be information for changing the class labeled in the first type image (IMG1). For example, the first type image (IMG1) may be data labeled with multiple classes, such as female for gender, black for hair color, smile for expression, and oval for face shape. In this case, the text information may be information for outputting the hair color as blue. In other words, the text information may be information for changing the hair color class among the classes labeled in the first type image (IMG1).

[0080] The generative model can generate a second type of image (IMG2) based on the context information and text information by utilizing the decoder (140). For example, when the first type of image (IMG1) is a side image of a face, the second type of image (IMG2) can be an image in which the first type of image (IMG1) is converted into a frontal image and then the hair color is changed to blue.

[0081] The learning data generation device (100A) can automatically label a second type image (IMG2) based on the labeled information in the first type image (IMG1). If no text information is input to the generation model, the second type image (IMG2) can be automatically labeled with the same labeling information as the first type image (IMG1).

[0082] When text information is input into the generative model, the learning data generation device (100A) can label a second type of image (IMG2) based on the information labeled in the first type of image (IMG1) and the text information. In an embodiment where the text information is information indicating that the hair color should be output as blue, the second type of image (IMG2) can be labeled with multiple classes, such as a gender of female, a hair color of blue, an expression of smile, and a face shape of oval.

[0083]

[0084] FIG. 4 is a drawing for explaining a method for detecting feature points according to an embodiment of the present disclosure.

[0085] Referring to FIG. 4, the feature point detection model (110A) may include a landmark detection unit (111), a rotation angle calculation unit (112), and a landmark adjustment unit (113). A first type of image (IMG1) may be input to each of the landmark detection unit (111) and the rotation angle calculation unit (112).

[0086] The landmark detection unit (111) can detect multiple landmarks from a first type image (IMG1) using a landmark detection algorithm. The landmark detection unit (111) may be a model utilizing the landmark detection algorithm described above.

[0087] The rotation angle calculation unit (112) can calculate the rotation angle of the face from the first type of image (IMG1). The rotation angle calculation unit (112) may be a model that receives a side image of the face and outputs a rotation angle of the face based on the side image of the face. For example, the rotation angle calculation unit (112) may extract a plurality of feature points from the side image of the face and output the rotation angle of the face using the PosIt (Pose from Orthography and Scaling) algorithm. As another example, the rotation angle calculation unit (112) may be a pre-trained neural network model that receives a side image of the face and predicts the rotation angle of the face.

[0088] In another embodiment, the rotation angle calculation unit (112) can receive a plurality of landmarks from the landmark detection unit (111). The rotation angle calculation unit (112) can output the rotation angle of the face based on the plurality of landmarks.

[0089] The landmark adjustment unit (113) can receive a plurality of landmarks from the landmark detection unit (111) and a rotation angle of the face from the rotation angle calculation unit (112). The landmark adjustment unit (113) can adjust the position of at least one landmark included in the plurality of landmarks using the rotation angle of the face.

[0090] The feature point detection model (110A) can output a plurality of landmarks including at least one landmark whose position has been adjusted as a plurality of feature points. That is, the feature point detection model (110A) can output a plurality of feature points for a frontal image of a face corresponding to a side image of the face.

[0091]

[0092] FIG. 5 is a diagram for explaining a method for detecting feature points according to another embodiment of the present disclosure.

[0093] Referring to FIG. 5, the feature point detection model (110B) may include a landmark detection unit (111) and a landmark deformation unit (113). A first type of image (IMG1) may be input to the landmark detection unit (111).

[0094] The landmark detection unit (111) can detect multiple landmarks from a first type image (IMG1) using a landmark detection algorithm. The landmark detection unit (111) may be a model utilizing the landmark detection algorithm described above.

[0095] The landmark detection unit (111) can output a plurality of landmarks classified into each of a plurality of face regions. For example, the landmark detection unit (111) can output at least one first landmark classified into an eye region, at least one second landmark classified into a nose region, and at least one third landmark classified into a mouth region.

[0096] The landmark deformation unit (113) can acquire a plurality of landmarks classified into each of a plurality of facial regions. The landmark deformation unit (113) can adjust the position of at least one landmark in a selected region among the plurality of facial regions. The selected region may be selected based on a user input entered through a user interface or may be selected randomly. The selected region may be a facial region in which muscle movement changes significantly depending on facial expression.

[0097] By adjusting the position of at least one landmark in a selected area among multiple face areas, the landmark deformation unit (113) can output multiple deformed landmarks.

[0098] The feature point detection model (110B) can output a plurality of transformed landmarks as a plurality of feature points. That is, the feature point detection model (110B) can output a plurality of feature points for an image in which a facial expression has changed in a frontal image of a face.

[0099]

[0100] FIG. 6 is a block diagram of a learning data generation device according to an additional embodiment of the present disclosure.

[0101] Referring to FIG. 6, the learning data generation device (100B) may further include a first classification model (150) and a second classification model (160) from the learning data generation device (100A) of FIG. 3. The learning data generation device (100B) is substantially the same as the learning data generation device (100A) of FIG. 3 except for the first classification model (150) and the second classification model (160), and therefore, any redundant description will be omitted.

[0102] As described with reference to FIG. 5, the feature point detection model (110) can detect a plurality of landmarks from a first type image (IMG1) and output the transformed plurality of landmarks as a plurality of feature points. The first classification model (150) can obtain the transformed plurality of landmarks. The first classification model (150) can classify facial expressions based on the transformed plurality of landmarks.

[0103] The first classification model (150) may be a model trained to perform facial expression classification by receiving multiple landmarks as input. The first classification model (150) may be a CNN-based deep learning model. Facial expressions classified by the first classification model (150) may include 'smile', 'joy', 'surprise', 'angry', 'fear', 'sadness', 'embarrassment', etc. For example, the first classification model (150) may receive multiple landmarks as input and classify the facial expression as 'embarrassment'.

[0104] The learning data generation device (100B) can label a second type image (IMG2) based on the information labeled in the first type image (IMG1) and the first classification result. In an embodiment where the classification result for a plurality of landmarks transformed in the first type image (IMG1) is 'embarrassed', the second type image (IMG2) can be labeled with a plurality of classes, such as a gender of female, a hair color of black, an expression of embarrassed, and an oval face shape.

[0105] According to embodiments, the learning data generation device (100B) can label a second type of image (IMG2) based on the reliability of the classification model. The learning data generation device (100B) can generate a second type of image (IMG2) based on the plurality of transformed landmarks by utilizing the generation model. The second classification model (160) can obtain the second type of image (IMG2). The second classification model (160) can classify a facial expression based on the second type of image (IMG2).

[0106] The second classification model (160) may be a model trained to perform facial expression classification by receiving a facial image as input. The first classification model (150) may be a CNN-based deep learning model. For example, the first classification model (150) may receive a second type of image (IMG2) as input and classify the facial expression as "sadness."

[0107] The learning data generation device (100B) can compare the first classification result of the first classification model (150) with the second classification result of the second classification model (160). Specifically, the first classification model (150) can output the first classification result and a first reliability for the first classification result. The second classification model (160) can output the second classification result and a second reliability for the second classification result. The learning data generation device (100B) can compare the first reliability and the second reliability.

[0108] The learning data generation device (100B) can select a classification result with a higher reliability value among the first and second classification results as a result of the comparison. The learning data generation device (100B) can label the second type of image (IMG2) based on the selected classification result.

[0109] In an embodiment where the first classification result is 'embarrassment' and the second classification result is 'sadness', if the reliability of the second classification result is greater than the reliability of the first classification result, the learning data generation device (100B) can label the second type of image (IMG2) into multiple classes in which the gender is female, the hair color is black, the expression is sad, and the face shape is oval.

[0110]

[0111] FIG. 7 is a flowchart illustrating a method for generating learning data according to an embodiment of the present disclosure.

[0112] Referring to FIG. 7, a learning data generation device can acquire a first type of image (S110). The first type of image may be a low-quality image. A low-quality image may refer to an image from which accurate feature extraction is difficult.

[0113] The feature point detection model can detect a plurality of feature points from the first type of image using a landmark detection algorithm (S120).

[0114] The encoder can obtain context information based on the above multiple feature points (S130).

[0115] The generative model can generate a second type of image based on the above context information (S140). The second type of image may be a high-quality image. A high-quality image may refer to an image from which accurate features can be extracted.

[0116] The learning data generation device can automatically label a second type image (IMG2) based on information labeled in a first type image (IMG1).

[0117] The generative model can acquire text information as additional conditional information. The text information may be information for changing the class labeled in a first type of image (IMG1). The generative model can generate a second type of image (IMG2) based on the context information and text information.

[0118] When text information is input into the generation model, the learning data generation device (100A) can label a second type image (IMG2) based on the information labeled in the first type image (IMG1) and the text information.

[0119]

[0120] A computer-readable medium storing a data structure according to one embodiment of the present disclosure is disclosed. The aforementioned data structure can be stored in a storage unit in the present disclosure, executed by a processor, and transmitted and received by a communication unit.

[0121] A data structure can refer to the organization, management, and storage of data that enables efficient access and modification. A data structure can refer to the organization of data to solve specific problems (e.g., data analysis, data retrieval, data storage, data modification). A data structure can also be defined as the physical or logical relationships between data elements designed to support specific data processing functions. Logical relationships between data elements can include connections between user-defined data elements. Physical relationships between data elements can include actual relationships between data elements physically stored on a computer-readable storage medium (e.g., persistent storage). Specifically, a data structure can include a collection of data, relationships between data, and functions or commands applicable to the data. An effectively designed data structure allows a computing device to perform operations while minimizing the use of its resources. Specifically, a computing device can improve the efficiency of operations, reading, inserting, deleting, comparing, exchanging, and searching through an effectively designed data structure.

[0122] Data structures can be categorized as linear or nonlinear, depending on their form. A linear data structure can be a structure in which only one data item is linked to the next. Linear data structures can include lists, stacks, queues, and deques. A list can refer to a series of data sets with an internal order. Lists can also include linked lists. A linked list is a data structure in which data is linked in a single line, each item having a pointer. In a linked list, a pointer can contain information about the next or previous item. Linked lists can be expressed as singly linked lists, doubly linked lists, or circular linked lists, depending on their form. A stack can be a data listing structure with limited data access. A stack can be a linear data structure in which data operations (e.g., insertion or deletion) can only be performed at one end of the data structure. Data stored in a stack can be a Last-in-First-out (LIFO) data structure. A queue is a data structure with limited access to data. Unlike a stack, it can be a first-in, first-out (FIFO) data structure, with later data being retrieved later. A deck can be a data structure that can process data at both ends.

[0123] A nonlinear data structure can be a structure in which multiple pieces of data are connected behind a single piece of data. Nonlinear data structures can include graph data structures. A graph data structure can be defined by vertices and edges, and an edge can include a line connecting two different vertices. Graph data structures can include tree data structures. A tree data structure can be a data structure in which there is only one path connecting two different vertices among multiple vertices included in the tree. In other words, it can be a data structure that does not form a loop in a graph data structure.

[0124] Throughout this specification, the terms artificial intelligence-based model, computational model, neural network, network function, and neural network may be used interchangeably. Hereinafter, they are collectively referred to as neural networks. A data structure may include a neural network. And, a data structure including a neural network may be stored on a computer-readable medium. A data structure including a neural network may also include preprocessed data for processing by a neural network, data input to a neural network, neural network weights, neural network hyperparameters, data obtained from a neural network, activation functions associated with each node or layer of a neural network, loss functions for neural network learning, etc. A data structure including a neural network may include any of the components disclosed above. That is, a data structure including a neural network may be configured to include all or any combination of preprocessed data for processing by a neural network, data input to a neural network, neural network weights, neural network hyperparameters, data obtained from a neural network, activation functions associated with each node or layer of a neural network, loss functions for neural network learning, etc. In addition to the aforementioned configurations, a data structure including a neural network may include any other information that determines the characteristics of the neural network. Furthermore, the data structure may include any form of data used or generated in the computational process of the neural network, and is not limited to the aforementioned. The computer-readable medium may include a computer-readable recording medium and / or a computer-readable transmission medium. A neural network may be composed of a set of interconnected computational units, which may generally be referred to as nodes. These nodes may also be referred to as neurons. A neural network is composed of at least one node.

[0125] The data structure may include data input to a neural network. The data structure including the data input to the neural network may be stored on a computer-readable medium. The data input to the neural network may include training data input during the neural network training process and / or input data input to the neural network after training has been completed. The data input to the neural network may include data that has undergone preprocessing and / or data that is the target of preprocessing. Preprocessing may include a data processing process for inputting data to the neural network. Accordingly, the data structure may include data that is the target of preprocessing and data generated by the preprocessing. The above-described data structure is merely an example, and the present disclosure is not limited thereto.

[0126] The data structure may include weights of the neural network. (In this specification, the terms "weight" and "parameter" may be used interchangeably.) And the data structure including the weights of the neural network may be stored in a computer-readable medium. The neural network may include a plurality of weights. The weights may be variable and may be varied by a user or an algorithm so that the neural network can perform a desired function. For example, when one or more input nodes are interconnected to one output node by respective links, the output node may determine a data value output from the output node based on the values ​​input to the input nodes connected to the output node and the weights set for the links corresponding to each input node. The above-described data structure is merely an example, and the present disclosure is not limited thereto.

[0127] By way of example and not limitation, the weights may include weights that vary during the neural network training process and / or weights that have completed neural network training. The weights that vary during the neural network training process may include weights at the start of the training cycle and / or weights that vary during the training cycle. The weights that have completed neural network training may include weights that have completed the training cycle. Accordingly, a data structure including the weights of a neural network may include a data structure including weights that vary during the neural network training process and / or weights that have completed neural network training. Therefore, the above-described weights and / or combinations of each weight are included in the data structure including the weights of a neural network. The above-described data structures are merely examples and the present disclosure is not limited thereto.

[0128] A data structure including neural network weights can be stored in a computer-readable storage medium (e.g., memory, hard disk) after going through a serialization process. Serialization can be a process of converting a data structure into a form that can be stored on the same or different computing devices and later reconstructed and used. A computing device can serialize the data structure to transmit and receive data over a network. The serialized data structure including neural network weights can be reconstructed on the same computing device or another computing device through deserialization. The data structure including neural network weights is not limited to serialization. Furthermore, the data structure including neural network weights can include a data structure that increases computational efficiency while minimizing the use of computing device resources (e.g., a B-Tree, an R-Tree, a Trie, an m-way search tree, an AVL tree, a Red-Black Tree in nonlinear data structures). The foregoing is merely an example, and the present disclosure is not limited thereto.

[0129] The data structure may include hyperparameters of a neural network. Furthermore, the data structure including the hyperparameters of the neural network may be stored on a computer-readable medium. The hyperparameters may be variables that can be varied by the user. The hyperparameters may include, for example, a learning rate, a cost function, the number of learning cycle repetitions, weight initialization (e.g., setting a range of weight values ​​to be subject to weight initialization), and the number of hidden units (e.g., the number of hidden layers, the number of nodes in the hidden layer). The above-described data structure is merely an example, and the present disclosure is not limited thereto.

[0130] An AI model according to one embodiment of the present disclosure may be a generative model. A generative model is a model that learns the distribution of given training data and generates similar data that follows the distribution of the training data. Generative models may include a variational autoencoder (VAE), a generative adversarial network (GAN), a diffusion model, and the like. A variational autoencoder (VAE) may refer to a model that estimates the distribution of training data (approximate density) and learns to reconstruct input data based on the estimated distribution of the training data. A generative adversarial network (GAN) may refer to a model that learns to generate data without explicitly defining the distribution of the training data. A diffusion model may include a diffusion process that gradually adds noise generated from a fixed normal distribution to the data. Furthermore, the diffusion model can reconstruct data by learning a reverse process that removes noise. In other words, a diffusion model can mean a model that generates a result image with a probability distribution similar to the input data by gradually removing noise generated from a learned normal distribution.

[0131] Typically, autoencoders perform unsupervised learning, learning low-dimensional feature representations from unlabeled data. The features extracted by the encoder are latent codes defined by specific numerical values. Unlike autoencoders, variational autoencoders (VAEs) estimate a probability distribution over the latent space (Variational Inference) and learn based on values ​​randomly sampled from the estimated probability distribution. As an example, but not a limitation, variational autoencoders can be trained using the loss function shown in Equation 1 below.

[0132] [Mathematical Formula 1]

[0133]

[0134] Here, the first term represents the reconstruction error, which is the difference between the input data (xi) and the value generated based on the value (z) sampled from the estimated probability distribution (posterior distribution). Here, minimizing the loss function can mean maximizing the likelihood for the value (z) sampled from the estimated probability distribution (posterior distribution).

[0135] The second term represents the regularization error, which is the difference between the estimated posterior distribution and the target distribution, the prior distribution. Since the prior distribution typically follows a Gaussian normal distribution, minimizing the loss function here can mean making the estimated posterior distribution follow a Gaussian normal distribution as closely as possible.

[0136] A generative adversarial network (GAN) may include a generator and a discriminator. The generator can generate fake data from samples in the latent space. The discriminator can distinguish between real and fake data. A GAN is an unsupervised machine learning technique that learns from data through a process in which the generator and discriminator compete against each other. As an example, but not a limitation, a GAN can be trained using a loss function, as shown in Equation 2 below.

[0137] [Equation 2]

[0138]

[0139] Here, the first term could mean that the discriminator learns to accurately distinguish between real data and fake data generated by the generator. The second term could mean that the generator learns to create data so similar to real data that the discriminator cannot distinguish them. During actual training, rather than training the generator and discriminator networks simultaneously, one network can be fixed while the other network is updated, allowing them to be updated separately.

[0140] Generative adversarial networks (GANs) include Deep Convolutional GANs (DCGANs), Conditional GANs (CGANs), Cycle-Consistent Generative Adversarial Networks (CycleGANs), VAE-GANs, PGGANs, StyleGANs, and Self-Attention GANs (SAGANs). DCGANs can be models that apply a Convolutional Neural Network (CNN) to the generator and discriminator. CGANs can be trained to generate data that aligns with specific conditions by adding labels or conditional information to the input. CycleGANs consist of two generators and two discriminators and can learn to transform images between two different domains. VAE-GANs can simultaneously leverage the efficient latent space learning of VAEs and the high-quality data generation capabilities of GANs. PGGANs can improve the quality of generated images by gradually increasing the resolution of the images while training. StyleGAN applies the concept of style transfer to the PGGAN architecture, allowing it to reflect more disentangled style information. SAGAN introduces a self-attention mechanism to the GAN model, enabling it to effectively learn relationships between spatially distant image regions.

[0141] Models according to one embodiment of the present disclosure can be trained to remove part or all of the predicted noise for Gaussian distributed noise, thereby obtaining data from which part or all of the predicted noise has been removed.

[0142] A computing device (10) according to one embodiment of the present disclosure may train a neural network to obtain data from which part or all of the noise has been removed by removing part or all of the noise from an isotropic Gaussian distributed noise. At this time, the neural network may include a conditional noise prediction model. In addition, the conditional noise prediction model may include a U-Net structure in which input and output have the same size, and may input data x(t) containing noise and a diffusion time step t, and may predict and output diffusion noise contained in the data x(t) containing noise.

[0143] A computing device (10) may perform a forward process of repeating a process of adding random Gaussian noise little by little over T time steps to original data x(0) that does not contain noise, and obtaining isotropic Gaussian distributed noise x(T) as a result. The type of the original data x(0) may include various examples such as audio data in addition to image data. Meanwhile, the forward process according to an embodiment of the present disclosure may be performed, for example, through the following formula.

[0144] [Mathematical Formula 1]

[0145]

[0146]

[0147] In the above mathematical formula 1 can be used as a hyperparameter in the process of calculating the diffusion coefficient, can be set to any value and 0< < < … < It can be set to a value of <1. For example, With this value of 0.0001 can have a value of 0.02, from until The value of can increase linearly, can increase along a cosine function, and T, which means the total number of diffusion steps, can be set to 1000. However, It is only an example that increases linearly or along a cosine function, and according to embodiments of the present disclosure, depending on the type of the original data The increase amount can be determined differently, and a specific explanation is provided below through [Mathematical Formula 2]. In addition, can mean random Gaussian distribution noise. The general equation representing the data x(t) containing the noise at time step t in terms of the original data x(0) without noise and the included diffusion noise can be expressed as follows.

[0148] [Equation 2]

[0149]

[0150]

[0151]

[0152]

[0153] According to one embodiment of the present disclosure, equations (1) and (2) in the above mathematical expression 2 are diffusion coefficients is an expression that expresses the specific meaning of . In equation (1) of the above mathematical expression 2, the diffusion coefficient at a specific time step t is the hyperparameter from 1 It can be calculated as a value minus the diffusion coefficient in equation (2) of the above mathematical expression 2. can mean the diffusion coefficient sequentially accumulated from time step 1 to t. Therefore, in the above mathematical expression 2, expression (3) is the diffusion coefficient ( ) of the noise-containing data x(t) at time step t and the original data x(0) without noise. ) and random Gaussian distribution noise ( ) is expressed as a formula for the original data. In addition, formula (4) in the above mathematical formula 2 may mean the ratio n(t) of noise and the ratio s(t) of signal determined based on the type of the original data according to one embodiment of the present disclosure. Therefore, data x(t) containing noise at time step t may be expressed based on the ratio n(t) of noise and the ratio s(t) of signal. Specifically, data x(t) containing noise at time step t becomes closer to the form of the original data as the ratio of original data x(0) not containing noise (i.e., the ratio s(t) of signal) increases, and random Gaussian distribution noise ( ) (i.e., the ratio of noise n(t)) increases, the closer it can get to the form of random Gaussian distribution noise.

[0154] For example, the computing device (10) may obtain first data by adding first noise to the original data based on the determined noise ratio in a forward process of adding random Gaussian noise little by little over T time steps to original data x(0) that does not contain noise, and may obtain second data by adding second noise to the first data based on the determined noise ratio. In one embodiment, the type of the original data x(0) may include various examples such as text data and audio data in addition to image data.

[0155] Through this, the computing device (10) can perform a forward process of repeating the process of adding random Gaussian noise over T time steps based on the “ratio of noise determined through equation (4) in the above mathematical expression 2” to the original data x(0) that does not contain noise, and consequently obtaining isotropic Gaussian distributed noise x(T). However, the forward process is not limited to the example of the above mathematical expression 2, and various processes of adding noise to data can be included in the forward process.

[0156] In addition, the computing device (10) can train the neural network to perform a reverse process in which a process of removing random Gaussian noise little by little over T time steps is repeated for isotropic Gaussian distributed noise x(T) in the opposite direction to the forward process, and as a result, original data x(0) that does not contain noise is obtained. In this regard, an equation representing the reverse process can be expressed as follows.

[0157] [Equation 3]

[0158]

[0159]

[0160]

[0161] In the above mathematical expression 3, expression (1) is the noise prediction result predicted by the neural network for “data x(t) containing noise” ( ) is an equation representing a reverse process of obtaining “the data x(t-1) of the previous step from which the noise has been partially removed” by removing the noise. For example, the computing device (10) can obtain “the first data x(1) of the previous step from which the second noise has been removed” by removing the noise prediction result predicted by the neural network for “the second data x(2) including the second noise.” In the mathematical expression 3, equation (2) is the diffusion coefficient at the current time step t. , and in equation (3) of the above mathematical expression 3, means the dispersion parameter, and the diffusion coefficient It can be calculated based on. However, the reverse process is not limited to the above [Mathematical Formula 3], and various processes for removing noise from data containing noise can be included in the reverse process.

[0162] Specifically, the computing device (10) inputs data x(t) containing noise and a time step t to the neural network, compares the noise prediction result predicted by the neural network with the diffusion noise actually included to calculate a loss function, and performs gradient down according to the loss function to train the neural network. For example, the loss function calculated by comparing the noise prediction result predicted by the neural network with the diffusion noise actually included can be expressed by the following formula.

[0163] [Equation 4]

[0164]

[0165]

[0166] In the above [Mathematical Formula 4], the loss function is actually the included diffusion noise ( ) and the noise prediction result predicted by the above neural network ( ) can be calculated by comparing them. For example, the computing device (10) can predict “the first noise included in the first data x(1)” and calculate the first loss function by comparing the predicted first noise with the first noise added to the original data x(0). The computing device (10) can predict “the second noise included in the second data x(2)” and calculate the second loss function by comparing the predicted second noise with the second noise added to the first data x(1). However, the loss function is not limited to the example of Equation 4, and may include various loss functions calculated by comparing the noise prediction result with the actually included diffusion noise.

[0167] Additionally, the neural network may be trained to obtain “data x(1) with the second noise removed” by predicting the diffusion noise included in the data x(2) containing the second noise and removing the predicted second noise. For example, the computing device (10) may also train the neural network based on at least one of the first loss function and the second loss function. In addition, the neural network may be trained to obtain the original data x(0) with all noise removed by repeating the process of predicting the diffusion noise included in the “data x(t) containing the noise” and removing the predicted diffusion noise one or more times to completely remove the diffusion noise included in the “data x(t) containing the noise.” Meanwhile, in the process of training to remove diffusion noise from data containing noise by utilizing the neural network, the ratio of noise is determined based on the type of the original data, so that the neural network may be trained to generate data of better quality.

[0168] The computing device (10) can determine the ratio of noise based on the size of data that can be represented for the type of the determined original data. Specifically, the computing device (10) can determine the ratio of noise so that the size of noise added to the original data is smaller than the minimum size of data that can be represented for the type of the determined original data. For example, when the type of the original data is determined to be image data, the original data may have a depth of 8 bits per channel. Accordingly, the computing device (10) can determine the first ratio of noise so that the size of noise added to the original data is smaller than 1 / (2^8) (= approximately 0.00039), which is the minimum size of data that can be represented for image data.

[0169] Meanwhile, according to another embodiment of the present disclosure, if the type of the original data is determined to be audio data, the original data may have a depth of 16 bits per sample. Accordingly, the computing device (10) may determine the second ratio of the noise so that the size of the noise added to the original data is smaller than 1 / (2^16) (= approximately 0.000015), which is the minimum size of data that can be expressed for audio data. Accordingly, the second ratio of the noise when the original data is audio data may be determined to be smaller than the first ratio of the noise when the original data is image data. For example, the ratio of the noise may be determined through the following mathematical formula.

[0170] [Equation 5]

[0171]

[0172]

[0173] Specifically, according to one embodiment of the present disclosure, with reference to equation (1) in [Mathematical Formula 5], the noise ratio n(t) may be determined to increase exponentially, and with reference to equation (2) in [Mathematical Formula 5], the signal ratio s(t) may be determined to decrease exponentially. At this time, r may mean a hyperparameter that determines the shape of the curves of the noise ratio n(t) and the signal ratio s(t). However, [Mathematical Formula 5] is merely an example, and the noise ratio and the signal ratio may be determined in various other ways. Through this, the neural network model may be trained so that the results generated through the neural network model are hardly affected by noise even in a data domain (for example, audio data, etc.) that is relatively sensitive to noise compared to image data.

[0174]

[0175] FIG. 8 is a simplified, general schematic diagram of an exemplary computing environment in which embodiments of the present disclosure may be implemented.

[0176] Although the present disclosure has been described above as being generally implemented by a computing device, those skilled in the art will appreciate that the present disclosure may also be implemented in combination with computer-executable instructions and / or other program modules that may be executed on one or more computers and / or as a combination of hardware and software.

[0177] Generally, program modules include routines, programs, components, data structures, and the like that perform particular tasks or implement particular abstract data types. Furthermore, those skilled in the art will appreciate that the methods of the present disclosure can be implemented with other computer system configurations, including single-processor or multiprocessor computer systems, minicomputers, mainframe computers, as well as personal computers, handheld computing devices, microprocessor-based or programmable consumer electronics, and the like, each of which may be operatively connected to one or more associated devices.

[0178] The described embodiments of the present disclosure can be practiced in a distributed computing environment, where certain tasks are performed by remote processing devices that are connected through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

[0179] Computers typically include a variety of computer-readable media. Computer-readable media can be any media that can be accessed by a computer, and includes both volatile and nonvolatile media, transitory and non-transitory media, removable and non-removable media. By way of example, and not limitation, computer-readable media can include computer-readable storage media and computer-readable transmission media. Computer-readable storage media includes both volatile and nonvolatile media, transitory and non-transitory media, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital video disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be accessed by a computer and used to store the desired information.

[0180] Computer-readable transmission media typically includes any information delivery media that embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transport mechanism. The term modulated data signal means a signal that has one or more of its characteristics set or changed so as to encode information in the signal. By way of example, and not limitation, computer-readable transmission media includes wired media, such as a wired network or direct-wired connection, and wireless media, such as acoustic, RF, infrared, or other wireless media. Combinations of any of the above are also intended to be included within the scope of computer-readable transmission media.

[0181] An exemplary environment for implementing various aspects of the present disclosure is illustrated, including a computer (1102), which includes a processing unit (1104), system memory (1106), and a system bus (1108). The system bus (1108) connects system components, including but not limited to the system memory (1106), to the processing unit (1104). The processing unit (1104) may be any of a variety of commercially available processors. Dual processors and other multiprocessor architectures may also be utilized as the processing unit (1104).

[0182] The system bus (1108) may be any of several types of bus structures that may be additionally interconnected to a memory bus, a peripheral bus, and a local bus using any of a variety of commercial bus architectures. The system memory (1106) includes read-only memory (ROM) (1110) and random access memory (RAM) (1112). A basic input / output system (BIOS) is stored in non-volatile memory (1110), such as ROM, EPROM, or EEPROM, and includes basic routines that help transfer information between components within the computer (1102), such as during start-up. The RAM (1112) may include high-speed RAM, such as static RAM, for caching data.

[0183] The computer (1102) includes an internal hard disk drive (HDD) (1114) (e.g., EIDE, SATA) - which may be configured for external use within a suitable chassis (not shown), a magnetic floppy disk drive (FDD) (1116) (e.g., for reading from or writing to a removable diskette (1118)), and an optical disk drive (1120) (e.g., for reading from or writing to a CD-ROM disk (1122) or other high-capacity optical media such as a DVD). The hard disk drive (1114), the magnetic disk drive (1116), and the optical disk drive (1120) may be connected to the system bus (1108) by a hard disk drive interface (1124), a magnetic disk drive interface (1126), and an optical drive interface (1128), respectively. The interface (1124) for implementing an external drive includes at least one or both of Universal Serial Bus (USB) and IEEE 1394 interface technologies.

[0184] These drives and their associated computer-readable media provide non-volatile storage of data, data structures, computer-executable instructions, and the like. In the case of the computer (1102), the drives and media correspond to storing any data in a suitable digital format. While the description of computer-readable media above refers to HDDs, removable magnetic disks, and removable optical media such as CDs or DVDs, those of ordinary skill in the art will appreciate that other types of computer-readable media, such as zip drives, magnetic cassettes, flash memory cards, cartridges, and the like, may also be used in the exemplary operating environment, and that any such media may contain computer-executable instructions for performing the methods of the present disclosure.

[0185] A number of program modules, including an operating system (1130), one or more application programs (1132), other program modules (1134), and program data (1136), may be stored in the drive and RAM (1112). All or portions of the operating system, applications, modules, and / or data may be cached in RAM (1112). It will be appreciated that the present disclosure may be implemented in various commercially available operating systems or combinations of operating systems.

[0186] A user may enter commands and information into the computer (1102) via one or more wired / wireless input devices, such as a keyboard (1138) and a pointing device such as a mouse (1140). Other input devices (not shown) may include a microphone, an IR remote control, a joystick, a game pad, a stylus pen, a touch screen, and the like. These and other input devices are often connected to the processing unit (1104) via an input device interface (1142) that is connected to the system bus (1108), but may be connected by other interfaces such as a parallel port, an IEEE 1394 serial port, a game port, a USB port, an IR interface, and the like.

[0187] A monitor (1144) or other type of display device is also connected to the system bus (1108) via an interface, such as a video adapter (1146). In addition to the monitor (1144), the computer typically includes other peripheral output devices (not shown), such as speakers, a printer, and so on.

[0188] The computer (1102) may operate in a networked environment using logical connections to one or more remote computers, such as remote computer(s) (1148), via wired and / or wireless communications. The remote computer(s) (1148) may be a workstation, a computing device computer, a router, a personal computer, a portable computer, a microprocessor-based entertainment device, a peer device, or other conventional network node, and generally include many or all of the components described for the computer (1102), although for simplicity, only the memory storage device (1150) is shown. The logical connections shown include wired / wireless connections to a local area network (LAN) (1152) and / or a larger network, such as a wide area network (WAN) (1154). Such LAN and WAN networking environments are common in offices and companies and facilitate enterprise-wide computer networks, such as intranets, all of which may be connected to a worldwide computer network, such as the Internet.

[0189] When used in a LAN networking environment, the computer (1102) is connected to a local network (1152) via a wired and / or wireless communication network interface or adapter (1156). The adapter (1156) may facilitate wired or wireless communications to the LAN (1152), which may include a wireless access point installed therein for communicating with the wireless adapter (1156). When used in a WAN networking environment, the computer (1102) may include a modem (1158), be connected to a communications computing device on the WAN (1154), or have other means of establishing communications over the WAN (1154), such as via the Internet. The modem (1158), which may be internal or external and wired or wireless, is connected to the system bus (1108) via a serial port interface (1142). In a networked environment, program modules or portions thereof described for the computer (1102) may be stored in a remote memory / storage device (1150). It will be appreciated that the network connections depicted are exemplary and other means of establishing a communications link between the computers may be used.

[0190] The computer (1102) operates to communicate with any wireless device or object that is arranged and operates via wireless communication, such as a printer, a scanner, a desktop and / or portable computer, a portable data assistant (PDA), a communication satellite, any equipment or location associated with a radio-detectable tag, and a telephone. This includes at least Wi-Fi and Bluetooth wireless technologies. Accordingly, the communication may be a predefined structure as in a conventional network, or may simply be an ad hoc communication between at least two devices.

[0191] Wi-Fi (Wireless Fidelity) enables connections to the Internet and other devices without wires. Wi-Fi is a wireless technology that allows devices, such as computers, to send and receive data anywhere within the coverage area of ​​a base station, both indoors and outdoors, similar to cell phones. Wi-Fi networks use wireless technologies called IEEE 802.11 (a, b, g, etc.) to provide secure, reliable, and high-speed wireless connections. Wi-Fi can be used to connect computers to each other, to the Internet, and to wired networks (using IEEE 802.3 or Ethernet). Wi-Fi networks can operate in the unlicensed 2.4 and 5 GHz radio bands, at data rates of, for example, 11 Mbps (802.11a) or 54 Mbps (802.11b), or in products that include both bands (dual-band).

[0192] Those skilled in the art will appreciate that information and signals may be represented using any of a variety of different technologies and techniques. For example, the data, instructions, commands, information, signals, bits, symbols, and chips referenced in the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0193] Those skilled in the art will appreciate that the various illustrative logical blocks, modules, processors, means, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, various forms of programs or design code (referred to herein, for convenience, as software), or a combination of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

[0194] The various embodiments presented herein can be implemented as a method, apparatus, or article of manufacture using standard programming and / or engineering techniques. The term article of manufacture includes a computer program, carrier, or media accessible from any computer-readable storage device. For example, computer-readable storage media include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic strips, etc.), optical disks (e.g., CDs, DVDs, etc.), smart cards, and flash memory devices (e.g., EEPROMs, cards, sticks, key drives, etc.). Furthermore, various storage media presented herein include one or more devices and / or other machine-readable media for storing information.

[0195] It should be understood that the specific order or hierarchy of steps in the presented processes is merely an example of exemplary approaches. It should be understood that the specific order or hierarchy of steps in the processes may be rearranged within the scope of the present disclosure based on design priorities. The appended method claims provide elements of various steps in a sample order, but are not intended to be limited to the specific order or hierarchy presented.

[0196] The description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the embodiments disclosed herein, but is to be construed in the broadest scope consistent with the principles and novel features disclosed herein.

[0197] As described above, the relevant contents have been described in the best form for carrying out the invention.

Claims

1. A method for modeling the location of an item by analyzing multiple images, performed by a computing device. A step of acquiring multiple images related to a store; A step of generating a semantic map based on the plurality of images by utilizing a neural network model; Steps to obtain customer analysis information; A step of obtaining the location coordinates of an item based on the customer analysis information by utilizing a location prediction model; and A step of generating an image in which the location of the item within the store is modeled based on the location coordinates and the semantic map by utilizing a generative model. including, method.

2. In paragraph 1, The steps of obtaining the above customer analysis information are: A step of obtaining customer statistical information based on first data obtained from a first sensor; and A step of acquiring customer characteristic information based on second data acquired from a second sensor. including, method.

3. In paragraph 2, The steps for obtaining the above customer statistical information are: A step of analyzing the first data acquired from the depth sensor to acquire customer movement information, method.

4. In paragraph 3, The step of obtaining the above customer characteristic information is: A step of analyzing the second data acquired from the camera sensor to acquire personal information of the customer, method.

5. In paragraph 2, The step of obtaining the location coordinates of the above item is: A step of obtaining the location coordinates of the item based on the customer statistical information, the customer characteristic information, and additional information by utilizing the location prediction model. Including, The above additional information includes at least one of item information and store grid information, The above item information includes information on the size of the space occupied by the item within the store and analysis information on the target customer base for the item. The grid information of the above store is location information assigned to each grid cell formed by dividing the space within the store into a grid shape. method.

6. In paragraph 1, The step of generating the semantic map based on the above plurality of images is: A step of extracting egocentric features of the above plurality of images; A step of converting the above egocentric features into an environment-centric (allocentric) feature map; and A step of predicting environment-centric meaning based on the environment-centric feature map and generating the semantic map. including, method.

7. In paragraph 6, The step of converting the above ego-centric features into the above environment-centric feature map is: A step of projecting the egocentric features onto an environment-centric memory map based on the camera internal parameter matrix and the depth information of each pixel of the image. including, method.

8. In paragraph 7, The step of converting the above ego-centric features into the above environment-centric feature map is: A step of bidirectionally projecting the above egocentric features and fusing the bidirectionally projected environment-centric memory features to create an environment-centric representation. including more, method.

9. A computer program stored in a computer-readable storage medium, wherein when the computer program is executed by one or more processors, the computer program causes the one or more processors to perform operations of analyzing a plurality of images to model the location of an item, the operations comprising: The act of acquiring multiple images related to a store; An operation of generating a semantic map based on the plurality of images by utilizing a neural network model; Actions to obtain customer analytics information; An operation of obtaining the location coordinates of an item based on the customer analysis information by utilizing a location prediction model; and An operation of generating an image in which the location of the item within the store is modeled based on the location coordinates and the semantic map by utilizing a generative model. including, A computer program stored on a computer-readable storage medium.

10. In paragraph 9, The action of obtaining the above customer analysis information is as follows: An operation of obtaining customer statistical information based on first data obtained from a first sensor; and An action to obtain customer characteristic information based on second data obtained from a second sensor. including, A computer program stored on a computer-readable storage medium.

11. In paragraph 10, The action of obtaining the location coordinates of the above item is: An operation of obtaining the location coordinates of the item based on the customer statistical information, the customer characteristic information, and additional information by utilizing the location prediction model. Including, The above additional information includes at least one of item information and store grid information, The above item information includes information on the size of the space occupied by the item within the store and analysis information on the target customer base for the item. The grid information of the above store is location information assigned to each grid cell formed by dividing the space within the store into a grid shape. A computer program stored on a computer-readable storage medium.

12. In paragraph 9, The operation of generating the semantic map based on the above plurality of images is: An operation of extracting egocentric features of the above multiple images; An operation of converting the above egocentric features into environment-centric (allocentric) feature maps; and An operation of predicting environment-centric meaning based on the environment-centric feature map and generating the semantic map. including, A computer program stored on a computer-readable storage medium.

13. As a computing device, at least one processor; and memory Including, At least one processor, Obtain multiple images related to the store, By utilizing a neural network model, a semantic map is created based on the above multiple images, Obtain customer analytics information, By utilizing the location prediction model, the location coordinates of the item are obtained based on the customer analysis information, and By utilizing a generative model, an image in which the location of the item within the store is modeled is generated based on the location coordinates and the semantic map. Computing device.

14. In paragraph 13, Obtain customer statistical information based on the first data acquired from the first sensor, and It is further configured to acquire customer characteristic information based on second data acquired from a second sensor. Computing device.

15. In paragraph 14, By utilizing the above location prediction model, it is further configured to obtain the location coordinates of the item based on the customer statistical information, the customer characteristic information, and additional information, The above additional information includes at least one of item information and store grid information, The above item information includes information on the size of the space occupied by the item within the store and analysis information on the target customer base for the item. The grid information of the above store is location information assigned to each grid cell formed by dividing the space within the store into a grid shape. Computing device.

Images