Content generation method, and electronic device

Abstract

Disclosed is a method by which at least one processor generates content, the content generation method comprising the steps of: acquiring at least one piece of first content; and generating at least one piece of second content associated with the at least one piece of first content by using a machine learning model, wherein the machine learning model includes: an encoder for generating at least one feature vector on the basis of the at least one piece of first content; and a decoder for generating the at least one piece of second content on the basis of the generated at least one feature vector.

Description

Content creation method and electronic device

[0001] The present disclosure relates to a content creation method and an electronic device.

[0002] Artificial Intelligence (AI) technology leverages machine learning and deep learning to develop systems that learn from massive amounts of data, recognize patterns, and make intelligent decisions. It is being utilized in diverse fields, including predictive analytics, autonomous driving, medical diagnosis, language processing, and image generation. In particular, as generative AI technology advances, it is being utilized in a variety of fields.

[0003] Meanwhile, using content generated through generative AI (e.g., synthetic data) in AI model training can yield relatively high-performance models compared to using only real data. Accordingly, the value of content generated through generative AI is increasing, and active research is underway on generating information associated with the content (e.g., labels, annotations, segmentation maps, etc.) (hereinafter referred to as "content information") during content creation.

[0004] Existing methods for generating content information include generating content and then using a separate model to generate information like labels, and predicting information like annotations using only the modules or intermediate results used during content generation. However, since both methods generate content and content information independently, errors can arise between them. Therefore, there is a need for technology that allows content and content information to interact simultaneously and be generated within a single network.

[0005] The present disclosure provides a content creation method and an electronic device for solving the above problems.

[0006] The present disclosure can be implemented in various ways, including a computer-readable, non-transitory recording medium having recorded thereon methods, devices (systems), and / or instructions.

[0007] According to one embodiment of the present disclosure, a content generation method performed by at least one processor includes the steps of obtaining at least one first content, and the step of generating at least one second content associated with the at least one first content using a machine learning model, wherein the machine learning model may include an encoder generating at least one feature vector based on the at least one first content, and a decoder generating at least one second content based on the generated at least one feature vector.

[0008] According to one embodiment, at least one first content includes at least one of a first image, a contour image associated with the first image, a segmentation map associated with the first image, a depth map associated with the first image, bounding box information of an object included in the first image, facial landmark information of a person included in the first image, pose information of a person included in the first image, or a prompt associated with the first image, and at least one second content includes at least one of the first image, an IR image associated with the first image, a second image associated with the first image and having a different domain style in at least a portion of the second image, a contour image associated with the first image, a segmentation map associated with the first image, a depth map associated with the first image, bounding box information of an object included in the first image, facial landmark information of a person included in the first image, pose information of a person included in the first image, table-type data including physical characteristic information of an object included in the first image, a text sequence including physical characteristic information of an object included in the first image, or a data set representing coordinate information of an object included in the first image, and at least one first content and at least one second The content may be at least partially different data.

[0009] According to one embodiment, the decoder may be configured to generate third data represented by a third matrix by connecting first data represented by a first matrix and second data represented by a second matrix by channel, and output the generated third data as at least one second content.

[0010] According to one embodiment, the first matrix and the second matrix included in the third data generated by the decoder may have the same dimension and shape.

[0011] According to one embodiment, the at least one second content includes 2-1 content and 2-2 content different from the 2-1 content, and the decoder may include a first decoder that generates the 2-1 content based on the generated at least one feature vector, and a second decoder that generates the 2-2 content based on the generated at least one feature vector.

[0012] According to one embodiment, the first decoder can generate the 2-1 content based on at least one feature vector and an intermediate vector received from the second decoder, and the second decoder can generate the 2-2 content based on at least one feature vector and an intermediate vector received from the first decoder.

[0013] According to one embodiment, at least one of the first decoder and the second decoder may include a first layer that generates first information associated with content to be generated by the at least one decoder, and a second layer that mixes the first information and second information received from the outside.

[0014] According to one embodiment, the first decoder may include a first layer that generates first information associated with 2-1 content to be generated by the first decoder, and a second layer that mixes second information received from the second decoder with the first information, and the second decoder may include a third layer that generates second information associated with 2-2 content to be generated by the second decoder, and a fourth layer that mixes the first information received from the first decoder with the second information.

[0015] According to one embodiment, the first decoder may be configured to connect first data represented by a first matrix and second data represented by a second matrix by channel to generate third data represented by a third matrix, and output the generated third data as 2-1 content.

[0016] According to one embodiment, at least one of the first decoder and the second decoder may include a first layer that generates first information associated with content to be generated by the at least one decoder, and a second layer that mixes the first information and second information received from the outside.

[0017] A computer-readable non-transitory recording medium having recorded thereon instructions for executing a method according to one embodiment of the present disclosure on a computer may be provided.

[0018] According to one embodiment of the present disclosure, an electronic device includes a memory, and at least one processor connected to the memory and configured to execute at least one computer-readable program included in the memory, wherein the at least one program includes instructions for obtaining at least one first content and generating at least one second content associated with the at least one first content using a machine learning model, wherein the machine learning model may include an encoder generating at least one feature vector based on the at least one first content, and a decoder generating at least one second content based on the generated at least one feature vector.

[0019] According to some embodiments of the present disclosure, content and information associated with the content can be generated within a single network by simultaneously interacting with each other, thereby minimizing errors between the content and information associated with the content.

[0020] The effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by a person having ordinary knowledge in the technical field to which the present disclosure belongs (referred to as “ordinary skilled person”) from the description of the claims.

[0021] Embodiments of the present disclosure will be described below with reference to the accompanying drawings, wherein like reference numerals represent similar elements, but are not limited thereto.

[0022] FIG. 1 is a diagram illustrating an electronic device for generating content according to one embodiment of the present disclosure.

[0023] FIG. 2 is a schematic diagram showing a configuration in which an information processing system is connected to enable communication with a plurality of user terminals in relation to data processing according to one embodiment of the present disclosure.

[0024] FIG. 3 is a block diagram showing the internal configuration of a user terminal and an information processing system according to one embodiment of the present disclosure.

[0025] FIG. 4 is a diagram for explaining the configuration of a machine learning model that generates content by connecting multiple contents in image format by channel according to one embodiment of the present disclosure.

[0026] FIG. 5 is a diagram for explaining a method for generating content in an image format based on content in an image format and content in a text format according to one embodiment of the present disclosure.

[0027] FIG. 6 is a diagram for explaining a method for generating content in an image format based on content in a text format according to one embodiment of the present disclosure.

[0028] FIG. 7 is a diagram for explaining a method for generating content in an image format based on content in an image format according to one embodiment of the present disclosure.

[0029] FIG. 8 is a diagram illustrating the configuration of a machine learning model including a plurality of decoders according to one embodiment of the present disclosure.

[0030] FIG. 9 is a diagram illustrating a method of mixing information centered on one decoder according to one embodiment of the present disclosure.

[0031] FIG. 10 is a diagram illustrating a method of mixing information associated with all decoders according to one embodiment of the present disclosure.

[0032] FIG. 11 is a diagram illustrating a layer within a decoder according to one embodiment of the present disclosure.

[0033] FIG. 12 is a diagram for explaining a method in which each of a plurality of decoders generates content in an image format based on content in an image format and content in a text format according to one embodiment of the present disclosure.

[0034] FIG. 13 is a diagram for explaining a method in which a plurality of decoders generate content in an image format and content in a table format based on content in an image format and content in a text format according to one embodiment of the present disclosure.

[0035] FIG. 14 is a diagram for explaining a method in which a plurality of decoders generate a plurality of contents in image format and a plurality of contents in table format based on contents in image format and contents in text format according to one embodiment of the present disclosure.

[0036] FIG. 15 is a diagram for explaining a method of generating content by connecting multiple contents in image format by channel among multiple decoders based on contents in image format and contents in text format according to one embodiment of the present disclosure.

[0037] FIG. 16 is a diagram for explaining a method in which a plurality of decoders generate content in an image format and content in a table format based on content in an image format according to one embodiment of the present disclosure.

[0038] FIG. 17 is a diagram for explaining a content creation method according to one embodiment of the present disclosure.

[0039] Hereinafter, specific details for implementing the present disclosure will be described in detail with reference to the attached drawings. However, in the following description, specific descriptions of widely known functions or configurations will be omitted if they may unnecessarily obscure the gist of the present disclosure.

[0040] In the attached drawings, identical or corresponding components are assigned the same reference numerals. Furthermore, in the description of the embodiments below, duplicate descriptions of identical or corresponding components may be omitted. However, even if a description of a component is omitted, it is not intended that such component is not included in any embodiment.

[0041] The advantages and features of the disclosed embodiments, and methods for achieving them, will become clearer with reference to the embodiments described below, along with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below and may be implemented in various different forms. These embodiments are provided solely to ensure the completeness of the disclosure and to fully inform those skilled in the art of the scope of the invention.

[0042] The terms used in this specification will be briefly explained, followed by a detailed description of the disclosed embodiments. The terms used in this specification have been selected from widely used, current terms, taking into account the functions of the present disclosure. However, these terms may vary depending on the intentions of engineers working in the relevant field, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the relevant description of the invention. Therefore, the terms used in this disclosure should not be defined simply as names of terms, but rather based on their meanings and the overall content of the present disclosure.

[0043] In this specification, singular expressions include plural expressions unless the context clearly indicates otherwise. Furthermore, plural expressions include singular expressions unless the context clearly indicates otherwise. When a part of the specification is said to include a component, this does not exclude other components, but rather implies that other components may be included, unless otherwise specifically stated.

[0044] Also, the term 'module' or 'part' used in the specification means a software or hardware component, and the 'module' or 'part' performs certain roles. However, the 'module' or 'part' is not limited to software or hardware. The 'module' or 'part' may be configured to reside on an addressable storage medium and may be configured to execute one or more processors. Thus, as an example, the 'module' or 'part' may include at least one of components such as software components, object-oriented software components, class components, and task components, processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, or variables. The functionality provided within the components and 'modules' or 'parts' may be combined into a smaller number of components and 'modules' or 'parts', or further separated into additional components and 'modules' or 'parts'.

[0045] According to one embodiment of the present disclosure, a 'module' or 'unit' may be implemented as a processor and a memory. 'Processor' should be broadly construed to include a general-purpose processor, a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a controller, a microcontroller, a state machine, and the like. In some circumstances, a 'processor' may also refer to an application-specific integrated circuit (ASIC), a programmable logic device (PLD), a field-programmable gate array (FPGA), and the like. A 'processor' may also refer to a combination of processing devices, such as, for example, a combination of a DSP and a microprocessor, a combination of multiple microprocessors, a combination of one or more microprocessors in conjunction with a DSP core, or any other such combination of configurations. In addition, 'memory' should be broadly construed to include any electronic component capable of storing electronic information. 'Memory' may refer to various types of processor-readable media, such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, magnetic or marking data storage, registers, etc. Memory is said to be in electronic communication with the processor if the processor can read information from, and / or write information to, the memory. Memory integrated in a processor is in electronic communication with the processor.

[0046] In addition, terms such as first, second, A, B, (a), (b), etc. used in the following embodiments are only used to distinguish certain components from other components, and the nature, order, or sequence of the components are not limited by the terms.

[0047] Additionally, in the embodiments below, when it is described that a component is 'connected', 'coupled' or 'connected' to another component, it should be understood that the component may be directly connected or connected to the other component, but another component may also be 'connected', 'coupled' or 'connected' between each component.

[0048] Additionally, the terms 'comprises' and / or 'comprising' used in the following embodiments do not exclude the presence or addition of one or more other components, steps, operations and / or elements.

[0049] Hereinafter, various embodiments of the present disclosure will be described in detail with reference to the attached drawings.

[0050] FIG. 1 is a diagram exemplarily illustrating an electronic device (100) for generating content according to one embodiment of the present disclosure. Referring to FIG. 1, the electronic device (100) may obtain at least one first content (120) and generate at least one second content (130) associated with the at least one first content (120) using a machine learning model (110). Here, the machine learning model (110) may include an encoder (112) that generates at least one feature vector (114) based on the at least one first content (120) and a decoder (116) that generates at least one second content (130) based on the generated at least one feature vector (114). For example, the machine learning model (110) may be a generative AI model.

[0051] An electronic device (100) for generating content may include memory and at least one processor. However, the configuration of the electronic device (100) is not limited thereto. According to various embodiments, the electronic device (100) may further include at least one other component in addition to the components described above. For example, the electronic device (100) may further include a communication circuit for receiving various data from an external device.

[0052] The memory can store various data used by at least one component (e.g., a processor) of the electronic device (100). The data can include, for example, input data or output data for software (or a program) and instructions related thereto. The memory can include volatile memory or non-volatile memory.

[0053] The processor may be connected to a memory and configured to execute at least one computer-readable program contained in the memory. For example, the processor may execute software (or a program) to control at least one other component (e.g., hardware or software component) of an electronic device (100) connected to the processor, and may perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculation, the processor may load a command or data received from another component (e.g., a communication circuit) into a volatile memory, process the command or data stored in the volatile memory, and store the resulting data in a non-volatile memory. Here, at least one program may include instructions for obtaining at least one first content (120) and generating at least one second content (130) associated with the at least one first content (120) using a machine learning model (110).

[0054] The first content (120) input to the machine learning model (110) may include at least one of an image, an outline image associated with the image, a segmentation map associated with the image, a depth map associated with the image, bounding box information of an object included in the image, facial landmark information of a person included in the image, pose information of a person included in the image, or a prompt associated with the image. In addition, the second content (130) output through the machine learning model (110) may include at least one of an image, an IR image associated with the image, an image associated with the image and having a different domain style in at least a part of the image, a contour image associated with the image, a segmentation map associated with the image, a depth map associated with the image, bounding box information of an object included in the image, facial landmark information of a person included in the image, pose information of a person included in the image, tabular data including physical characteristic information of an object included in the image, a text sequence including physical characteristic information of an object included in the image, or a data set representing coordinate information of an object included in the image. In this case, the first content (120) and the second content (130) may be data that are at least partially different from each other. For example, at least one first content (120) input to the machine learning model (110) and at least one second content (130) output through the machine learning model (110) may be data that are at least partially different from each other.

[0055] In the present disclosure, at least one second content (130) output through the machine learning model (110) may include content and content information. That is, the decoder (116) of the machine learning model (110) may simultaneously generate content and content information based on at least one feature vector (114). At this time, the content may include at least one of an image (e.g., a still image or a moving image), an IR image associated with the image, an image associated with the image and having a different domain style of at least a portion of the image, or a data set (e.g., point cloud data) representing coordinate information of an object included in the image. In addition, the content information generated simultaneously with the content may be related to the content and may be different from the content. For example, the content information may include at least one of a contour image associated with the image, a segmentation map associated with the image, a depth map associated with the image, bounding box information of an object included in the image, facial landmark information of a person included in the image, pose information of a person included in the image, tabular data including physical characteristic information of an object included in the image, or a text sequence including physical characteristic information of an object included in the image. In this way, errors between content and content information can be minimized by simultaneously interacting with each other and being created within a single network.

[0056] FIG. 2 is a schematic diagram illustrating a configuration in which an information processing system (230) is connected to a plurality of user terminals (210_1, 210_2, 210_3) so as to be able to communicate with each other, in relation to data processing according to one embodiment of the present disclosure. The information processing system (230) may include system(s) capable of providing data processing services (e.g., content creation-based services). In one embodiment, the information processing system (230) may include one or more server devices and / or databases capable of storing, providing, and executing computer-executable programs (e.g., downloadable applications) and data related to the data processing services, or one or more distributed computing devices and / or distributed databases based on cloud computing services. For example, the information processing system (230) may include separate systems (e.g., servers) for data processing services.

[0057] Data processing services, etc. provided by the information processing system (230) can be provided to users through data processing applications, web browser applications, etc. installed on each of a plurality of user terminals (210_1, 210_2, 210_3).

[0058] A plurality of user terminals (210_1, 210_2, 210_3) can communicate with an information processing system (230) via a network (220). The network (220) can be configured to enable communication between the plurality of user terminals (210_1, 210_2, 210_3) and the information processing system (230). Depending on the installation environment, the network (220) can be configured as a wired network such as Ethernet, a wired home network (Power Line Communication), a telephone line communication device, and RS-serial communication, a wireless network such as a mobile communication network, WLAN (Wireless LAN), Wi-Fi, Bluetooth, and ZigBee, or a combination thereof. The communication method is not limited, and may include not only a communication method utilizing a communication network (e.g., a mobile communication network, wired Internet, wireless Internet, broadcasting network, satellite network, etc.) that the network (220) may include, but also short-range wireless communication between user terminals (210_1, 210_2, 210_3).

[0059] For example, multiple user terminals (210_1, 210_2, 210_3) can transmit data processing requests and commands related to user requests for data processing to an information processing system (230) via a network (220), and the information processing system (230) can receive them.

[0060] In FIG. 2, a mobile phone terminal (210_1), a tablet terminal (210_2), and a PC terminal (210_3) are illustrated as examples of user terminals, but are not limited thereto, and the user terminals (210_1, 210_2, 210_3) may be any computing device capable of wired and / or wireless communication and capable of installing and executing data processing applications, etc. For example, the user terminals may include smartphones, mobile phones, navigation devices, computers, laptops, digital broadcasting terminals, PDAs (Personal Digital Assistants), PMPs (Portable Multimedia Players), tablet PCs, game consoles, wearable devices, IoT (Internet of Things) devices, VR (virtual reality) devices, AR (augmented reality) devices, etc. In addition, although FIG. 2 illustrates three user terminals (210_1, 210_2, 210_3) communicating with the information processing system (230) via the network (220), this is not limited thereto, and a different number of user terminals may be configured to communicate with the information processing system (230) via the network (220).

[0061] FIG. 3 is a block diagram illustrating the internal configuration of a user terminal (210) and an information processing system (230) according to one embodiment of the present disclosure. The user terminal (210) may refer to any computing device capable of executing a data processing application and capable of wired / wireless communication, and may include, for example, a mobile phone terminal (210_1), a tablet terminal (210_2), a PC terminal (210_3) of FIG. 2 . As illustrated, the user terminal (210) may include a memory (312), a processor (314), a communication module (316), and an input / output interface (318). Similarly, the information processing system (230) may include a memory (332), a processor (334), a communication module (336), and an input / output interface (338). As illustrated in FIG. 3, the user terminal (210) and the information processing system (230) may be configured to communicate information and / or data via a network (220) using respective communication modules (316, 336). In addition, the input / output device (320) may be configured to input information and / or data to the user terminal (210) or output information and / or data generated from the user terminal (210) via the input / output interface (318).

[0062] The memory (312, 332) may include any non-transitory computer-readable recording medium. According to one embodiment, the memory (312, 332) may include a permanent mass storage device such as a read-only memory (ROM), a disk drive, a solid state drive (SSD), a flash memory, etc. As another example, a permanent mass storage device such as a ROM, an SSD, a flash memory, a disk drive, etc. may be included in the user terminal (210) or the information processing system (230) as a separate permanent storage device distinct from the memory. In addition, the memory (312, 332) may store an operating system and at least one program code (e.g., code for an application associated with a data processing service, etc.).

[0063] These software components may be loaded from a computer-readable recording medium separate from the memory (312, 332). This separate computer-readable recording medium may include a recording medium directly connectable to the user terminal (210) and the information processing system (230), and may include, for example, a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD / CD-ROM drive, a memory card, etc. As another example, the software components may be loaded into the memory (312, 332) through a communication module (316, 336) other than a computer-readable recording medium. For example, at least one program may be loaded into the memory (312, 332) based on a computer program (e.g., an application associated with a data processing service, etc.) that is installed by files provided by developers or a file distribution system that distributes installation files of applications through a network (220).

[0064] The processor (314, 334) may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. Instructions may be provided to the processor (314, 334) by a memory (312, 332) or a communication module (316, 336). For example, the processor (314, 334) may be configured to execute instructions received according to program code stored in a storage device such as the memory (312, 332).

[0065] The communication module (316, 336) may provide a configuration or function for the user terminal (210) and the information processing system (230) to communicate with each other via the network (220), and may provide a configuration or function for the user terminal (210) and / or the information processing system (230) to communicate with another user terminal or another system (e.g., a separate cloud system, etc.). For example, a request or data (e.g., a data processing request or data, etc.) generated by the processor (314) of the user terminal (210) according to a program code stored in a recording device such as a memory (312) may be transmitted to the information processing system (230) via the network (220) under the control of the communication module (316). Conversely, a control signal or command provided under the control of the processor (334) of the information processing system (230) can be received by the user terminal (210) through the communication module (316) of the user terminal (210) via the communication module (336) and the network (220).

[0066] The input / output interface (318) may be a means for interfacing with an input / output device (320). As an example, the input device may include a device such as a camera, a keyboard, a microphone, a mouse, etc., including an audio sensor and / or an image sensor, and the output device may include a device such as a display, a speaker, a haptic feedback device, etc. As another example, the input / output interface (318) may be a means for interfacing with a device that has a configuration or function integrated into one for performing input and output, such as a touch screen. In FIG. 3, the input / output device (320) is illustrated as not being included in the user terminal (210), but is not limited thereto and may be configured as a single device with the user terminal (210). In addition, the input / output interface (338) of the information processing system (230) may be a means for interfacing with a device (not shown) for input or output that is connected to the information processing system (230) or that the information processing system (230) may include. In FIG. 3, the input / output interface (318, 338) is illustrated as an element configured separately from the processor (314, 334), but is not limited thereto, and the input / output interface (318, 338) may be configured to be included in the processor (314, 334).

[0067] The user terminal (210) and the information processing system (230) may include more components than those shown in FIG. 3. However, it is not necessary to explicitly illustrate most of the conventional technical components. In one embodiment, the user terminal (210) may be implemented to include at least some of the input / output devices (320) described above. In addition, the user terminal (210) may further include other components such as a transceiver, a Global Positioning System (GPS) module, a camera, various sensors, a database, etc. For example, if the user terminal (210) is a smartphone, it may include components that a smartphone generally includes, and for example, various components such as an acceleration sensor, a gyro sensor, a microphone module, a camera module, various physical buttons, buttons using a touch panel, input / output ports, and a vibrator for vibration may be implemented to be further included in the user terminal (210).

[0068] According to one embodiment, the processor (314) of the user terminal (210) may be configured to operate a data processing application or a web browser application that provides a data processing service. At this time, program code associated with the application may be loaded into the memory (312) of the user terminal (210). While the application is operating, the processor (314) of the user terminal (210) may receive information and / or data provided from the input / output device (320) through the input / output interface (318) or may receive information and / or data from the information processing system (230) through the communication module (316), and may process the received information and / or data and store it in the memory (312). In addition, such information and / or data may be provided to the information processing system (230) through the communication module (316).

[0069] While the data processing application is running, the processor (314) may receive voice data, text, images, videos, etc. input or selected through input devices such as a camera, microphone, including a touch screen, keyboard, audio sensor, and / or image sensor connected to the input / output interface (318), and may store the received voice data, text, images, and / or videos in the memory (312) or provide them to the information processing system (230) through the communication module (316) and the network (220). In one embodiment, the processor (314) may receive user input input through the input device, and provide data / requests corresponding to the received user input to the information processing system (230) through the network (220) and the communication module (316).

[0070] The processor (314) of the user terminal (210) can output information and / or data by transmitting the information and / or data to an input / output device (320) through an input / output interface (318). For example, the processor (314) of the user terminal (210) can output the processed information and / or data through an output device (320), such as a display output capable device (e.g., a touch screen, a display, etc.) or a voice output capable device (e.g., a speaker).

[0071] The processor (334) of the information processing system (230) may be configured to manage, process, and / or store information and / or data received from multiple user terminals (210) and / or multiple external systems. Information and / or data processed by the processor (334) may be provided to the user terminal (210) via a communication module (336) and a network (220).

[0072] FIG. 4 is a diagram for explaining the configuration of a machine learning model (110) that generates content by connecting multiple contents in image format by channel according to one embodiment of the present disclosure. Referring to FIG. 4, the machine learning model (110) may include an encoder (112) that generates at least one feature vector (114) based on at least one first content (120) and a decoder (116) that generates at least one second content (130) based on the generated at least one feature vector (114). The decoder (116) may simultaneously generate content (410) and content information (420). For example, at least one second content (130) output through the machine learning model (110) may include content (410) and content information (420). In FIG. 4, a method for a decoder (116) to generate second content (130) including content (410) and content information (420) when both content (410) and content information (420) are content in image format is described. Here, the content in image format may include, for example, at least one of an image, an IR image, a contour image, a segmentation map, or a depth map.

[0073] The content (410) in the image format can be represented as a matrix. In addition, the content information (420) in the image format can also be represented as a matrix. Accordingly, the decoder (116) can generate third data represented as a third matrix by merging first data represented as a first matrix corresponding to the content (410) and second data represented as a second matrix corresponding to the content information (420). According to one embodiment, the decoder (116) can concatenate the first data represented as a first matrix and the second data represented as a second matrix channel-wise to generate third data represented as a third matrix, and output the generated third data as at least one second content (130). For example, when the first data is represented as an n-channel matrix and the second data is represented as an m-channel matrix, the decoder (116) can generate third data represented as an (n + m)-channel matrix.

[0074] According to one embodiment, the decoder (116) may adjust the size of at least one of the content (410) and the content information (420) if the content (410) and the content information (420) have different sizes (e.g., height and width). For example, the decoder (116) may resize or zero-pad at least one of the first data or the second data. Accordingly, the first matrix and the second matrix included in the third data generated by the decoder (116) may have the same dimensions and behavior.

[0075] FIG. 5 is a diagram for explaining a method for generating image-format content based on image-format content and text-format content according to one embodiment of the present disclosure. Referring to FIG. 5, a machine learning model (110) can be trained to simultaneously generate content (532) and content information (534). In FIG. 5, when image-format content (512) and text-format content (514) are used as input content (510) of the machine learning model (110) (e.g., first content (120) of FIGS. 1 and 4), a method for the machine learning model (110) to generate image-format content (530) (e.g., second content (130) of FIGS. 1 and 4) will be explained.

[0076] During the learning process, when image-format content (512) and text-format content (514) are input, the machine learning model (110) can extract at least one first feature vector from the image-format content (512) and at least one second feature vector from the text-format content (514). Then, the machine learning model (110) can be trained to output image-format content (530) based on the at least one first feature vector and the at least one second feature vector. For example, the machine learning model (110) can generate content (532) and content information (534) based on the at least one first feature vector and the at least one second feature vector. Then, the machine learning model (110) can be trained to generate third data expressed as a third matrix by connecting first data expressed as a first matrix corresponding to content (532) and second data expressed as a second matrix corresponding to content information (534) by channel, and output the generated third data as content (530) in image format.

[0077] The machine learning model (110) can output image-format content (530) based on image-format content (512) and text-format content (514) during the inference process. For example, the machine learning model (110) can extract at least one first feature vector from the image-format content (512) and at least one second feature vector from the text-format content (514). Then, the machine learning model (110) can generate content (532) and content information (534) based on the at least one first feature vector and the at least one second feature vector. Then, the machine learning model (110) can connect first data represented by a first matrix corresponding to the content (532) and second data represented by a second matrix corresponding to the content information (534) by channel to generate third data represented by a third matrix, and output the generated third data as image-format content (530).

[0078] In FIG. 5, when the content (512) in the image format used as the input content (510) is an outline image (e.g., a canny edge image) representing a driving scene, and the content (514) in the text format is a prompt describing the driving scene, a state is shown in which a 3-channel RGB image representing the driving scene as the content (532) and a 3-channel segmentation map associated with the RGB image representing the driving scene as the content information (534) are output as 6-channel images connected by channel.

[0079] FIG. 6 is a diagram illustrating a method for generating image-format content based on text-format content according to one embodiment of the present disclosure. Referring to FIG. 6, a machine learning model (110) can be trained to simultaneously generate content (632) and content information (634). In FIG. 6, when text-format content (610) is used as input content of the machine learning model (110) (e.g., first content (120) of FIG. 1 and FIG. 4), a method for the machine learning model (110) to generate image-format content (630) (e.g., second content (130) of FIG. 1 and FIG. 4) will be described.

[0080] During the learning process, when text-format content (610) is input, the machine learning model (110) can extract at least one feature vector from the text-format content (610). Then, the machine learning model (110) can be trained to output image-format content (630) based on the at least one feature vector. For example, the machine learning model (110) can generate content (632) and content information (634) based on the at least one feature vector. Then, the machine learning model (110) can be trained to connect first data represented by a first matrix corresponding to the content (632) and second data represented by a second matrix corresponding to the content information (634) by channel to generate third data represented by a third matrix, and output the generated third data as image-format content (630).

[0081] The machine learning model (110) can output image-format content (630) based on text-format content (610) during the inference process. For example, the machine learning model (110) can extract at least one feature vector from the text-format content (610). Then, the machine learning model (110) can generate content (632) and content information (634) based on the at least one feature vector. Then, the machine learning model (110) can connect first data represented by a first matrix corresponding to the content (632) and second data represented by a second matrix corresponding to the content information (634) by channel to generate third data represented by a third matrix, and output the generated third data as image-format content (630).

[0082] In Fig. 6, when the text-format content (610) used as input content is a prompt describing a driving scene, a 3-channel RGB image representing a driving scene as content (632) and a 3-channel segmentation map associated with the RGB image representing the driving scene as content information (634) are output as 6-channel images connected by channel.

[0083] FIG. 7 is a diagram illustrating a method for generating image-format content based on image-format content according to one embodiment of the present disclosure. Referring to FIG. 7, a machine learning model (110) can be trained to simultaneously generate content (732) and content information (734). In FIG. 7, when image-format content (710) is used as input content of the machine learning model (110) (e.g., first content (120) of FIG. 1 and FIG. 4), a method for the machine learning model (110) to generate image-format content (730) (e.g., second content (130) of FIG. 1 and FIG. 4) will be described.

[0084] During the learning process, when content (710) in the form of an image is input, the machine learning model (110) can extract at least one feature vector from the content (710) in the form of an image. Then, the machine learning model (110) can be trained to output content (730) in the form of an image based on the at least one feature vector. For example, the machine learning model (110) can generate content (732) and content information (734) based on the at least one feature vector. Then, the machine learning model (110) can be trained to connect first data represented by a first matrix corresponding to the content (732) and second data represented by a second matrix corresponding to the content information (734) by channel to generate third data represented by a third matrix, and output the generated third data as content (730) in the form of an image.

[0085] The machine learning model (110) can output image-format content (730) based on the image-format content (710) during the inference process. For example, the machine learning model (110) can extract at least one feature vector from the image-format content (710). Then, the machine learning model (110) can generate content (732) and content information (734) based on the at least one feature vector. Then, the machine learning model (110) can connect first data represented by a first matrix corresponding to the content (732) and second data represented by a second matrix corresponding to the content information (734) by channel to generate third data represented by a third matrix, and output the generated third data as image-format content (730).

[0086] In Fig. 7, when the content (710) in the image format used as input content is an RGB image representing a tank, a state is shown in which a 1-channel IR image representing a tank as content (732) and a 1-channel depth map associated with the RGB image representing the tank as content information (734) are output as a 2-channel image connected by channel.

[0087] FIG. 8 is a diagram illustrating a configuration of a machine learning model (110) including a plurality of decoders (116a, 116b) according to one embodiment of the present disclosure. Referring to FIG. 8, the machine learning model (110) may include an encoder (112) that generates at least one feature vector (114) based on at least one first content (120), a first decoder (116a) that generates a third content (810) based on the generated at least one feature vector (114), and a second decoder (116b) that generates a fourth content (820) based on the generated at least one feature vector (114). However, the number of decoders (116a, 116b) is not limited thereto. According to various embodiments, the machine learning model (110) may further include at least one other decoder (e.g., a third decoder) in addition to the first decoder (116a) and the second decoder (116b).

[0088] Each of the plurality of decoders (116a, 116b) can generate content or content information. For example, when the first decoder (116a) generates content (i.e., third content (810)), the second decoder (116b) can generate content information (i.e., fourth content (820)). In another example, when the first decoder (116a) generates content information (i.e., third content (810)), the second decoder (116b) can generate content (i.e., fourth content (820)). In yet another example, when the first decoder (116a) generates content (i.e., third content (810)), the second decoder (116b) can also generate content (i.e., fourth content (820)). As another example, when the first decoder (116a) generates content information (i.e., third content (810)), the second decoder (116b) can also generate content information (i.e., fourth content (820)).

[0089] The content generated by each of the plurality of decoders (116a, 116b) may include at least one of an image, an IR image associated with the image, an image associated with the image and having a different domain style of at least a portion of the image, or a data set representing coordinate information of an object included in the image (e.g., point cloud data). In addition, the content information generated by each of the plurality of decoders (116a, 116b) may include at least one of a contour image associated with the image, a segmentation map associated with the image, a depth map associated with the image, bounding box information of an object included in the image, facial landmark information of a person included in the image, pose information of a person included in the image, tabular data including physical characteristic information of an object included in the image, or a text sequence including physical characteristic information of an object included in the image.

[0090] At least two of the plurality of decoders (116a, 116b) can share and mix information with each other. For example, as illustrated in FIG. 8, when the plurality of decoders (116a, 116b) include two decoders, i.e., a first decoder (116a) and a second decoder (116b), the first decoder (116a) can generate third content (810) based on at least one feature vector generated based on input content and an intermediate vector received from the second decoder (116b). In addition, the second decoder (116b) can generate fourth content (820) based on at least one feature vector generated based on input content and an intermediate vector received from the first decoder (116a).

[0091] In one embodiment, the plurality of decoders (116a, 116b) can mix information using a cross attention algorithm. For example, the plurality of decoders (116a, 116b) can generate a query vector, a key vector, and a value vector from a first vector corresponding to the first information and a second vector corresponding to the second information. Then, the plurality of decoders (116a, 116b) can produce an attention score indicating a similarity between the query vector and the key vector. In one embodiment, the plurality of decoders (116a, 116b) can produce the attention score using a matrix multiplication operation (or a dot product between matrices). Then, the plurality of decoders (116a, 116b) can apply a softmax function to the attention score to produce an attention weight. Here, the application of the softmax function is to obtain a probability distribution in which the sum of all values ​​becomes 1, and each value obtained by applying the softmax function, i.e., the attention weight, can represent the importance of each key vector corresponding to the query vector. Then, multiple decoders (116a, 116b) can calculate a weighted sum for the value vector through a matrix multiplication operation (or inner product) of the attention weight and the value vector. At this time, the calculated weighted sum value can be a new vector in which the first information and the second information are mixed.

[0092] FIG. 9 is a diagram illustrating a method for mixing information centered on a decoder that generates image-format content according to one embodiment of the present disclosure. Referring to FIG. 9 , a machine learning model (110) may include multiple decoders (116a, 116b, 116c). At least two of the multiple decoders (116a, 116b, 116c) may share and mix information with each other. For example, as illustrated in FIG. 9, when a plurality of decoders (116a, 116b, 116c) include three decoders, i.e., a first decoder (116a), a second decoder (116b), and a third decoder (116c), the first decoder (116a) and the second decoder (116b) can share and mix information, and the second decoder (116b) and the third decoder (116c) can share and mix information. That is, the plurality of decoders (116a, 116b, 116c) can share and mix information centered around the second decoder (116b). In this case, the first decoder (116a) can generate the third content (910) based on at least one feature vector generated based on the input content and the intermediate vector received from the second decoder (116b). In addition, the second decoder (116b) can generate the fourth content (920) based on at least one feature vector generated based on the input content, the intermediate vector received from the first decoder (116a), and the intermediate vector received from the third decoder (116c). In addition, the third decoder (116c) can generate the fifth content (930) based on at least one feature vector generated based on the input content and the intermediate vector received from the second decoder (116b).

[0093] Each of the plurality of decoders (116a, 116b, 116c) can generate content or content information. In one embodiment, in the process of sharing and mixing information, a central decoder (e.g., the second decoder (116b)) generates content (e.g., the fourth content (920)), and the remaining decoders (e.g., the first decoder (116a) and the third decoder (116c)) generate content information (e.g., the third content (910) and the fifth content (930)). In some embodiments, in the process of sharing and mixing information, a central decoder (e.g., a second decoder (116b)) may generate content (e.g., a fourth content (920)), one of the remaining decoders (e.g., a first decoder (116a) and a third decoder (116c)) (e.g., the first decoder (116a)) may also generate content (e.g., a third content (910)), and another of the remaining decoders (e.g., a first decoder (116a) and a third decoder (116c)) (e.g., the third decoder (116c)) may generate content information (e.g., a fifth content (930)).

[0094] FIG. 10 is a diagram illustrating a method for mixing information associated with all decoders according to one embodiment of the present disclosure. Referring to FIG. 10 , a machine learning model (110) may include multiple decoders (116a, 116b, 116c). Furthermore, each of the multiple decoders (116a, 116b, 116c) may generate content or content information. At this time, the multiple decoders (116a, 116b, 116c) may share and mix information with each other. For example, as illustrated in FIG. 10, when a plurality of decoders (116a, 116b, 116c) include three decoders, i.e., a first decoder (116a), a second decoder (116b), and a third decoder (116c), the first decoder (116a) and the second decoder (116b) can share and mix information, the second decoder (116b) and the third decoder (116c) can share and mix information, and the first decoder (116a) and the third decoder (116c) can share and mix information. That is, all of the plurality of decoders (116a, 116b, 116c) can share and mix information with each other. In this case, the first decoder (116a) can generate third content (1010) based on at least one feature vector generated based on the input content, the intermediate vector received from the second decoder (116b), and the intermediate vector received from the third decoder (116c). In addition, the second decoder (116b) can generate fourth content (1020) based on at least one feature vector generated based on the input content, the intermediate vector received from the first decoder (116a), and the intermediate vector received from the third decoder (116c). In addition, the third decoder (116c) can generate fifth content (1030) based on at least one feature vector generated based on the input content, the intermediate vector received from the first decoder (116a), and the intermediate vector received from the second decoder (116b).

[0095] FIG. 11 is a diagram for explaining a layer in a decoder according to an embodiment of the present disclosure. Referring to FIG. 11, a machine learning model (110) may include a plurality of decoders (e.g., a first decoder (116a), a second decoder (116b), or a third decoder (116c)). At this time, at least one decoder among the plurality of decoders may include first information (X) associated with content to be generated by the decoder. 11 , X 21 , X 31 ) and the first layer generating the first information and the second information (X) received from the outside 12 , X 22 , X 32 ) may include a second layer that mixes the first information (X) associated with the content to be generated by the first decoder. For example, the first decoder may include first-1 information (X) associated with the content to be generated by the first decoder. 11 ) and the 2-1 information (X) received from the 1-1 layer and the 2nd decoder 12 ) may include a second-1 layer that mixes the first-1 information with the first-1 information. In addition, the second decoder may include first-2 information (X) associated with the content to be generated by the second decoder. 21 ) may include a first-2 layer generating the first decoder and a second-2 layer mixing the second-2 information (X22) received from the first decoder with the first-2 information.

[0096] According to one embodiment, the second layer may include a first preprocessor that preprocesses the first information, a second preprocessor that preprocesses the second information, and a mixin module that mixes the preprocessed first information and the preprocessed second information. For example, at least one decoder among the plurality of decoders may preprocess information within the decoder (the first information) and information external to the decoder (the second information), and then reflect the external information to the decoder through the mixin module. The mixin module may mix the information using, for example, a cross-attention algorithm.

[0097] According to one embodiment, the first information, the second information, and the third information (Y1, Y2, Y3) output through the decoder can each be expressed as a vector or matrix. In this case, the first preprocessor and the second preprocessor can match the dimensions and forms of the first information and the second information.

[0098] According to one embodiment, at least one of the plurality of decoders may generate a single content by concatenating a plurality of content in image format by channel. For example, at least one of the plurality of decoders may generate a third data represented by a third matrix by concatenating first data represented by a first matrix and second data represented by a second matrix by channel, and output the generated third data as content (e.g., the second content (130) of FIG. 4).

[0099] FIG. 12 is a diagram illustrating a method in which each of a plurality of decoders generates image-format content based on image-format content and text-format content according to one embodiment of the present disclosure. Referring to FIG. 12 , a machine learning model (110) including a plurality of decoders (116a, 116b) can be trained to simultaneously generate content (1232) and content information (1234). In FIG. 12, when image-format content (1212) and text-format content (1214) are used as input content (1210) of a machine learning model (110) (e.g., first content (120) of FIG. 1 and FIG. 8), a method for each of a plurality of decoders (116a, 116b) included in the machine learning model (110) to generate image-format content (1232, 1234) (e.g., second content (130) of FIG. 1 or third content (810), fourth content (820) of FIG. 8) will be described.

[0100] During the learning process, when content in image format (1212) and content in text format (1214) are input, the machine learning model (110) can extract at least one first feature vector from the content in image format (1212) and at least one second feature vector from the content in text format (1214). Then, the first decoder (116a) of the machine learning model (110) can be trained to output content in image format (1232) based on the at least one first feature vector and the at least one second feature vector, and the second decoder (116b) of the machine learning model (110) can be trained to output content information (1234) in image format based on the at least one first feature vector and the at least one second feature vector. At this time, the content information (1234) output by the second decoder (116b) may be content related to the content (1232) output by the first decoder (116a). In addition, the plurality of decoders (116a, 116b) may share and mix information (e.g., at least a portion of the content (1232) and at least a portion of the content information (1234)) with each other. For example, the plurality of decoders (116a, 116b) may mix the information using a cross-attention algorithm.

[0101] In the inference process, the machine learning model (110) may output image-format content (1232, 1234) based on image-format content (1212) and text-format content (1214), respectively, through a plurality of decoders (116a, 116b.) For example, the machine learning model (110) may extract at least one first feature vector from the image-format content (1212) and at least one second feature vector from the text-format content (1214). Then, the first decoder (116a) of the machine learning model (110) may generate content (1232) based on the at least one first feature vector and the at least one second feature vector, and the second decoder (116b) of the machine learning model (110) may generate content information (1234) based on the at least one first feature vector and the at least one second feature vector. At this time, at least a portion of the content (1232) and at least a portion of the content information (1234) may be mixed.

[0102] In FIG. 12, when the content (1212) in the image format used as the input content (1210) is a segmentation map representing a sailing scene and the content (1214) in the text format is a prompt describing the sailing scene, a state is shown in which the first decoder (116a) outputs an RGB image representing the sailing scene as the content (1232), and the second decoder (116b) outputs a depth map associated with the RGB image representing the sailing scene as the content information (1234).

[0103] FIG. 13 is a diagram illustrating a method in which multiple decoders generate image-format content and table-format content based on image-format content and text-format content, according to one embodiment of the present disclosure. Referring to FIG. 13 , a machine learning model (110) including multiple decoders (116a, 116b) can be trained to simultaneously generate content (1332) and content information (1334). In FIG. 13, when image-format content (1312) and text-format content (1314) are used as input content (1310) of a machine learning model (110) (e.g., the first content (120) of FIG. 1 and FIG. 8), a method for generating image-format content (1332) (e.g., the second content (130) of FIG. 1 or the third content (810) of FIG. 8) and table-format content (1334) (e.g., the second content (130) of FIG. 1 or the fourth content (820) of FIG. 8) by a plurality of decoders (116a, 116b) included in the machine learning model (110) is described.

[0104] During the learning process, when content in image format (1312) and content in text format (1314) are input, the machine learning model (110) can extract at least one first feature vector from the content in image format (1312) and at least one second feature vector from the content in text format (1314). Then, the first decoder (116a) of the machine learning model (110) can be trained to output content in image format (1332) based on the at least one first feature vector and the at least one second feature vector, and the second decoder (116b) of the machine learning model (110) can be trained to output content information (1334) in table format based on the at least one first feature vector and the at least one second feature vector. At this time, the content information (1334) output by the second decoder (116b) may be content related to the content (1332) output by the first decoder (116a). In addition, the plurality of decoders (116a, 116b) may share and mix information (e.g., at least a portion of the content (1332) and at least a portion of the content information (1334)) with each other. For example, the plurality of decoders (116a, 116b) may mix the information using a cross-attention algorithm.

[0105] In the inference process, the machine learning model (110) may output image-format content (1332) and table-format content information (1334) based on image-format content (1312) and text-format content (1314) through multiple decoders (116a, 116b). For example, the machine learning model (110) may extract at least one first feature vector from the image-format content (1312) and at least one second feature vector from the text-format content (1314). Then, the first decoder (116a) of the machine learning model (110) may generate content (1332) based on the at least one first feature vector and the at least one second feature vector, and the second decoder (116b) of the machine learning model (110) may generate content information (1334) based on the at least one first feature vector and the at least one second feature vector. At this time, at least a portion of the content (1332) and at least a portion of the content information (1334) may be mixed.

[0106] In Fig. 13, when the image-format content (1312) used as input content (1310) is a segmentation map representing a sailing scene, and the text-format content (1314) is a prompt describing the sailing scene, a first decoder (116a) outputs an RGB image representing the sailing scene as content (1332), and a second decoder (116b) outputs table data including physical characteristic information of an object included in the RGB image representing the sailing scene as content information (1334). The table data may include, for example, information such as sailing time, weather, and visibility.

[0107] FIG. 14 is a diagram illustrating a method for generating a plurality of image-format content and a table-format content based on image-format content and text-format content, using a plurality of decoders, according to one embodiment of the present disclosure. Referring to FIG. 14 , a machine learning model (110) including a plurality of decoders (116a, 116b, 116c) can be trained to simultaneously generate content (1434) and content information (1432, 1436). In FIG. 14, when image-format content (1412) and text-format content (1414) are used as input content (1410) of a machine learning model (110) (e.g., the first content (120) of FIG. 1 and FIG. 8), a method for generating a plurality of image-format contents (1434, 1436) (e.g., the second content (130) of FIG. 1 or the third content (810) of FIG. 8) and table-format contents (1432) (e.g., the second content (130) of FIG. 1 or the fourth content (820) of FIG. 8) by a plurality of decoders (116a, 116b, 116c) included in the machine learning model (110) is described.

[0108] During the learning process, when content in image format (1412) and content in text format (1414) are input, the machine learning model (110) can extract at least one first feature vector from the content in image format (1412) and extract at least one second feature vector from the content in text format (1414). Then, the first decoder (116a) of the machine learning model (110) may be trained to output first content information (1432) in a table format based on at least one first feature vector and at least one second feature vector, the second decoder (116b) of the machine learning model (110) may be trained to output content (1434) in an image format based on at least one first feature vector and at least one second feature vector, and the third decoder (116c) of the machine learning model (110) may be trained to output second content information (1436) in an image format based on at least one first feature vector and at least one second feature vector. At this time, the first content information (1432) output by the first decoder (116a) and the second content information (1436) output by the third decoder (116c) may be content related to the content (1434) output by the second decoder (116b). Additionally, the plurality of decoders (116a, 116b, 116c) may share and mix information with each other (e.g., at least a portion of the content (1434), at least a portion of the first content information (1432), and at least a portion of the second content information (1436). For example, the plurality of decoders (116a, 116b, 116c) may mix the information using a cross-attention algorithm.

[0109] In the inference process, the machine learning model (110) can output image-format content (1434), table-format first content information (1432), and image-format second content information (1436) based on image-format content (1412) and text-format content (1414) through multiple decoders (116a, 116b, 116c). For example, the machine learning model (110) can extract at least one first feature vector from image-format content (1412), and can extract at least one second feature vector from text-format content (1414). Then, the first decoder (116a) of the machine learning model (110) may generate first content information (1432) based on at least one first feature vector and at least one second feature vector, the second decoder (116b) of the machine learning model (110) may generate content (1434) based on at least one first feature vector and at least one second feature vector, and the third decoder (116c) of the machine learning model (110) may generate second content information (1436) based on at least one first feature vector and at least one second feature vector. At this time, at least a part of the content (1434), at least a part of the first content information (1432), and at least a part of the second content information (1436) may be mixed.

[0110] In FIG. 14, when the image-format content (1412) used as the input content (1410) is a segmentation map representing a sailing scene, and the text-format content (1414) is a prompt describing the sailing scene, the second decoder (116b) outputs an RGB image representing the sailing scene as the content (1434), the first decoder (116a) outputs table data including physical characteristic information of an object included in the RGB image representing the sailing scene as the first content information (1432), and the third decoder (116c) outputs a depth map associated with the RGB image representing the sailing scene as the second content information (1436). The table data may include, for example, information such as sailing time, weather, and visibility.

[0111] FIG. 15 is a diagram illustrating a method for generating content by connecting multiple pieces of image-format content by channel among multiple decoders based on image-format content and text-format content according to one embodiment of the present disclosure. Referring to FIG. 15 , a machine learning model (110) including multiple decoders (116a, 116b, 116c) can be trained to simultaneously generate content (1532a) and content information (1532b, 1534). In FIG. 15, when image-format content (1512) and text-format content (1514) are used as input content (1510) of a machine learning model (110) (e.g., first content (120) of FIG. 1 and FIG. 8), a method for generating image-format content (1532) (e.g., second content (130) of FIG. 1 or third content (810) of FIG. 8) and table-format content (1534) (e.g., second content (130) of FIG. 1 or fourth content (820) of FIG. 8) by a plurality of decoders (116a, 116b) included in the machine learning model (110) is described.

[0112] During the learning process, when content in image format (1512) and content in text format (1514) are input, the machine learning model (110) can extract at least one first feature vector from the content in image format (1512) and at least one second feature vector from the content in text format (1514). Then, the first decoder (116a) of the machine learning model (110) can generate content in image format (1532a) and first content information in image format (1532b) based on the at least one first feature vector and the at least one second feature vector, and the second decoder (116b) of the machine learning model (110) can generate second content information (1534) in table format based on the at least one first feature vector and the at least one second feature vector. Then, the first decoder (116a) may be trained to connect the first data represented by the first matrix corresponding to the content (1532a) and the second data represented by the second matrix corresponding to the first content information (1532b) by channel to generate third data represented by the third matrix, output the generated third data as content (1532) in an image format, and output the second content information (1534) in a table format. At this time, the first content information (1532b) generated by the first decoder (116a) and the second content information (1534) output by the second decoder (116b) may be content related to the content (1532a) output by the first decoder (116a). Additionally, the plurality of decoders (116a, 116b) may share and mix information with each other (e.g., at least a portion of the content (1532a), at least a portion of the first content information (1532b), and at least a portion of the second content information (1534). For example, the plurality of decoders (116a, 116b) may mix the information using a cross-attention algorithm.

[0113] In the inference process, the machine learning model (110) may output image-format content (1532a), first image-format content information (1532b), and second table-format content information (1534) based on image-format content (1512) and text-format content (1514) through multiple decoders (116a, 116b). For example, the machine learning model (110) may extract at least one first feature vector from the image-format content (1512), and may extract at least one second feature vector from the text-format content (1514). Then, the first decoder (116a) of the machine learning model (110) can generate content (1532a) in an image format and first content information (1532b) in an image format based on at least one first feature vector and at least one second feature vector, and the second decoder (116b) of the machine learning model (110) can generate second content information (1534) in a table format based on at least one first feature vector and at least one second feature vector. Then, the first decoder (116a) may connect the first data represented by the first matrix corresponding to the content (1532a) and the second data represented by the second matrix corresponding to the first content information (1532b) by channel to generate third data represented by the third matrix, output the generated third data as content (1532) in an image format, and the second decoder (116b) may output content information (1534) in a table format. At this time, at least a part of the content (1532a), at least a part of the first content information (1532b), and at least a part of the second content information (1534) may be mixed.

[0114] In FIG. 15, when the image-format content (1512) used as the input content (1510) is a segmentation map representing a sailing scene, and the text-format content (1514) is a prompt describing the sailing scene, the first decoder (116a) outputs a four-channel image in which a three-channel RGB image representing the sailing scene is connected by channel as the content (1532a) and a one-channel depth map associated with the RGB image representing the sailing scene is connected by channel as the first content information (1532b), and the second decoder (116b) outputs table data including physical characteristic information of an object included in the RGB image representing the sailing scene as the second content information (1534). The table data may include, for example, information such as sailing time, weather, and visibility.

[0115] FIG. 16 is a diagram illustrating a method in which multiple decoders generate image-format content and table-format content based on image-format content according to one embodiment of the present disclosure. Referring to FIG. 16 , a machine learning model (110) including multiple decoders (116a, 116b) can be trained to simultaneously generate content (1632) and content information (1634). In FIG. 16, when image-format content (1610) is used as input content of a machine learning model (110) (e.g., first content (120) of FIG. 1 and FIG. 8), a method for generating image-format content (1632) (e.g., second content (130) of FIG. 1 or third content (810) of FIG. 8) and table-format content (1634) (e.g., second content (130) of FIG. 1 or fourth content (820) of FIG. 8) by multiple decoders (116a, 116b) included in the machine learning model (110) is described.

[0116] During the learning process, when content (1610) in the form of an image is input, the machine learning model (110) can extract at least one feature vector from the content (1610) in the form of an image. Then, the first decoder (116a) of the machine learning model (110) can be trained to output content (1632) in the form of an image based on the at least one feature vector, and the second decoder (116b) of the machine learning model (110) can be trained to output content information (1634) in the form of a table based on the at least one feature vector. At this time, the content information (1634) output by the second decoder (116b) can be content related to the content (1632) output by the first decoder (116a). Additionally, the plurality of decoders (116a, 116b) may share and mix information with each other (e.g., at least a portion of the content (1632) and at least a portion of the content information (1634)). For example, the plurality of decoders (116a, 116b) may mix the information using a cross-attention algorithm.

[0117] In the inference process, the machine learning model (110) may output image-format content (1632) and table-format content information (1634) based on image-format content (1610) through multiple decoders (116a, 116b). For example, the machine learning model (110) may extract at least one feature vector from the image-format content (1610). Then, the first decoder (116a) of the machine learning model (110) may generate content (1632) based on the at least one feature vector, and the second decoder (116b) of the machine learning model (110) may generate content information (1634) based on the at least one feature vector. At this time, at least a portion of the content (1632) and at least a portion of the content information (1634) may be mixed.

[0118] In FIG. 16, when the content (1610) in the image format used as input content is a plurality of images taken from various directions by an autonomous vehicle, a first decoder (116a) outputs point cloud data based on the plurality of images taken from various directions by an autonomous vehicle as content (1632), and a second decoder (116b) outputs table data including physical characteristic information of an object included in the plurality of images taken from various directions by an autonomous vehicle as content information (1634). The table data may include, for example, information such as weather and rainfall.

[0119] FIG. 17 is a diagram for explaining a content generation method according to one embodiment of the present disclosure. Referring to FIG. 17, a processor of an electronic device (e.g., the electronic device (100) of FIG. 1) for generating content may, in step S1710, obtain at least one first content (e.g., the first content (120) of FIG. 1, the first content (120) of FIG. 4, or the first content (120) of FIG. 8). Here, the first content may include at least one of an image, a contour image associated with the image, a segmentation map associated with the image, a depth map associated with the image, bounding box information of an object included in the image, facial landmark information of a person included in the image, pose information of a person included in the image, or a prompt associated with the image.

[0120] In step S1720, the processor may generate at least one second content (e.g., the second content (130) of FIG. 1, the second content (130) of FIG. 4, or the third content (810), fourth content (820) of FIG. 8) associated with at least one first content using a machine learning model. Here, the machine learning model may include an encoder that generates at least one feature vector based on at least one first content and a decoder that generates at least one second content based on the generated at least one feature vector. For example, the machine learning model may be a generative AI model. Here, the second content may include at least one of an image, an IR image associated with the image, an image associated with the image and having a different domain style in at least a part of the image, a contour image associated with the image, a segmentation map associated with the image, a depth map associated with the image, bounding box information of an object included in the image, facial landmark information of a person included in the image, pose information of a person included in the image, tabular data including physical characteristic information of an object included in the image, a text sequence including physical characteristic information of an object included in the image, or a data set representing coordinate information of an object included in the image. In addition, the first content and the second content may be data that differs at least in part. For example, at least one first content input into a machine learning model and at least one second content output through the machine learning model may be data that differs at least in part.

[0121] The above flowchart and description are merely examples, and some embodiments may implement the system differently. For example, in some embodiments, the order of each step may be changed, some steps may be repeated, some steps may be omitted, or some steps may be added.

[0122] The above-described method may be provided as a computer program stored on a computer-readable recording medium for execution on a computer. The medium may be one that continuously stores a computer-executable program or one that temporarily stores it for execution or download. In addition, the medium may be various recording means or storage means in the form of a single or multiple hardware combinations, and is not limited to a medium directly connected to a computer system, but may also be distributed over a network. Examples of the medium may include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and those configured to store program instructions, including ROM, RAM, and flash memory. In addition, examples of other media may include recording or storage media managed by app stores that distribute applications, sites that supply or distribute various software, servers, etc.

[0123] The methods, operations, or techniques of the present disclosure may be implemented by various means. For example, these techniques may be implemented in hardware, firmware, software, or a combination thereof. Those skilled in the art will appreciate that the various exemplary logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various exemplary components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software will depend on the particular application and the design requirements imposed on the overall system. Those skilled in the art may implement the described functionality in various ways for each particular application, but such implementations should not be construed as departing from the scope of the present disclosure.

[0124] In a hardware implementation, the processing units used to perform the techniques may be implemented within one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, electronic devices, other electronic units designed to perform the functions described herein, a computer, or a combination thereof.

[0125] Accordingly, the various exemplary logical blocks, modules, and circuits described in connection with the present disclosure may be implemented or performed by any combination of a general-purpose processor, a DSP, an ASIC, an FPGA or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or those designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0126] In a firmware and / or software implementation, the techniques may be implemented as instructions stored on a computer-readable medium, such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, a compact disc (CD), a magnetic or marking data storage device, etc. The instructions may be executable by one or more processors and may cause the processor(s) to perform certain aspects of the functionality described herein.

[0127] When implemented in software, the techniques described above may be stored on or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer storage media and communication media, including any medium that facilitates transfer of a computer program from one place to another. Storage media may be any available media that can be accessed by a computer. By way of example, and not limitation, such computer-readable media can include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. Also, any connection is properly termed a computer-readable medium.

[0128] For example, if the software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, digital subscriber line, or wireless technologies such as infrared, radio, and microwave are included within the definition of media. Disk and disc, as used herein, includes compact discs, laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, whereas discs reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0129] A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium may be coupled to the processor such that the processor can read information from, and write information to, the storage medium. Alternatively, the storage medium may be integral to the processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. Alternatively, the processor and the storage medium may reside as discrete components in the user terminal.

[0130] While the embodiments described above have been described as utilizing aspects of the presently disclosed subject matter in one or more standalone computer systems, the present disclosure is not limited thereto and may be implemented in conjunction with any computing environment, such as a network or distributed computing environment. Furthermore, aspects of the present disclosure may be implemented in multiple processing chips or devices, and storage may be similarly affected across multiple devices. Such devices may include personal computers, network servers, and portable devices.

[0131] While the present disclosure has been described in connection with certain embodiments herein, various modifications and variations may be made without departing from the scope of the present disclosure, which would be apparent to those skilled in the art. Furthermore, such modifications and variations are intended to fall within the scope of the claims appended to this specification.

Claims

1. A content generation method performed by at least one processor, A step of obtaining at least one first content; and A step of generating at least one second content related to at least one first content using a machine learning model Including, The above machine learning model is, an encoder that generates at least one feature vector based on at least one first content, and A decoder that generates at least one second content based on at least one generated feature vector. A method of creating content, including:

2. In paragraph 1, At least one of the above first contents, A first image, an outline image associated with the first image, a segmentation map associated with the first image, a depth map associated with the first image, bounding box information of an object included in the first image, facial landmark information of a person included in the first image, pose information of a person included in the first image, or at least one of a prompt associated with the first image, At least one of the second contents, At least one of the first image, an IR image associated with the first image, a second image associated with the first image and having a different domain style in at least a portion of the first image, a contour image associated with the first image, a segmentation map associated with the first image, a depth map associated with the first image, bounding box information of an object included in the first image, facial landmark information of a person included in the first image, pose information of a person included in the first image, tabular data including physical characteristic information of an object included in the first image, a text sequence including physical characteristic information of an object included in the first image, or a data set representing coordinate information of an object included in the first image, A content creation method, wherein at least one of the first contents and the at least one second content are at least partially different data.

3. In paragraph 1, The above decoder, The first data expressed in the first matrix and the second data expressed in the second matrix are concatenated by channel to generate the third data expressed in the third matrix, A content generation method configured to output the generated third data as at least one second content.

4. In paragraph 3, A content generation method, wherein the first matrix and the second matrix included in the third data generated by the decoder have the same dimension and shape.

5. In paragraph 1, The at least one second content includes the 2-1 content and the 2-2 content that is different from the 2-1 content, The above decoder, A first decoder that generates the second-1 content based on at least one feature vector generated above, and A second decoder that generates the second-2 content based on at least one feature vector generated above. A method of creating content, including:

6. In paragraph 5, The first decoder generates the second-1 content based on the at least one feature vector and the intermediate vector received from the second decoder, A content generation method, wherein the second decoder generates the second-2 content based on the at least one feature vector and the intermediate vector received from the first decoder.

7. In paragraph 5, At least one decoder among the first decoder or the second decoder, A first layer generating first information associated with content to be generated by at least one decoder, and A second layer that mixes the first information and the second information received from the outside. A method of creating content, including:

8. In paragraph 5, The above first decoder, A first layer that generates first information associated with the second-1 content to be generated by the first decoder, and A second layer that mixes the second information received from the second decoder with the first information. Including, The above second decoder, A third layer that generates the second information associated with the second-2 content generated by the second decoder, and A fourth layer that mixes the first information received from the first decoder with the second information. A method of creating content, including:

9. In paragraph 5, The above first decoder, Connecting the first data expressed in the first matrix and the second data expressed in the second matrix by channel to generate the third data expressed in the third matrix, A content generation method configured to output the generated third data as the 2-1 content.

10. In paragraph 9, At least one decoder among the first decoder or the second decoder, A first layer generating first information associated with content to be generated by at least one decoder, and A second layer that mixes the first information and the second information received from the outside. A method of creating content, including:

11. A computer-readable non-transitory recording medium recording commands for executing a method according to any one of claims 1 to 10 on a computer.

12. In electronic devices, memory; and At least one processor connected to said memory and configured to execute at least one computer-readable program contained in said memory, At least one program above, Acquire at least one piece of first content, comprising instructions for generating at least one second content associated with the at least one first content using a machine learning model; The above machine learning model is, an encoder that generates at least one feature vector based on at least one first content, and A decoder that generates at least one second content based on at least one generated feature vector. An electronic device comprising:

Images