Method and device for embedding multimodal information

Abstract

An embodiment of the present disclosure provides a multimodal embedding method. The present method provides a multimodal embedding method. The present method includes the steps of: receiving target inputs to be embedded, which include an image and text, and generating an image feature embedding and a text feature embedding for the inputs; generating a fusion feature embedding by fusing the image feature embedding and the text feature embedding by using a preconfigured algorithm; generating a sub-goal for the fusion feature embedding by using an artificial intelligence model having been trained to output a sub-goal for a feature embedding input using training feature embeddings as a training data set.

Description

Multimodal information embedding method and device

[0001] The present invention relates to a multi-modal information embedding method and device, and more particularly, to a method and device for generating a sub-goal to be ultimately performed by fusing text feature embedding and image feature embedding.

[0002] Recently, various approaches for predicting sub-goals have emerged, but when integrating information from various modalities, they use a simple connection method to pass it to a transformer-based encoder, an RNN-based decoder, and a multi-layer perceptron, which has the problem of not sufficiently reflecting the relationships between various modalities.

[0003] Furthermore, performing high-level natural language commands is a complex problem, requiring the agent to plan and execute subgoals from a long-term perspective. To successfully accomplish these long-term goals, the agent must consider past data (historical data) when predicting the subgoals to be performed at each stage.

[0004] However, we are unable to effectively integrate and utilize historical data from various modalities.

[0005] Accordingly, there is a need for a method to integrate modalities by effectively fusing observed RGB, bounding boxes, sub-target history, and high-level instructions.

[0006] The present disclosure is intended to solve the problems of the above-mentioned prior art, and to provide a method and device for generating a sub-goal to be ultimately performed by fusing text feature embedding and image feature embedding.

[0007] The technical problems to be solved by the present invention are not limited to the technical problems described above, and other technical problems of the present invention can be derived from the following description.

[0008] As a technical means for solving the above-described technical problem, an embodiment according to a first aspect of the present disclosure provides a multi-modal embedding method. The method includes the steps of receiving embedding target inputs including images and texts, generating an image feature embedding and a text feature embedding for the inputs, merging the image feature embedding and the text feature embedding using a preset algorithm to generate a fusion feature embedding, and generating a sub-goal for the fusion feature embedding using an artificial intelligence model trained to output a sub-goal for the input feature embedding using learning feature embeddings as a learning data set.

[0009] As a technical means for solving the above-described technical problem, an embodiment according to a second aspect of the present disclosure provides a multi-modal embedding device. The device includes a communication module that is communicatively connected to a terminal, at least one processor, and a memory that is electrically connected to the processor and stores at least one code that is executed by the processor, wherein the memory stores a code that, when executed through the processor, causes the processor to receive embedding target inputs including images and texts, generate image feature embeddings and text feature embeddings for the inputs, fuse the image feature embeddings and the text feature embeddings using a preset algorithm to generate a fusion feature embedding, and generate a sub-goal for the fusion feature embedding using an artificial intelligence model trained to output a sub-goal for the input feature embedding using learning feature embeddings as a learning data set.

[0010] According to the present invention, it is possible to recognize one's own failure and replan the sub-goal by considering both text and images.

[0011] In addition, according to the present invention, the accuracy of command processing can be increased by considering both text and images.

[0012] The effects of the present invention are not limited to the effects described above, and include all effects understood from the following description.

[0013] FIG. 1 is a drawing illustrating a multi-modal embedding device and a terminal connected to the multi-modal embedding device according to one embodiment of the present invention.

[0014] FIG. 2 is a drawing showing a detailed configuration of the multi-modal embedding device illustrated in FIG. 1.

[0015] Figures 3 and 4 are drawings illustrating examples of multi-modal embedding.

[0016] FIG. 5 is a flowchart illustrating the sequence of a multi-modal embedding method according to another embodiment of the present invention.

[0017] Hereinafter, the present disclosure will be described in detail with reference to the attached drawings. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In addition, the attached drawings are only intended to facilitate understanding of the embodiments disclosed in the present specification, and the technical concepts disclosed in the present specification are not limited by the attached drawings. All terms, including technical and scientific terms, used herein should be interpreted as having meanings generally understood by a person of ordinary skill in the technical field to which the present disclosure pertains. Terms defined in the dictionary should be interpreted as having additional meanings consistent with the relevant technical literature and the present disclosure, and shall not be interpreted in an extremely ideal or restrictive sense unless otherwise defined.

[0018] To clearly explain the present disclosure in the drawings, parts irrelevant to the description have been omitted, and the size, shape, and appearance of each component shown in the drawings may vary. Identical / similar parts throughout the specification are designated with identical / similar drawing reference numerals.

[0019] The suffixes "module" and "part" used in the following description for components are assigned or used interchangeably solely for the convenience of writing the specification, and do not in themselves have distinct meanings or roles. Furthermore, in describing the embodiments disclosed herein, detailed descriptions of related known technologies have been omitted if they are deemed to obscure the gist of the embodiments disclosed herein.

[0020] Throughout the specification, when a part is said to be "connected (connected, in contact with, or coupled)" to another part, this includes not only cases where it is "directly connected (connected, in contact with, or coupled)" but also cases where it is "indirectly connected (connected, in contact with, or coupled)" with another member in between. Furthermore, when a part is said to "include (have or provide)" a certain component, this does not mean that it excludes other components, but rather that it may "include (have or provide)" other components, unless otherwise specifically stated.

[0021] As used herein, ordinal terms such as "first," "second," etc., are used solely to distinguish one component from another and do not limit the order or relationship of the components. For example, the first component of the present disclosure may be referred to as the "second component," and similarly, the second component may also be referred to as the "first component." As used herein, singular forms should be construed to include plural forms, unless explicitly stated otherwise.

[0022] FIG. 1 is a drawing illustrating a multi-modal embedding device and a terminal connected to the multi-modal embedding device according to one embodiment of the present invention.

[0023] Referring to FIG. 1, a multi-modal embedding device (100) can be connected to a terminal (200) via a preset communication network to transmit and receive information. The multi-modal embedding device (100) can be attached, mounted, and connected to a robot or module for processing natural language commands.

[0024] A multi-modal embedding device (100) receives embedding target inputs including images and text, and generates image feature embeddings and text feature embeddings for the inputs.

[0025] A multi-modal embedding device (100) generates a fusion feature embedding by fusing image feature embedding and text feature embedding using a preset algorithm.

[0026] A multi-modal embedding device (100) generates a sub-goal for a fusion feature embedding by using an artificial intelligence model trained to output a sub-goal for an input feature embedding using learning feature embeddings as a learning data set.

[0027] The multi-modal embedding device (100) can be implemented in the form of a server, a computing device, or various smart devices, and can operate in a cloud computing service model such as SaaS (Software as a Service), PaaS (Platform as a Service), or IaaS (Infrastructure as a Service). In addition, the multi-modal embedding device (100) can be constructed in the form of a private cloud, a public cloud, or a hybrid cloud system, but the scope of the present invention is not limited thereto.

[0028] The terminal (200) may be at least one of an observer's terminal for observing the process of the multi-modal embedding device (100), an instruction's terminal for transmitting a natural language command to the multi-modal embedding device (100), and a driving device such as a driving terminal or robot that receives a sub-goal from the multi-modal embedding device (100) and controls the operation according to the sub-goal.

[0029] However, it is not limited thereto, and the terminal (200) may mean any type of handheld-based wireless communication device such as a notebook, desktop, laptop, wireless communication device with portability and mobility, or a smartphone, tablet PC, etc. equipped with a web browser.

[0030] FIG. 2 is a drawing showing a detailed configuration of the multi-modal embedding device illustrated in FIG. 1.

[0031] Referring to FIG. 2, a multi-modal embedding device (100) may include a communication module (110), a processor (120), and a memory (130).

[0032] The communication module (110) may include a device including hardware and software necessary to transmit and receive signals such as control signals or data signals through wired or wireless connections with other network devices.

[0033] The communication module (110) can receive a natural language command from a terminal and transmit a subgoal for the natural language command to the terminal.

[0034] It is possible to receive risk information from at least one of a communication module and a plurality of sensors and provide a customized image to a terminal.

[0035] The processor (120) may include various types of devices that control and process data. The processor (120) may refer to a data processing device built into hardware that has a physically structured circuit to perform a function expressed in code or instructions included in a program.

[0036] In one example, the processor (120) may be implemented in the form of a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), etc., but the scope of the present invention is not limited thereto.

[0037] The processor (120) performs operations according to the code stored in the memory (130).

[0038] The memory (130) can store at least one of information and data input to the communication module (110), information and data required for functions performed by the processor (120), and data generated according to the execution of the processor (120).

[0039] Memory (130) should be interpreted as a general term for a non-volatile storage device that maintains stored information even when no power is supplied and a volatile storage device that requires power to maintain stored information. In addition to a volatile storage device that requires power to maintain stored information, memory (130) may include cloud storage, SSD, magnetic storage media, or flash storage media, but the scope of the present invention is not limited thereto.

[0040] The memory (130) is electrically connected to the processor (120) and stores at least one code to be executed by the processor (120). The memory (130) stores code that, when executed by the processor (120), causes the processor (120) to perform the following functions and procedures.

[0041] The memory (130) stores code that receives embedding target inputs including images and text, and causes image feature embeddings and text feature embeddings to be generated for the inputs. Here, the embedding target inputs may include at least one of an image for the current field of view, an image for at least one previous field of view, a list of previously performed subgoals, and a natural language command.

[0042] Image feature embeddings may include current view feature embeddings and previous view feature embeddings.

[0043] The current field of view feature embedding can be generated by analyzing an image for the current field of view to extract an RGB image and a bounding box image, and based on the RGB image and the bounding box image. For example, the memory (130) can store code that causes the RGB image and the bounding box image to be input into a Vision Transformer (ViT) encoder to extract the current field of view feature embedding.

[0044] The bounding box image may be an image resulting from masking the interior of the bounding box of objects detected by an object detection device connected to the communication module (110).

[0045] The previous view feature embedding can be generated by analyzing images for at least one previous view to extract an RGB image and a bounding box image, and based on the RGB image and the bounding box image. For example, the memory (130) can store code that causes the RGB image and the bounding box image to be input to the vision converter encoder to extract the previous view feature embedding.

[0046] The memory (130) may store code that causes the feature embeddings for past views to be generated by iterating over the views accumulated in the previous step.

[0047] The memory (130) may store code that causes the image feature embedding, which is a feature embedding of the final image data, to be generated based on the current field of view image feature embedding and the previous field of view feature embedding. For example, the memory (130) may store code that causes the image feature embedding, which is a feature embedding of the final image data, to be generated by combining the current field of view feature embedding and the previous field of view feature embedding.

[0048] The memory (130) may store code that causes the generation of a text feature embedding, which is a feature embedding of the final text data, based on a previously performed subgoal list and a natural language command. Here, the natural language command may be a sentence received from a terminal.

[0049] Text feature embeddings may include subgoal feature embeddings and natural language command feature embeddings. Subgoal feature embeddings may be generated by inputting a subgoal list into a preset encoder, and natural language command feature embeddings may be generated by inputting natural language commands into the preset encoder. For example, the memory (130) may store code that causes a previously performed subgoal list and natural language commands to be input into a BERT (Bidirectional Encoder Representations from Transformers) encoder to generate text feature embeddings.

[0050] The memory (130) stores code that causes a fusion feature embedding to be generated by fusing an image feature embedding and a text feature embedding using a preset algorithm. Here, the fusion feature embedding may include an image fusion feature embedding, a text fusion feature embedding, and a final fusion feature embedding.

[0051] The memory (130) may store code that causes the image feature embedding and the text feature embedding to be multiplied by a first attention weight matrix and a second attention weight matrix, respectively, to generate a query for the image feature embedding and a key for the text feature embedding.

[0052] For example, in the memory (130), for example, in the memory (130), an attention weight matrix (attention weight matrix) and a first attention weight matrix ( ) and the second attention weight matrix ( ) can be multiplied by each of them to generate a query (q) and a key (k).

[0053] The memory (130) may store code that causes the image feature embedding and the text feature embedding to be multiplied by the third attention weight matrix and the fourth attention weight matrix, respectively, to generate the first value and the second value.

[0054] For example, the memory (130) includes a third attention weight matrix, which is a different attention weight matrix for each of the image feature embedding and the text feature embedding. ) and the fourth attention weight matrix ( ) and multiply it by the first value ( ) and the second value ( ) can be stored in the code that causes it to be created.

[0055] The memory (130) may store code that causes attention scores to be calculated based on a query and a key, and a first result value and a second result value to be calculated by multiplying the attention scores by a first value and a second value.

[0056] For example, the memory (130) may store a code that causes the query and key to be inner-producted and normalized. The memory (130) may apply the normalized value to the softmax function to produce an attention score, and multiply the attention score by the first value and the second value to produce a first result value ( ) and the second result ( ) can be stored in the code that causes it to produce the result.

[0057] The memory (130) may store a code that causes the first result value and the second result value to be multiplied by the fifth attention weight matrix and the sixth attention weight matrix, respectively, to produce the image final feature embedding and the text final feature embedding.

[0058] For example, the memory (130) contains the first result value ( ) and the second result ( ) each with a different weight matrix, the fifth attention weight matrix ( ) and the 6th attention weight matrix ( ) can be stored to produce the final feature embeddings of each image and text data, which are the image final feature embeddings and the text final feature embeddings.

[0059] The memory (130) may store code that causes image fusion feature embedding to be generated based on the image final feature embedding and the text feature embedding, and that causes text fusion feature embedding to be generated based on the text final feature embedding and the image feature embedding.

[0060] For example, the memory (130) contains the image final feature embedding, which is the final feature embedding of the image data. ) and the text feature embedding (L), which is the text data feature embedding generated in the first step, are combined to form the image fusion feature embedding ( ) can be stored in the code that causes it to produce the result.

[0061] Additionally, in the memory (130), text fusion feature embedding ( ) is implemented based on the text final feature embedding and image feature embedding in the same way. ) can be stored in the code that causes it to produce the result.

[0062] A code that causes the process of inputting image fusion feature embedding into ViT and text fusion feature embedding into BERT and then fusion again to be repeated n (e.g., n=2) times can be stored in the memory (130).

[0063] The memory (130) may store code that causes the final fusion feature embedding to be generated based on the image fusion feature embedding and the text fusion feature embedding.

[0064] For example, the memory (130) contains the final fusion image feature embedding ( ) and final fusion text feature embeddings ( ) can be stored to generate a final fusion feature embedding (F). Here, the final fusion image feature embedding may be the result of inputting the first fusion feature embedding into ViT and fusing it n times, and the final fusion text feature embedding may be the result of inputting the second fusion feature embedding into BERT and repeating the process of fusing it again n times.

[0065] For example, the size of image feature embedding using an image with a width of 2 and a height of 2 can be (1, 5, 2). 1 represents the number of images, 5 is the result of multiplying the width and height of the image and adding 1 to it, which is the result of converting a 2-dimensional image into a 1-dimensional vector through the feature embedding process, and 2 can represent the dimension of the image feature embedding.

[0066] Here, the size of the text feature embedding may match the size of the image feature embedding (1, 5, 2). 1 may be the number of texts, as in the image. The size 5 in the text feature embedding may be the number of tokens present in the text. The number of tokens present in the text may have an arbitrary value depending on the length of the sentence, but padding tokens, which are empty tokens, may be added to generate the feature embedding so that the length is constant. As a result, the size may be 5, which is the same as the size of the image feature embedding. 2 may be the dimension of the text feature embedding.

[0067] When the dimensions of image feature embedding and text feature embedding are as above, the first attention weight matrix ( ) and the second attention weight matrix ( ) can all be (1, 2, 3). 1 can match the first size of the weight matrix to the number of images or the number of texts. 2 can match the second size of the weight matrix to the dimension of the image feature embedding or the dimension of the text feature embedding for matrix multiplication calculation. 3 can be any number that represents the intermediate dimension (hidden dimension) used when performing the attention process. Accordingly, the sizes of the query and key can both be (1, 5, 2) X (1, ​​2, 3) = (1, 5, 3).

[0068] The first value can be generated by multiplying the image feature embedding by a third attention matrix having the same size as the first attention weight matrix.

[0069] The second value can also be generated by multiplying the text feature embedding by a fourth attention matrix, which has the same size as the first attention weight matrix. The sizes of the first and second values ​​can also be (1, 5, 3), which is the same as the sizes of the query and key.

[0070] The attention scores required to obtain the first and second results can be obtained by taking the inner product of the query and the key and normalizing them. At this time, the size of the attention score can be calculated through the process of (1, 5, 3) X (1, ​​3, 5) = (1, 5, 5). Since the normalization process does not change the size, the first and second results obtained by multiplying the attention score by the first and second values ​​can have a size of (1, 5, 5) X (1, ​​5, 3) = (1, 5, 3).

[0071] The final image feature embedding can be generated by adding the image feature embedding to the product of the first result value and the fifth attention weight matrix.

[0072] Using the size of each matrix, the size of the final feature embedding of the image becomes (1, 5, 3) X (1, ​​3, 2) = (1, 5, 2), which can match the size of the image feature embedding.

[0073] The final text feature embedding can be generated by adding the text feature embedding to the product of the second output value and the sixth attention weight matrix. The final text feature embedding can also be generated with dimensions (1, 5, 2) through the same process.

[0074] Assuming there is only one image, the first dimension of each feature embedding and weight matrix can be omitted. Therefore, the overall computation process can proceed as follows.

[0075] The image feature embeddings are [[ 1.1667, -0.6327], [ 1.4648, -0.0113], [ 0.2565, 0.4545], [-0.2304, 0.7471], [-0.7542, -0.6391]], the text feature embeddings are [[-0.2141, 0.6769], [-0.0290, -0.0963], [ 0.1181, 0.4626], [-0.6527, 0.8820], [ 1.5034, -0.6778]], and the first attention weight matrix is ​​[[-0.5736, 1.1794, -0.1105], [-1.0603, -2.3911, -1.0558]] and the second attention weight matrix is ​​[[-0.2861, -0.0027, 0.0987], [-0.6047, -1.3930, -1.1405]], the query is [[ 1.1667, -0.6327], [ 1.4648, -0.0113], [ 0.2565, 0.4545], [-0.2304, 0.7471], [-0.7542, -0.6391]] X [[-0.5736, 1.1794, -0.1105], [-1.0603, -2.3911, -1.0558]] = [[ It can be calculated as [0.0016, 2.8888, 0.5390], [ -0.8282, 1.7547, -0.1500], [ -0.6290, -0.7843, -0.5082], [ -0.6600, -2.0582, -0.7633], [ 1.1102, 0.6388, 0.7581]].

[0076] Height is [[-0.2141, 0.6769], [-0.0290, -0.0963], [ 0.1181, 0.4626], [-0.6527, 0.8820], [ 1.5034, -0.6778]]X [[-0.2861, -0.0027, 0.0987], [-0.6047, -1.3930, -1.1405]]= [[-0.3480, -0.9424, -0.7932], [ 0.0666, 0.1343, 0.1070], [-0.3135, -0.6447, -0.5159], [-0.3465, It can be calculated as [-1.2267, -1.0704], [-0.0203, 0.9401, 0.9215]].

[0077] If the third attention weight is [[ 0.1411, -2.3910, -1.7138], [-2.1881, -0.1114, 1.4981]] and the fourth attention weight is [[-2.1174, 0.3043, -0.0425], [-0.2160, 0.6407, -0.5706]], then the first value is [[ 1.1667, -0.6327], [ 1.4648, -0.0113], [ 0.2565, 0.4545], [-0.2304, 0.7471], [-0.7542, -0.6391]] X [[ 0.1411, -2.3910, -1.7138], [-2.1881, -0.1114, 1.4981]]= [[ 1.5489, -2.7191, -2.9472], [ 0.2314, -3.5011, -2.5273], [-0.9583, -0.6639, 0.2413], [-1.6673, 0.4678, 1.5142], [ 1.2921, 1.8744, 0.3350]] can be calculated.

[0078] The second value is [[-0.2141, 0.6769], [-0.0290, -0.0963], [ 0.1181, 0.4626], [-0.6527, 0.8820], [ 1.5034, -0.6778]]X [[-2.1174, 0.3043, -0.0425], [-0.2160, 0.6407, -0.5706]]= [[ 0.3072, 0.3685, -0.3772], [ 0.0823, -0.0706, 0.0562], [-0.3501, 0.3323, -0.2690], [ 1.1915, It can be calculated as [0.3664, -0.4756], [-3.0368, 0.0232, 0.3229].

[0079] The attention scores are Softmax(([[ 0.0016, 2.8888, 0.5390], [ -0.8282, 1.7547, -0.1500], [ -0.6290, -0.7843, -0.5082], [ -0.6600, -2.0582, -0.7633], [ 1.1102, 0.6388, 0.7581]] [[-0.3480, 0.0666, -0.3135, -0.3465, -0.0203], [-0.9424, 0.1343, -0.6447, -1.2267, 0.9401], [-0.7932, 0.1070, -0.5159, -1.0704, 0.9215]])÷ 5)= [[0.0202, 0.0415, 0.0247, 0.0166, 0.0722], [0.0296, 0.0392, 0.0324, 0.0270, 0.0515], [0.0498, 0.0365, 0.0460, 0.0536, 0.0299], [0.0661, 0.0350, 0.0558, 0.0775, 0.0225], [0.0276, 0.0398, 0.0302, 0.0256, 0.0490]].

[0080] The first result is [[0.0202, 0.0415, 0.0247, 0.0166, 0.0722], [0.0296, 0.0392, 0.0324, 0.0270, 0.0515], [0.0498, 0.0365, 0.0460, 0.0536, 0.0299], [0.0661, 0.0350, 0.0558, 0.0775, 0.0225], [0.0276, 0.0398, 0.0302, 0.0256, 0.0490]]X [[ 1.5489, -2.7191, -2.9472], [ 0.2314, -3.5011, -2.5273], [-0.9583, -0.6639, 0.2413], [-1.6673, 0.4678, 1.5142], [ 1.2921, 1.8744, 0.3350]] = [[ 0.0827, -0.0736, -0.1091], [ 0.0454, -0.1301, -0.1204], [-0.0092, -0.2126, -0.1368], [-0.0432, -0.2611, -0.1450], [ 0.0438, -0.1308, -0.1197]] can be calculated.

[0081] The second result is [[0.0202, 0.0415, 0.0247, 0.0166, 0.0722], [0.0296, 0.0392, 0.0324, 0.0270, 0.0515], [0.0498, 0.0365, 0.0460, 0.0536, 0.0299], [0.0661, 0.0350, 0.0558, 0.0775, 0.0225], [0.0276, 0.0398, 0.0302, 0.0256, 0.0490]]X [[ 0.3072, 0.3685, -0.3772], [ 0.0823, -0.0706, 0.0562], [-0.3501, 0.3323, -0.2690], [ 1.1915, 0.3664, -0.4756], [-3.0368, 0.0232, 0.3229]]= [[-0.1984, 0.0205, 0.0034], [-0.1234, 0.0300, -0.0139], [-0.0248, 0.0514, -0.0450], [ 0.0278, 0.0694, -0.0676], [-0.1172, 0.0279, -0.0126]] can be calculated.

[0082] When the fifth attention weight is [[-0.7440, 1.2087], [-0.2449, -0.3979], [-1.5525, -0.3535]] and the sixth attention weight is [[-0.9226, 0.5855], [ 0.1653, 0.3733], [ 1.5265, -2.2627]], the image fusion feature embedding is [[ 0.0827, -0.0736, -0.1091], [ 0.0454, -0.1301, -0.1204], [-0.0092, -0.2126, -0.1368], [-0.0432, -0.2611, -0.1450], [ 0.0438, -0.1308, -0.1197]] -0.0963], [ 0.1181, 0.4626], [-0.6527, 0.8820], [ 1.5034, -0.6778]]= [[-0.0882, 0.8448], [ 0.1559, 0.0529], [ 0.3894, 0.5844], [-0.3314, It can be calculated as [0.9849], [1.6886, -0.5305]].

[0083] The text fusion feature embeddings are [[-0.1984, 0.0205, 0.0034], [-0.1234, 0.0300, -0.0139], [-0.0248, 0.0514, -0.0450], [ 0.0278, 0.0694, -0.0676], [-0.1172, 0.0279, -0.0126]]X [[-0.9226, 0.5855], [ 0.1653, 0.3733], [ 1.5265, -2.2627]]+ [[ 1.1667, -0.6327], [ 1.4648, -0.0113], [ 0.2565, 0.4545], [-0.2304, 0.7471], [-0.7542, -0.6391]] = [[ 1.3583, -0.7490], [ 1.5624, -0.0410], [ 0.2192, 0.5609], [-0.3479, 0.9423], [-0.6607, -0.6688]].

[0084] In the memory (130), a code is stored that causes a sub-goal for a fusion feature embedding to be generated using an artificial intelligence model trained to output a sub-goal for an input feature embedding using learning feature embeddings as a learning data set.

[0085] The memory (130) may store a code that causes the output of a subgoal for the final fusion feature embedding based on an artificial neural network.

[0086] For example, the memory (130) may store code that causes the final fusion feature embedding (F) to generate the next action to be performed, the target object, and the target location through three artificial neural networks. Here, the artificial neural networks may be multilayer perceptrons (MLPs), and may be an artificial neural network for the action to be performed, an artificial neural network for the target object, and an artificial neural network for the target location.

[0087] The memory (130) may store code that causes the sub-goal to be performed next based on three factors. The action to be performed may be selected from seven actions, such as PickuopObject, PutObject, OpenObject, CloseObject, ToggleObjectOn, ToggleObjectOff, and SliceObject. The target object may be selected from 108 objects existing in the home environment, and the target location may be selected from 38 objects on which other objects can be placed among the 108 objects.

[0088] Figures 3 and 4 are drawings illustrating examples of multi-modal embedding.

[0089] Referring to FIGS. 3 and 4, the multi-modal embedding device outputs a subgoal consisting of an action to be performed next from input data, a target object, and a target location (receptacle), and can be composed of a total of three stages.

[0090] In the first step, the multimodal embedding device can generate feature embeddings by taking as input the view currently observed by the agent, previously observed views, a list of previously performed subgoals, and finally, the natural language commands to be performed.

[0091] To this end, the multimodal embedding device can receive an RGB image of the agent's field of view and a corresponding bounding box image. The bounding box image may be an image obtained by masking the interior of the bounding box of objects detected by the object detection module.

[0092] Multimodal embedding device RGB image and bounding box image can be converted into feature embedding of current field of view through Vision Transformer (ViT) encoder.

[0093] A multimodal embedding device can repeat the same process for the agent's views accumulated in the previous step to generate feature embeddings for past views.

[0094] A multi-modal embedding device can generate a final image data feature embedding (V) by combining the feature embedding for the current view and the feature embedding for the past view.

[0095] The multi-modal embedding device can generate text data feature embeddings (L) by passing the previously performed subgoal list and natural language commands to the BERT encoder and then combining them.

[0096] In the second stage, the multi-modal embedding device can perform fusion of image data feature embedding and text data feature embedding according to the cross-attention mechanism.

[0097] The multi-modal embedding unit converts image data feature embedding (V) and text data feature embedding (L) into an attention weight matrix. and You can obtain the query and key by multiplying each.

[0098] At the same time, the multi-modal embedding unit uses different attention weight matrices for image data-specific embedding and text data-specific embedding. Multiply each by two values ​​( ) can be created.

[0099] After that, the multi-modal embedding unit takes the inner product of the query and the key, normalizes them, and applies the softmax function to produce the attention score with two values ​​( ) multiply each of them, and the resulting values ​​( ) with different weight matrices ( ) can be multiplied to generate the final feature embedding of each image and text data.

[0100] Multimodal embedding devices provide the final feature embedding of image data ( ) and the text data feature embedding (L) generated in the first step are combined to form the fusion image feature embedding ( ) can be created.

[0101] Multimodal embedding devices use the same method to fuse text feature embeddings ( ) can also be generated. The fusion image feature embedding obtained in this way goes through ViT, and the fusion text feature embedding goes through BERT, and the process of fusion again can be repeated n times (n=2 in this study). The fusion step can be expressed by the following mathematical expression 1.

[0102]

[0103]

[0104]

[0105]

[0106]

[0107]

[0108]

[0109] Here, q is a query, V is an image data feature embedding, Wq is a first attention weight matrix, k is a key, L is a text data feature embedding, and Wk is a second attention weight matrix. is the first value, is the third attention weight matrix, is the second value, is the fourth attention weight matrix, is the first result value, is the second result, is a fusion image feature embedding, is a fusion text feature embedding, is the fifth attention weight matrix, is the fifth attention weight matrix, and F can be the final fusion feature embedding.

[0110] In the third step, the multimodal embedding device can output the next subgoal to be performed based on the fused data.

[0111] The multi-modal embedding unit is used to embed the final fusion image feature ( ) and final fusion text feature embeddings ( ) can be concatenated to generate the final fusion feature embedding (F).

[0112] The final fusion feature embeddings are then processed through three multilayer perceptrons (MLPs) to generate the next action, target object, and target location. These three elements then form the next subgoal.

[0113] Multimodal embedding devices can recognize failures by considering both images and text.

[0114] For example, if a natural language command such as 'PUT A KNIFE IN A BLUE CONTAINER ONTO THE BLACK TABLE IN THE CORNER' is input, the multimodal embedding device can generate the first subgoal 'Pick up Knife'.

[0115] The multimodal embedding device can compare the first subgoal with the image of the current field of view to detect failure, and if not, generate the second subgoal, 'Navigate to Bowl'.

[0116] The multimodal embedding device can compare the second subgoal with the image of the current field of view to detect failure, and if not, generate the third subgoal, 'Put Knife Bowl'.

[0117] The multimodal embedding device can compare the third subgoal with the image of the current field of view to determine whether it has failed or not, and if not, can generate the fourth subgoal, 'Pick up Bowl'.

[0118] The multimodal embedding device can compare the fourth subgoal with the image of the current field of view to determine whether it has failed or not, and if not, can generate the fifth subgoal, 'Navigate to SideTable.'

[0119] The multimodal embedding device can determine that the current image of the field of view, which shows a knife and bowl on a stovetop, differs from the previous subgoal, "Navigate to SideTable," by comparing it to the fifth subgoal, "Navigate to SideTable." Consequently, the device can determine that the natural language command processing failed. The multimodal embedding device can then regenerate the sixth subgoal into the fifth subgoal, "Navigate to SideTable."

[0120] As another example, if the natural language command 'PUT A KNIFE IN A BLUE CONTAINER ONTO THE BLACK TABLE IN THE CORNER' is input, the multimodal embedding device can generate the first subgoal 'Pick up Knife'.

[0121] The multimodal embedding device can compare the first subgoal with the image of the current field of view to detect failure, and if not, generate the second subgoal, 'Navigate to Bowl'.

[0122] The multimodal embedding device can compare the second subgoal with the image of the current field of view to detect failure, and if not, generate the third subgoal, 'Put Knife Bowl'.

[0123] The multimodal embedding device can compare the third subgoal with the image of the current field of view to determine whether it has failed or not, and if not, can generate the fourth subgoal, 'Pick up Bowl.'

[0124] The multimodal embedding device can compare the fourth subgoal with the image of the current field of view to determine whether it has failed or not, and if not, can generate the fifth subgoal, 'Navigate to SideTable.'

[0125] The multimodal embedding device can determine that the current field of view image does not identify the location of the knife, and compare it with the fifth subgoal, "Move to the side table." This determines that the current field of view image and the previous subgoal's execution results are different. The device can then determine that the natural language command processing was successful, but that the knife was dropped, resulting in an unexpected failure. The multimodal embedding device can then generate the sixth subgoal, "Pick up Knife."

[0126] FIG. 5 is a flowchart illustrating the sequence of a multi-modal embedding method according to another embodiment of the present invention.

[0127] The multi-modal embedding method described below can be performed by the multi-modal embedding device or server described above with reference to FIGS. 1 to 4. Accordingly, the content of the embodiments of the present disclosure described above with reference to FIGS. 1 to 4 can be equally applied to the embodiments described below, and any overlapping content with the above description will be omitted. The steps described below do not necessarily have to be performed in order, and the order of the steps can be set in various ways, and the steps can be performed almost simultaneously.

[0128] Referring to FIG. 5, the multi-modal embedding method includes an image feature embedding and text feature embedding generation step (S100), a fusion feature embedding generation step (S200), and a sub-goal output step (S300).

[0129] The image feature embedding and text feature embedding generation step (S100) is a step of receiving embedding target inputs including images and texts, and generating image feature embedding and text feature embedding for the inputs.

[0130] The image feature embedding and text feature embedding generation step (S100) may include a step of receiving at least one of an image for a current field of view, an image for at least one previous field of view, a list of previously performed subgoals, and a natural language command, a step of analyzing the image for the current field of view to extract an RGB image and a bounding box image, and a step of generating a first image feature embedding, which is a feature embedding for the current field of view, based on the RGB image and the bounding box image, a step of analyzing the image for at least one previous field of view to extract an RGB image and a bounding box image, and a step of generating a second image feature embedding, which is a feature embedding for the current field of view, based on the RGB image and the bounding box image, a step of generating an image feature embedding, which is a feature embedding of final image data, based on the first image feature embedding and the second image feature embedding, and a step of generating a text feature embedding, which is a feature embedding of final text data, based on the list of previously performed subgoals and the natural language command.

[0131] The fusion feature embedding generation step (S200) is a step of generating a fusion feature embedding by fusing image feature embeddings and text feature embeddings using a preset algorithm. Here, the fusion feature embedding may include an image fusion feature embedding, a text fusion feature embedding, and a final fusion feature embedding.

[0132] The fusion feature embedding generation step (S200) includes a step of generating a query for the image feature embedding and a key for the text feature embedding by multiplying the image feature embedding and the text feature embedding by a first attention weight matrix and a second attention weight matrix, respectively, a step of generating a first value and a second value by multiplying the image feature embedding and the text feature embedding by a third attention weight matrix and a fourth attention weight matrix, respectively, a step of calculating an attention score based on the query and the key, and calculating a first result value and a second result value by multiplying the attention score by the first value and the second value, respectively, a step of calculating an image final feature embedding and a text final feature embedding by multiplying the first result value and the second result value by a fifth attention weight matrix and a sixth attention weight matrix, a step of calculating an image fusion feature embedding based on the image final feature embedding and the text feature embedding, and a step of calculating a text final feature embedding based on the text final feature embedding and the image feature embedding. The step may include generating a fusion feature embedding, and a step of combining the image fusion feature embedding and the text fusion feature embedding to generate a final fusion feature embedding.

[0133] The subgoal output step (S300) is a step for generating subgoals for fusion feature embeddings using an artificial intelligence model trained to output subgoals for input feature embeddings using learning feature embeddings as a learning data set. Here, the subgoal may include information about at least one of an action to be performed, a target object, and a target location.

[0134] Those skilled in the art will appreciate that the present disclosure can be easily modified into other specific forms based on the above description without changing the technical spirit or essential characteristics of the present disclosure. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. The scope of the present disclosure is indicated by the following claims, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be construed as being included in the scope of the present disclosure. The scope of the present application is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be construed as being included in the scope of the present application.

[0135] The form for carrying out the invention is substantially the same as the best form for carrying out the invention.

[0136] The present invention can be used to manufacture a robot that performs command processing, and thus has industrial applicability.

Claims

1. In a multi-modal embedding method performed by a multi-modal embedding device, a) A step of receiving embedding target inputs including images and texts, and generating image feature embeddings and text feature embeddings for the inputs; b) a step of generating a fusion feature embedding by fusing the image feature embedding and the text feature embedding using a preset algorithm; and c) A multi-modal embedding method comprising a step of generating a subgoal for the fusion feature embedding using an artificial intelligence model trained to output a subgoal for the input feature embedding using learning feature embeddings as a learning data set.

2. In paragraph 1, A multi-modal embedding method, wherein the above embedding target inputs include at least one of an image for a current view, an image for at least one previous view, a list of previously performed subgoals, and a natural language command.

3. In paragraph 2, A multi-modal embedding method, wherein the image feature embedding includes analyzing an image for the current field of view to extract an RGB image and a bounding box image, and a current field of view feature embedding generated based on the RGB image and the bounding box image, and analyzing an image for at least one previous field of view to extract an RGB image and a bounding box image, and a previous field of view feature embedding generated based on the RGB image and the bounding box image.

4. In paragraph 2, The above text feature embedding is, A multi-modal embedding method comprising a sub-goal feature embedding generated by inputting the above sub-goal list into a preset encoder and a natural language command feature embedding generated by inputting the above natural language command into the preset encoder.

5. In paragraph 1, The above fusion feature embedding includes image fusion feature embedding, text fusion feature embedding, and final fusion feature embedding. Step b) above, A step of generating a query for the image feature embedding and a key for the text feature embedding by multiplying the image feature embedding and the text feature embedding, respectively, by a first attention weight matrix and a second attention weight matrix; A step of generating a first value and a second value by multiplying the image feature embedding and the text feature embedding, respectively, by a third attention weight matrix and a fourth attention weight matrix; A step of calculating an attention score based on the query and the key, and multiplying the attention score by the first value and the second value, respectively, to calculate a first result value and a second result value; A step of multiplying the first result value and the second result value by the fifth attention weight matrix and the sixth attention weight matrix, respectively, to produce the image final feature embedding and the text final feature embedding; A step of calculating the text fusion feature embedding based on the image final feature embedding and the text feature embedding, and calculating the image fusion feature embedding based on the text final feature embedding and the image feature embedding; and A multi-modal embedding method comprising the step of combining the image fusion feature embedding and the text fusion feature embedding to generate the final fusion feature embedding.

6. In paragraph 1, A multimodal embedding method, wherein the above subgoal includes information about at least one of an action to be performed, a target object, and a target location.

7. Communication module that communicates with the terminal; at least one processor; and A memory electrically connected to the processor and storing at least one code to be executed by the processor, The above memory, when executed through the processor, causes the processor to: A multi-modal embedding device that receives embedding target inputs including images and text, generates image feature embeddings and text feature embeddings for the inputs, fuses the image feature embeddings and the text feature embeddings using a preset algorithm to generate fusion feature embeddings, and stores code that causes a sub-goal for the fusion feature embedding to be generated using an artificial intelligence model trained to output sub-goals for feature embeddings input using learning feature embeddings as a learning data set.

8. In paragraph 7, A multi-modal embedding device, wherein the above embedding target inputs include at least one of an image for a current field of view, an image for at least one previous field of view, a list of previously performed subgoals, and a natural language command.

9. In paragraph 8, The above image feature embedding is, A multi-modal embedding device comprising: an image for the current field of view that is analyzed to extract an RGB image and a bounding box image; a current field of view feature embedding generated based on the RGB image and the bounding box image; and an image for at least one previous field of view that is analyzed to extract an RGB image and a bounding box image; and a previous field of view feature embedding generated based on the RGB image and the bounding box image.

10. In paragraph 8, The above text feature embedding is, A multi-modal embedding device comprising a sub-goal feature embedding generated by inputting the above sub-goal list into a preset encoder and a natural language command feature embedding generated by inputting the above natural language command into the preset encoder.

11. In paragraph 7, The above fusion feature embedding includes image fusion feature embedding, text fusion feature embedding, and final fusion feature embedding. The above memory causes the processor to: A query for the image feature embedding and a key for the text feature embedding are generated by multiplying the image feature embedding and the text feature embedding, respectively, by a first attention weight matrix and a second attention weight matrix, and a first value and a second value are generated by multiplying the image feature embedding and the text feature embedding, respectively, by a third attention weight matrix and a fourth attention weight matrix, and an attention score is calculated based on the query and the key, and a first result value and a second result value are calculated by multiplying the attention score by the first value and the second value, respectively, and a fifth attention weight matrix and a sixth attention weight matrix are multiplied by the first result value and the second result value, respectively, to produce an image final feature embedding and a text final feature embedding, and the text fusion feature embedding is calculated based on the image final feature embedding and the text feature embedding, and the image final feature embedding is generated based on the text final feature embedding and the image feature embedding. A multi-modal embedding device that stores code that causes a fusion feature embedding to be produced and causes the image fusion feature embedding and the text fusion feature embedding to be combined to produce the final fusion feature embedding.

12. In paragraph 7, A multimodal embedding device, wherein the subgoal includes information about at least one of an action to be performed, a target object, and a target location.

Images