Method, computer device, and computer program for providing search results based on multimodal features - Patents.com

Abstract

To provide a method for providing a retrieval result based on a multi-modal feature, a computer device, and a computer program.SOLUTION: A method for providing a retrieval result based on a multi-modal feature includes the steps of: executing vector operation between attributes by a user query on a multi-modal embedded space; and providing a retrieval result corresponding to the user query with reference to the embedded vector acquired by the vector operation.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The following description relates to a technology for providing search results suited to a user's intent. [Background technology]

[0002] A search engine is software or a website that makes it easy to find materials on the Internet. Such search engines search databases using search terms entered by users, extract search results, and provide them to users.

[0003] As an example of a technique for providing such search results, Patent Document 1 (published on September 16, 2004) discloses a technique for generating a search result list for a search term using a search engine. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Korean Patent Publication No. 10-2004-0079604 Summary of the Invention [Problem to be solved by the invention]

[0005] It is able to understand multimodal input simultaneously and provide search results that match the user's intent. [Means for solving the problem]

[0006] A multimodal search method executed on a computer device, the computer device including at least one processor configured to execute computer-readable instructions contained in a memory, the multimodal search method including: performing, by the at least one processor, a vector operation between attributes based on a user query on a multimodal embedding space; and providing, by the at least one processor, search results corresponding to the user query based on an embedding vector obtained by the vector operation.

[0007] According to one aspect, the performing step may perform the vector operation using a correction function that takes into account a nonlinear error in the multimodal embedding space.

[0008] According to another aspect, the performing step may include estimating a correction function that takes into account a nonlinear error of the multimodal embedding space using each attribute vector in the multimodal embedding space, and performing a vector operation between attributes included in the user query by an approximation calculation using the correction function.

[0009] According to another aspect, the multimodal embedding space may be defined as a vector space that can be decomposed into a delta space having nonlinearity and a virtual vector space that can be linearly expressed.

[0010] According to another aspect, the performing step may include converting the multimodal embedding space into a vector space capable of being linearly represented, and performing vector operations between attributes included in the user query in the vector space.

[0011] According to another aspect, the providing step may restore the vector space to the multimodal embedding space, and provide as the search result at least one product having a vector within a certain distance in the multimodal embedding space from the embedding vector obtained by the vector operation.

[0012] According to another aspect, the performing step may include performing a vector operation corresponding to at least one image and at least one text input as the user query.

[0013] According to another aspect, the performing step may perform a vector operation excluding a specific attribute among attributes included in the user query by combining a vector corresponding to the specific attribute with an opposite direction vector.

[0014] According to another aspect, the performing step may include performing a vector operation to adjust a strength of a particular attribute among attributes included in the user query by a scalar product with a vector corresponding to the particular attribute.

[0015] According to yet another aspect, the performing may include performing a cross-modal feature vector manipulation between a text vector space and an image vector space using the same coordinate system.

[0016] A computer program for causing a computer to execute the multimodal search method is provided.

[0017] A computer device is provided, the computer device including at least one processor configured to execute computer-readable instructions contained in a memory, the at least one processor processing a process of performing a vector operation between attributes based on a user query on a multimodal embedding space, and a process of providing search results corresponding to the user query based on an embedding vector obtained by the vector operation. Effect of the Invention

[0018] According to an embodiment of the present invention, by simultaneously understanding multimodal input combining images and text, products that are more suited to a user's intent can be found, thereby improving search quality. [Brief description of the drawings]

[0019] [Figure 1] FIG. 1 is a diagram illustrating an example of a network environment in an embodiment of the present invention. [Diagram 2] FIG. 2 is a block diagram illustrating an example of a computing device according to an embodiment of the present invention. [Diagram 3] 1 is a flowchart illustrating an example of a product search method that can be executed by a computer device according to an embodiment of the present invention. [Figure 4] FIG. 2 is an exemplary diagram illustrating a multi-modal vector space according to an embodiment of the present invention. [Diagram 5] FIG. 11 is an illustrative diagram for explaining limitations of vector operations in an embodiment of the present invention. [Figure 6] FIG. 2 is an exemplary diagram illustrating a virtual vector space in which vector operations are possible in one embodiment of the present invention. [Figure 7] 11 is an exemplary diagram for explaining a vector operation result by error correction in the embodiment of the present invention. FIG. [Figure 8] FIG. 11 is an exemplary diagram illustrating a vector operation excluding attributes according to an embodiment of the present invention. [Figure 9]11 is an exemplary diagram illustrating a vector operation for adjusting the strength of an attribute according to an embodiment of the present invention; FIG. [Figure 10] FIG. 2 is an exemplary diagram for explaining vector operations based on vector entanglement in an embodiment of the present invention. [Figure 11] FIG. 2 is an exemplary diagram for explaining vector operations based on vector entanglement in an embodiment of the present invention. [Figure 12] FIG. 2 illustrates an example of a text-based query combination according to an embodiment of the present invention. [Figure 13] FIG. 2 illustrates an example of combining image and text queries in accordance with an embodiment of the present invention. [Figure 14] FIG. 13 is a diagram showing an example of a search screen in one embodiment of the present invention. [Figure 15] FIG. 13 is a diagram showing an example of a search screen in one embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0021] An embodiment of the present invention relates to a technology for providing search results suited to a user's intent.

[0022] Moreover, the embodiment of the present invention relates to a technique for combining images and text to enable a user to efficiently search for products with desired attributes.

[0023] In addition, an embodiment of the present invention relates to a technology for transforming a multimodal embedding of a product into a virtual space that can be linearly expressed, and for finding search results by vector operations.

[0024] Furthermore, the embodiment of the present invention relates to a technology that enables a search for a variety of combinations by freely adding or removing attributes using a virtual space where vector operations are possible.

[0025] An embodiment including the matters specifically disclosed in this specification can quickly and easily search for products that suit a user's intentions by performing vector operations that freely add or remove product attributes using a virtual space that linearly represents the multimodal embedding of the product.

[0026] The product search system according to the embodiment of the present invention may be realized by at least one computer device, and the product search method according to the embodiment of the present invention may be executed by at least one computer device included in the product search system. In this case, a computer program according to an embodiment of the present invention may be installed and executed in the computer device, and the computer device may execute the product search method according to the embodiment of the present invention according to the control of the executed computer program. The above-mentioned computer program may be recorded in a computer-readable recording medium in combination with the computer device to cause the computer to execute the product search method.

[0027] Fig. 1 is a diagram showing an example of a network environment in one embodiment of the present invention. The network environment in Fig. 1 shows an example including a plurality of electronic devices 110, 120, 130, 140, a plurality of servers 150, 160, and a network 170. Such Fig. 1 is merely an example for explaining the invention, and the number of electronic devices and the number of servers are not limited to those shown in Fig. 1. Furthermore, the network environment in Fig. 1 is merely an example of an environment applicable to this embodiment, and the environment applicable to this embodiment is not limited to the network environment in Fig. 1.

[0028] The electronic devices 110, 120, 130, and 140 may be fixed terminals or mobile terminals realized by computer devices. Examples of the electronic devices 110, 120, 130, and 140 include smartphones, mobile phones, navigation systems, personal computers (PCs), notebook PCs, digital broadcasting terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), and tablets. As an example, FIG. 1 shows a smartphone as an example of the electronic device 110, but in the embodiment of the present invention, the electronic device 110 may mean one of various physical computer devices that can communicate with other electronic devices 120, 130, and 140 and / or servers 150 and 160 via a network 170 using a substantially wireless or wired communication method.

[0029] The communication method is not limited, and may include not only a communication method using a communication network (for example, a mobile communication network, a wired Internet, a wireless Internet, a broadcasting network) that the network 170 can include, but also short-range wireless communication between devices. For example, the network 170 may include any one or more of a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), a broadband network (BBN), the Internet, and the like. Furthermore, the network 170 may include any one or more of a network topology including, but not limited to, a bus network, a star network, a ring network, a mesh network, a star-bus network, a tree or hierarchical network, and the like.

[0030] Each of the servers 150, 160 may be realized by one or more computer devices that communicate with the multiple electronic devices 110, 120, 130, 140 via the network 170 to provide instructions, codes, files, content, services, etc. For example, the server 150 may be a system that provides a service (such as a product search service, for example) to the multiple electronic devices 110, 120, 130, 140 connected via the network 170.

[0031] 2 is a block diagram showing an example of a computer device according to an embodiment of the present invention. Each of the electronic devices 110, 120, 130, and 140 and the servers 150 and 160 described above may be realized by a computer device 200 shown in FIG.

[0032] Such a computer device 200 may include a memory 210, a processor 220, a communication interface 230, and an input / output interface 240, as shown in FIG. 2. The memory 210 is a computer-readable recording medium and may include a random access memory (RAM), a read only memory (ROM), and a permanent mass storage device such as a disk drive. Here, a permanent mass storage device such as a ROM or a disk drive may be included in the computer device 200 as a separate permanent storage device separate from the memory 210. The memory 210 may also store an operating system and at least one program code. Such software components may be loaded into the memory 210 from a computer-readable recording medium separate from the memory 210. Such separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD / CD-ROM drive, a memory card, etc. In another embodiment, the software components may be loaded into the memory 210 through a communication interface 230 that is not a computer-readable recording medium. For example, the software components may be loaded into the memory 210 of the computing device 200 based on a computer program installed by a file received over the network 170 .

[0033] The processor 220 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. The instructions may be provided to the processor 220 by the memory 210 or the communication interface 230. For example, the processor 220 may be configured to execute instructions received according to program code stored in a storage device such as the memory 210.

[0034] The communication interface 230 may provide a function for the computer device 200 to communicate with other devices (for example, the above-mentioned recording device) via the network 170. For example, requests, instructions, data, files, etc. generated by the processor 220 of the computer device 200 according to program codes recorded in a recording device such as the memory 210 may be transmitted to other devices via the network 170 under the control of the communication interface 230. Conversely, signals, instructions, data, files, etc. from other devices may be received by the computer device 200 via the communication interface 230 of the computer device 200 via the network 170. The signals, instructions, data, etc. received via the communication interface 230 may be transmitted to the processor 220 or the memory 210, and the files, etc. may be recorded on a recording medium (the above-mentioned permanent recording device) that the computer device 200 may further include.

[0035] The input / output interface 240 may be a means for interfacing with the input / output device 250. For example, the input device may include a device such as a microphone, a keyboard, or a mouse, and the output device may include a device such as a display and a speaker. As another example, the input / output interface 240 may be a means for interfacing with a device in which functions for input and output are integrated into one, such as a touch screen. The input / output device 250 may be configured as a single device together with the computer device 200.

[0036] Also, in other embodiments, computing device 200 may include fewer or more components than those of Figure 2. However, most prior art components need not be explicitly illustrated. For example, computing device 200 may be implemented to include at least some of the input / output devices 250 described above, and may further include other components such as a transceiver, a database, etc.

[0037] Specific embodiments of a method and apparatus for providing search results based on multimodal features are described below.

[0038] FIG. 3 is a flowchart illustrating an example of a product search method that can be executed by a computer device according to an embodiment of the present invention.

[0039] The product search method according to the present embodiment may be executed by the above-mentioned computer device 200. In this case, the processor 220 of the computer device 200 may be realized to execute control instructions according to the code of an operating system and the code of at least one program included in the memory 210. Here, the processor 220 may control the computer device 200 so that the computer device 200 executes steps 310 to 330 included in the product search method of FIG. 3 according to the control instructions provided by the code recorded in the computer device 200.

[0040] Referring to FIG. 3, at step 310, the processor 220 may convert the vector space onto which the multimodal embedding for the product is mapped (hereinafter referred to as the “multimodal vector space”) into a vector space that can be linearly represented (hereinafter referred to as the “virtual attribute vector (VAV) space”).

[0041] As shown in Figure 4, a multimodal vector space 400 maps images and text of the same product into the same vector space. The multimodal vector space 400 is decomposed into two distinct vector spaces with the same coordinate system: a text vector space 410 and an image vector space 420.

[0042] In general, since deep learning models have inherent non-linearity, products found by the vector sum of attributes in a vector space may differ from products found by vectors generated by combining character strings. For example, as shown in Figure 5, the sum of a feature vector indicating "Air Max 97" and a feature vector indicating "light blue" is different from the feature vector corresponding to "light blue Air Max 97", so a product desired by a user cannot be found by a simple vector sum.

[0043] On the other hand, in some deep learning models, normalization may be required so that the magnitude of the feature vector is always constant (for example, so that the magnitude is always 1). In this specification, for convenience of explanation, normalization is shown to have been performed, but normalization is not necessarily required.

[0044] In this embodiment, the multimodal vector space 400 may be decomposed into a delta space 610 having anisotropy and nonlinearity and a VAV space 620 that can be linearly expressed, as shown in FIG. 6, and the decomposed delta space 610 and VAV space 620 may be recombined to restore the multimodal vector space 400.

[0045] A text encoder f(x) trained by multimodal training corresponds to a function that maps a given text x to a multimodal vector space 400.

[0046] When mapping product attributes to the multimodal vector space 400, if they can be expressed linearly, they can be expressed as the sum of each attribute, such as [f(Air Max 97 light blue) = f(Air Max 97) + f(light blue)].

[0047] However, the vectors used in deep learning are vectors in a nonlinear, high-dimensional, multimodal space, and therefore cannot be completely expressed by a linear combination of each vector. In other words, f(Air Max 97 light blue) ≠ f(Air Max 97) + f(light blue).

[0048] When the multimodal embedding is transformed into an arbitrary space, a linear representation is possible. In this embodiment, the arbitrary space is defined as the VAV space 620.

[0049] A vector on the multimodal vector space 400 can be expressed as a vector on the VAV space 620, which can be linearly expressed, and a vector on the delta space 610, which cannot.

[0050]

number

[0051] The present invention may define a VAV space 620 in which delta (δ) can be approximated by a constant epsilon (ε).

[0052]

number

[0053] Therefore, by correcting the error due to epsilon (ε), the embedding vector can be directly approximated linearly.

[0054]

number

[0055] Referring again to FIG. 3, in step 320, the processor 220 may perform vector operations between attributes according to the user query, utilizing the VAV space 620 that allows linear calculations between attributes.

[0056] If F is a function that maps text X indicating any attribute to the multimodal vector space 400, then the embedding of text X is expressed as F(X).

[0057] Assuming that the vector sum in the embedding space is valid, the relationship F(A+B)=F(A)+F(B) must hold between two attributes A and B, but in a vector space with inherent nonlinearity, the vector sum does not hold [F(A+B)≠F(A)+F(B)]. Because the vector sum in the embedding space is not valid, the vector difference does not hold either [F(A+B)-F(A)≠F(B)].

[0058] In this embodiment, a correction function (Δ) is applied to establish vector operations in the embedding space.

[0059] F(A+B)=F(A)+F(B)+Δ(θ) F(A+B)-F(A)=F(B)+Δ(θ)

[0060] Here, if the correct answer to a vector operation for an arbitrary attribute is defined as (ε), the following equation holds:

[0061] ε A+B =F(A)+F(B)+Δ(θ)

[0062] ε (A+B)-A =F(A+B)-F(A)+Δ(θ)

[0063] For example, the correction function (Δ) corresponding to specific attributes A and B can be estimated as follows:

[0064] Δ(θ)=F(A+B)-(F(A)+F(B))

[0065] Here, Δ(θ) can be calculated for various attribute vector operation values, and Δ(θ) is approximated using the average value. TIFF0007632783000004.tif32 can be obtained, and this can be used to approximate vector operations.

[0066]

number

[0067]

number

[0068] The processor 220 executes vector operations in the VAV space 620 that allows linear expression, and at this time, it becomes possible to execute linear vector operations between attributes by approximation calculations using a correction function (Δ).

[0069] Since linear calculation with error correction is possible for vector operations between attributes, for example, as shown in Figure 7, the sum of the feature vector indicating "Air Max 97" and the feature vector indicating "light blue" will approximate the feature vector corresponding to "light blue Air Max 97" through error correction.

[0070] In step 330, the processor 220 may restore the VAV space 620 for linear vector operation to the multimodal vector space 400 and provide search results corresponding to the user query based on the embedding vector obtained by the vector operation in the VAV space 620. In the VAV space 620 where vector operation is possible, a plurality of attributes can be combined using a vector sum, and a specific attribute can be removed using a vector difference. The processor 220 may provide at least one product having a vector within a certain distance from the embedding vector obtained by the vector operation in the VAV space 620 among the products in the multimodal vector space 400 as a search result. Therefore, in this embodiment, a product with a desired attribute can be quickly and easily searched for by only the vector operation (vector sum and / or vector difference) on the embedding space for a large number of attributes.

[0071] FIG. 8 is an exemplary diagram illustrating a vector operation excluding attributes according to an embodiment of the present invention.

[0072] 8, when trying to remove a specific attribute "purple" from a product "short-sleeved, A-line, purple tiered loose-fitting dress," the processor 220 may obtain a vector 804 corresponding to the product "short-sleeved, A-line, tiered loose-fitting dress" from which the attribute "purple" has been removed, by performing a vector operation of adding a vector 803 in the opposite direction of a vector 802 indicating the attribute "purple" to a vector 801 indicating this product in the VAV space 620. After the vector operation, the VAV space 620 can be restored to the multimodal vector space 400, thereby obtaining the same effect as when feature vector manipulation is performed.

[0073] FIG. 9 is an exemplary diagram illustrating a vector operation for adjusting the strength of an attribute according to an embodiment of the present invention.

[0074] A vector may be broken down into multiple components. For example, a scalar product may be used to adjust the strength of a particular attribute. That is, a vector operation is also possible where a vector of a particular attribute is multiplied by a number.

[0075] 9, the processor 220 may obtain a vector 903 corresponding to a "light blue Air Max 97" by combining a vector 901 indicating the attribute "Air Max 97" and a vector 902 indicating the attribute "light blue" in the VAV space 620. When the user's intention to search for a sky blue that is darker than the "sky blue Air Max 97" is recognized, a vector 904 corresponding to "sky blue x 2 Air Max 97" in which the intensity of the attribute "sky blue" is adjusted can be obtained in the VAV space 620 by a scalar product with the vector 902 indicating the attribute "sky blue".

[0076] 10 and 11 are exemplary diagrams for explaining vector operations based on vector entanglement in one embodiment of the present invention.

[0077] As described with reference to FIG. 4, the multimodal vector space 400 may be decomposed into a text vector space 410 and an image vector space 420. Similarly, the VAV space 620 may be divided into a text VAV space 621 and an image VAV space 622 as shown in FIG. 10. The text VAV space 621 and the image VAV space 622 use the same coordinate space system, so cross-modal feature vector manipulation is possible. The text vector corresponding to "sky blue Air Max 97" on the text VAV space 621 and the image vector corresponding to "sky blue Air Max 97" on the image VAV space 622 may have the same coordinates.

[0078] 11, the processor 220 may change attributes between the text VAV space 621 and the image VAV space 622 by cross-modal feature vector operations. As an example, the processor 220 may change image attributes by operating a text vector matched on the text VAV space 621 for a specific image vector on the image VAV space 622. For example, when a user's intention to search for a product including the attribute "light green" instead of the attribute "purple" in the product "short-sleeved, A-line, loose-fitting purple tiered dress" is recognized, a text vector corresponding to "short-sleeved, A-line, loose-fitting light green tiered dress" may be searched for in the text VAV space 621 by vector operations between attributes, and an image vector of "short-sleeved, A-line, loose-fitting light green tiered dress" matched on the image VAV space 622 may be searched for.

[0079] In this way, by defining the multimodal embedding of a product as a virtual attribute vector in the VAV space 620 and a vector having anisotropy and nonlinearity in the delta space 610, it is possible to find the correct answer in the multimodal vector space 400. Since linear vector operations are possible using the VAV space 620, it is possible to perform not only quick searches but also to easily respond to changes in the model with consistency. At this time, by defining the objective function so that delta (δ) is small, it is possible to generate a high-performance and robust model.

[0080] The processor 220 may use the natural language expression as a query to search for products desired by the user, and may provide a search with attribute transformation that adds or removes characteristic attributes according to the text.

[0081] For example, as shown in FIG. 12, it is possible to find products with attributes desired by the user by performing calculations to remove or add specific attributes, such as products corresponding to the query "old school rainbow hi-tops," products corresponding to a query obtained by removing the attribute "rainbow" from the query "old school rainbow hi-tops," and products corresponding to a query obtained by removing the attribute "rainbow" from the query "old school rainbow hi-tops" and adding the attribute "red."

[0082] In addition, the processor 220 may support voice input for natural language queries. In this case, a search may be provided by attribute conversion such as by modifying search terms through a natural language search based on voice input. For example, a user may issue a natural language command including an operation on a desired attribute, such as "Look for sneakers with a white Mickey Mouse design. Oh, not 'Mickey' but 'Donald'", or a specific command such as "What code do you think I should wear? Besides red" to exclude a specific attribute.

[0083] Speech input can be processed by recognizing the speech input and then performing natural language processing in the same way as text search (e.g., converting the speech input to text and then applying it to a shopping search model), or by embedding the speech signal itself in a multimodal space (e.g., the pair <speech, product name text, product image> is learned together and embedded in a multimodal space).

[0084] The processor 220 may also search for a product desired by a user by using a multimodal input consisting of an image and a text as a query, and may provide a search by performing an interaction between the image and the text input as the query.

[0085] For example, as shown in FIG. 13, when a text "leopard print" including a subtraction operator is input as a query together with a specific image, products from which the attribute "leopard print" has been removed may be searched for based on features acquired by vector operation between the corresponding image feature and the attribute "leopard print". When a text "leopard print" including a subtraction operator and a text "red" including an addition operator are input as a query together with a specific image, products from which the attribute "leopard print" has been removed and the attribute "red" has been added may be searched for based on features acquired by vector operation between the corresponding image feature and the attributes "leopard print" and "red". When a text "leopard print" including a subtraction operator, a text "red" including an addition operator, and a text "star" including a subtraction operator are input as a query together with a specific image, products from which the attributes "leopard print" and "star" have been removed and the attribute "red" has been added may be searched for based on features acquired by vector operation between the corresponding image feature and the attributes "leopard print", "red", and "star".

[0086] 14 and 15 are diagrams showing examples of a search screen in one embodiment of the present invention.

[0087] In this embodiment, after selecting an image to apply to a query, products can be searched for by combining the query with additional text search terms.

[0088] For example, FIG. 14 shows an image search screen 1410 that provides search results for a query in the form of an image. The image search screen 1410 may include a multimodal search layer 1420 for adding a search term. When a search term is to be added on the image search screen 1410, the multimodal search layer 1420 may be displayed by sliding up. When a search term is entered on the multimodal search layer 1420, products with attributes desired by the user may be provided as search results 1421 in combination with the image entered on the image search screen 1410.

[0089] In this embodiment, products can be searched for by combining queries in which an image is selected from the search results for a text query and / or a text search term is additionally entered.

[0090] For example, FIG. 15 illustrates an integrated search screen 1510 that provides search results for a text-based query. A floating button 1511 for moving to additional searches may be displayed on the integrated search screen 1510. In this case, the floating button 1511 may be displayed at an appropriate position at an appropriate time, taking into consideration the user experience. For example, the floating button 1511 may be displayed on the integrated search screen 1510 from the time when the topmost area of ​​a specific collection (e.g., a shopping collection) among the search results included in the integrated search screen 1510 is displayed on the screen to the time when the bottommost area of ​​this collection is displayed on the screen. In addition, the floating button 1511 may be displayed at a specific position on the screen after reserving an area for displaying a specific area (e.g., a title, a view more area, an additional area, etc.).

[0091] When a floating button 1511 is touched on the integrated search screen 1510, a multimodal search layer 1520 for additional searches may be slid up and displayed. The multimodal search layer 1520 may include an interface for selecting at least one image from among images included in the search results as an additional query, and an interface for inputting additional text-based search terms. When a specific image is selected from among the search results included in the multimodal search layer 1520 or additional text search terms are input, products with attributes desired by the user may be provided as additional search results in combination with the search terms initially input on the integrated search screen 1510.

[0092] As described above, according to the embodiment of the present invention, by simultaneously understanding multimodal input combining images and text, it is possible to find products that match the user's intention, thereby improving search quality. In particular, according to the embodiment of the present invention, it is possible to quickly and easily find desired search results by transforming the multimodal embedding of products into a virtual space capable of linear expression and using vector operations. Furthermore, according to the embodiment of the present invention, it is possible to accurately reflect the user's intention by providing a search for various combinations in which attributes can be freely added or removed using a virtual space capable of vector operations.

[0093] The above-mentioned devices may be realized by hardware components, software components, and / or a combination of hardware and software components. For example, the devices and components described in the embodiments may be realized using one or more general-purpose or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or various devices capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications that run on the OS. The processing device may also respond to the execution of the software to access, record, manipulate, process, and generate data. For convenience of understanding, a single processing device may be described as being used, but one skilled in the art will understand that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, the processing device may include multiple processors or one processor and one controller. Other processing configurations, such as parallel processors, are also possible.

[0094] The software may include computer programs, codes, instructions, or a combination of one or more of these, and may configure a processing device to operate as desired or instruct the processing device, either independently or collectively. The software and / or data may be embodied in any type of machine, component, physical device, computer storage medium, or device to be interpreted by the processing device or to provide instructions or data to the processing device. The software may be distributed and stored or executed in a distributed manner on computer systems connected by a network. The software and data may be stored on one or more computer-readable storage media.

[0095] The method according to the embodiment may be realized in the form of program instructions executable by various computer means and recorded on a computer-readable medium. In this case, the medium may be one that continuously records a program executable by a computer, or one that temporarily records the program for execution or download. The medium may be one of various recording means or storage means in the form of a single or multiple hardware devices combined, and may be one that is directly connected to a certain computer system, as well as one that is distributed on a network. Examples of the medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and those configured to record program instructions such as ROMs, RAMs, flash memories, etc. Examples of other media include recording media and storage media managed by app stores that distribute applications, sites that supply and distribute various other software, and servers.

[0096] Although the embodiments have been described based on the limited embodiments and drawings as above, those skilled in the art will be able to make various modifications and variations from the above description. For example, the described techniques may be performed in an order different from that described, and / or the components of the described systems, structures, devices, circuits, etc. may be combined or combined in a manner different from that described, or may be counterbalanced or replaced by other components or equivalents, and still achieve suitable results.

[0097] Therefore, different embodiments are within the scope of the appended claims provided that they are equivalent to the claims. [Explanation of symbols]

[0098] 200: Computer equipment 210: Memory 220: Processor 230: Communication interface 240: Input / Output Interface 250: Input / Output Devices 170: Network

Claims

1. 1. A method of multimodal searching implemented on a computer device, comprising: The computing device includes at least one processor configured to execute computer-readable instructions contained in a memory; The multimodal search method includes: performing vector operations between attributes according to a user query on a multimodal embedding space defined as a vector space that can be decomposed into a delta space having nonlinearity and a virtual vector space that can be linearly expressed by the at least one processor; providing, by the at least one processor, search results corresponding to the user query based on the embedding vectors obtained by the vector operation. Including, The performing step includes: estimating a correction function that takes into account a nonlinear error of the multimodal embedding space by using each attribute vector in the multimodal embedding space; and performing a vector operation between attributes included in the user query by an approximation calculation using the correction function; A multimodal search method, including:

2. The performing step includes: Transforming the multimodal embedding space into a linearly representable vector space; and performing vector operations in the vector space between attributes contained in the user query; The multimodal search method of claim 1 , comprising:

3. The providing step comprises: restoring the vector space to the multimodal embedding space, and providing, as the search result, at least one product having a vector within a certain distance from the embedding vector obtained by the vector operation in the multimodal embedding space; The multimodal search method according to claim 2 , characterized in that

4. The performing step includes: performing a vector operation corresponding to at least one image and at least one text input as the user query; The multimodal search method according to claim 1 , characterized in that

5. The performing step includes: performing a vector operation excluding a specific attribute by combining a vector corresponding to the specific attribute among the attributes included in the user query with a vector in an opposite direction of the vector; The multimodal search method according to claim 1 , characterized in that

6. The performing step includes: performing a vector operation to adjust the strength of a particular attribute among the attributes included in the user query by a scalar product of the vector corresponding to the particular attribute; The multimodal search method according to claim 1 , characterized in that

7. The performing step includes: Performing cross-modal feature vector manipulation between a text vector space and an image vector space using the same coordinate system. The multimodal search method of claim 1 , comprising:

8. A computer program for causing a computer to execute the multimodal search method according to any one of claims 1 to 7.

9. 1. A computing device comprising: At least one processor configured to execute computer-readable instructions contained in the memory. Including, The at least one processor A process of performing vector operations between attributes according to a user query on a multimodal embedding space defined as a vector space that can be decomposed into a delta space having nonlinearity and a virtual vector space that can be linearly expressed; and providing a search result corresponding to the user query based on the embedding vector obtained by the vector calculation; Process The at least one processor Using each attribute vector in the multimodal embedding space, a correction function is estimated that takes into account a nonlinear error in the multimodal embedding space; performing a vector operation between attributes included in the user query by an approximation calculation using the correction function; A computer device comprising:

10. The at least one processor Transforming the multimodal embedding space into a vector space that can be linearly represented; performing vector operations in the vector space between attributes included in the user query; 10. The computer device of claim 9.

11. The at least one processor restoring the vector space to the multimodal embedding space, and providing, as the search result, at least one product having a vector within a certain distance from the embedding vector obtained by the vector operation in the multimodal embedding space; The computer device of claim 10 .

12. The at least one processor performing a vector operation corresponding to at least one image and at least one text input as the user query; 10. The computer device of claim 9.

13. The at least one processor performing a vector operation to exclude a specific attribute from among attributes included in the user query by combining a vector corresponding to the specific attribute with a vector in a direction opposite to the vector; 10. The computer device of claim 9.

14. The at least one processor performing a vector operation to adjust the strength of a particular attribute among the attributes included in the user query by a scalar product of the vector corresponding to the particular attribute; 10. The computer device of claim 9.

Images