Method and system for selecting information provider for query of user

Abstract

To provide a method and a system for selecting an information provider for a query of a user.SOLUTION: The method for selecting an information provider according to an embodiment includes the steps of: confirming an LLM result generated based on an LLM (Large Language Models: LLM) to a prompt of a user; and selecting at least one information provider who provides an instance of a content for a prompt of the user from the information providers by reflecting the degree of relevance of the content of each information provider and the LLM result.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The following description relates to a method and system for selecting information providers for a user query. [Background technology]

[0002] A large language model (LLM) is a type of artificial intelligence trained on large amounts of text data to respond in human language to natural language input. It is a language model consisting of an artificial neural network with many parameters (usually billions or more). Such LLMs are trained on large amounts of unlabeled text using self-supervised or semi-self-supervised learning. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Korean Patent No. 10-2551531 Summary of the Invention [Problem to be solved by the invention]

[0004] A method and system for selecting information providers for a user query is provided. [Means for solving the problem]

[0005] Provided is a method for selecting an information provider in a computer device including at least one processor, the method including: checking, by the at least one processor, Large Language Models (LLM) results generated for a user prompt based on LLMs; and selecting, by the at least one processor, at least one information provider from among the plurality of information providers to provide an instance of content for the user prompt, reflecting a degree of relevance between each of the content of the plurality of information providers and the LLM result.

[0006] According to one aspect, the step of selecting at least one information provider may include a step of primarily selecting an information provider associated with at least one of the user prompt, the LLM result, and the recommendation query generated by the large-scale language model from the plurality of information providers, and a step of dynamically conducting an auction among the primarily selected information providers to select the at least one information provider as a final information provider.

[0007] According to another aspect, the step of selecting the at least one information provider as a final information provider may be characterized in that the at least one information provider is selected based on a quality index (QI) and a bid amount (BA) of each information provider, and the quality index is determined using at least a relevance between each of the contents of the plurality of information providers and the LLM result.

[0008] According to another aspect, the step of selecting at least one information provider may be characterized in that the at least one information provider is selected by further reflecting a relevance between each of the plurality of information provider contents and the user prompt.

[0009] According to another aspect, the step of selecting the at least one information provider may be characterized by selecting the at least one information provider further reflecting a relevance of at least one of the user prompts, the LLM results, and the recommendation queries generated by the LLM to an analytical dimension derived from the assets of the multiple information providers.

[0010] According to yet another aspect, the analysis dimensions may include at least one of content type, substance, style, and tone and manner.

[0011] According to another aspect, the assets may include at least one of a Uniform Resource Locator (URL) associated with the content that the information provider wishes to provide, a title of the content, an identification name of the content, a category of the content, multimedia associated with the content, content of the content, and content of an article associated with the content.

[0012] According to another aspect, an instance of the content of the at least one selected information provider may be dynamically generated using the LLM results and already registered assets of the at least one selected information provider and provided to the user.

[0013] According to another aspect, an instance of content of the at least one selected information provider may be dynamically generated by further utilizing at least one of the prompt, a pre-registered prompt of the at least one selected information provider, and information about the user.

[0014] According to yet another aspect, the pre-registered prompts may include at least one of phrases or keywords entered by the at least one selected information provider to be highlighted in association with the content to be provided by the at least one selected information provider, and a tone or format of an informational message provided through the instance for the content, and the information about the user may include at least one of the user's demos, interests, and purchasing information.

[0015] A computer program is provided for causing the computer device to carry out the method.

[0016] A computer-readable recording medium is provided on which a program for causing a computer device to execute the method is recorded.

[0017] A computer device is provided, comprising at least one processor implemented to execute computer-readable instructions, characterized in that the at least one processor checks Large Language Models (LLM) results generated for a user prompt based on LLMs, and selects at least one information provider from among a plurality of information providers to provide an instance of content for the user prompt, reflecting a degree of relevance between each of the content of a plurality of information providers and the LLM results. Effect of the Invention

[0018] A method and system for selecting information providers for a user query may be provided. [Brief description of the drawings]

[0019] [Figure 1] FIG. 1 illustrates an example of a network environment in accordance with an embodiment of the present invention. [Diagram 2]FIG. 2 is a block diagram illustrating an example of a computing device in accordance with an embodiment of the present invention. [Diagram 3] 1 is a diagram illustrating an example of an information provider selection system according to an embodiment of the present invention. [Figure 4] 1 is a flowchart illustrating an example of an information provider selection method according to an embodiment of the present invention. [Diagram 5] FIG. 2 illustrates an example of a dynamically created advertisement instance in accordance with one embodiment of the present invention. [Figure 6] FIG. 2 is a diagram illustrating an example of a process for providing an answer to a user prompt in accordance with an embodiment of the present invention. [Figure 7] FIG. 10 illustrates an example of providing search results in accordance with an embodiment of the present invention. [Figure 8] FIG. 13 is a diagram illustrating an example of selecting a first relevance through LLM in one embodiment of the present invention. [Figure 9] FIG. 13 is a diagram illustrating an example of selecting a second relevance through LLM in one embodiment of the present invention. [Figure 10] FIG. 13 is a diagram illustrating an example of a chat mode in one embodiment of the present invention, in which an answer is provided as an LLM result through a conversation between an LLM-based artificial intelligence and a user. [Figure 11] FIG. 13 is a diagram illustrating an example of a chat mode in one embodiment of the present invention, in which an answer is provided as an LLM result through a conversation between an LLM-based artificial intelligence and a user. [Figure 12] FIG. 13 is a diagram illustrating an example of a chat mode in one embodiment of the present invention, in which an answer is provided as an LLM result through a conversation between an LLM-based artificial intelligence and a user. [Figure 13] FIG. 13 is a diagram illustrating an example of a chat mode in one embodiment of the present invention, in which an answer is provided as an LLM result through a conversation between an LLM-based artificial intelligence and a user. [Figure 14] FIG. 13 is a diagram illustrating an example of a chat mode in one embodiment of the present invention, in which an answer is provided as an LLM result through a conversation between an LLM-based artificial intelligence and a user. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings.

[0021] The information provider selection system according to the embodiment of the present invention may be realized by at least one computer device. In this case, a computer program according to an embodiment of the present invention may be installed and executed in the computer device realizing the information provider selection system, and the computer device may execute the information provider selection method according to the embodiment according to the control of the executed computer program. The above-mentioned computer program may be recorded in a computer-readable recording medium in combination with the computer device to cause the computer to execute the information provider selection method.

[0022] Fig. 1 is a diagram showing an example of a network environment in one embodiment of the present invention. The network environment in Fig. 1 shows an example including a plurality of electronic devices 110, 120, 130, 140, a plurality of servers 150, 160, and a network 170. Fig. 1 is merely an example for explaining the invention, and the number of electronic devices and the number of servers are not limited to those shown in Fig. 1.

[0023] The electronic devices 110, 120, 130, and 140 may be fixed terminals or mobile terminals realized by computer systems. Examples of the electronic devices 110, 120, 130, and 140 include smartphones, mobile phones, navigation systems, personal computers (PCs), notebook PCs, digital broadcasting terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), tablets, game consoles, wearable devices, internet of things (IoT) devices, virtual reality (VR) devices, and augmented reality (AR) devices. As an example, FIG. 1 shows a smartphone as an example of the electronic device 110, but in the embodiment of the present invention, the electronic device 110 may mean one of various physical computer systems that can communicate with other electronic devices 120, 130, and 140 and / or servers 150 and 160 via a network 170 using a substantially wireless or wired communication method.

[0024] The communication method is not limited, and may include not only a communication method utilizing a communication network (for example, a mobile communication network, a wired Internet, a wireless Internet, a broadcast network, a phase network, etc.) that the network 170 can include, but also a short-range wireless communication between devices. For example, the network 170 may include any one or more of a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), a broadband network (BBN), the Internet, etc. Furthermore, the network 170 may include any one or more of a network topology including, but not limited to, a bus network, a star network, a ring network, a mesh network, a star-bus network, a tree or a hierarchical network, etc.

[0025] Each of the servers 150 and 160 may be realized by one or more computer devices that communicate with the multiple electronic devices 110, 120, 130, and 140 via the network 170 to provide instructions, codes, files, contents, services, and the like. For example, the server 150 may be a system that provides a first service to the multiple electronic devices 110, 120, 130, and 140 connected via the network 170, and the server 160 may be a system that provides a second service to the multiple electronic devices 110, 120, 130, and 140 connected via the network 170. As a more specific example, the server 150 may provide a service (such as a search service, for example) targeted by an application, which is a computer program installed and executed in the multiple electronic devices 110, 120, 130, and 140, as a first service to the multiple electronic devices 110, 120, 130, and 140. As another example, the server 160 may provide, as a second service, a service of distributing files for installing and executing the above-mentioned applications to the multiple electronic devices 110, 120, 130, and 140.

[0026] 2 is a block diagram showing an example of a computer device according to an embodiment of the present invention. Each of the electronic devices 110, 120, 130, and 140 and the servers 150 and 160 described above may be realized by a computer device 200 shown in FIG.

[0027] Such a computer device 200 may include a memory 210, a processor 220, a communication interface 230, and an input / output interface 240, as shown in FIG. 2. The memory 210 is a computer-readable recording medium and may include a random access memory (RAM), a read only memory (ROM), and a permanent mass storage device such as a disk drive. Here, a permanent mass storage device such as a ROM or a disk drive may be included in the computer device 200 as a separate permanent storage device separate from the memory 210. The memory 210 may also store an operating system and at least one program code. Such software components may be loaded into the memory 210 from a computer-readable recording medium separate from the memory 210. Such separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD / CD-ROM drive, a memory card, etc. In another embodiment, the software components may be loaded into the memory 210 through a communication interface 230 that is not a computer-readable recording medium. For example, the software components may be loaded into the memory 210 of the computing device 200 based on a computer program installed by a file received over the network 170 .

[0028] The processor 220 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. The instructions may be provided to the processor 220 by the memory 210 or the communication interface 230. For example, the processor 220 may be configured to execute instructions received according to program code stored in a storage device such as the memory 210.

[0029] The communication interface 230 may provide a function for the computer device 200 to communicate with other devices (for example, the above-mentioned recording device) via the network 170. For example, requests, instructions, data, files, etc. generated by the processor 220 of the computer device 200 according to program codes recorded in a recording device such as the memory 210 may be transmitted to other devices via the network 170 under the control of the communication interface 230. Conversely, signals, instructions, data, files, etc. from other devices may be received by the computer device 200 via the communication interface 230 of the computer device 200 via the network 170. The signals, instructions, data, etc. received via the communication interface 230 may be transmitted to the processor 220 or the memory 210, and the files, etc. may be recorded on a recording medium (the above-mentioned permanent recording device) that the computer device 200 may further include.

[0030] The input / output interface 240 may be a means for interfacing with the input / output device 250. For example, the input device may include a device such as a microphone, a keyboard, or a mouse, and the output device may include a device such as a display and a speaker. As another example, the input / output interface 240 may be a means for interfacing with a device in which functions for input and output are integrated into one, such as a touch screen. The input / output device 250 may be configured as one device together with the computer device 200.

[0031] Also, in other embodiments, computing device 200 may include fewer or more components than those of Figure 2. However, most prior art components need not be explicitly illustrated. For example, computing device 200 may be implemented to include at least some of the input / output devices 250 described above, and may further include other components such as a transceiver, a database, etc.

[0032] 3 is a schematic diagram illustrating an example of an information provider selection system according to an embodiment of the present invention. FIG 3 shows an information provider selection system 310, a search system 320, a number of users 330, and a number of information providers 340.

[0033] The search system 320 may correspond to a server (for example, the server 150) that provides a search service to a plurality of users 330, and may be realized by at least one computer device 200. Here, each of the plurality of users 330 may be a physical device of the user that connects to the search system 320 via the network 170 to receive the search service, and such a physical device may be realized by the computer device 200 described above.

[0034] The information provider selection system 310 according to the present embodiment may be included in the search system 320, or may be realized in a form linked with the search system 320 via the network 170. The embodiment of FIG. 3 illustrates an example in which the information provider selection system 310 is realized in a form included in the search system 320. In this case, the information provider selection system 310 may be realized on at least one physical device for realizing the search system 320. Depending on the embodiment, the information provider selection system 310 may be realized in a physical device separate from the physical device for realizing the search system 320, and may be realized in a form communicating with the search system 320 via the network 170.

[0035] The search service provided by the search system 320 to the multiple users 330 may include search results corresponding to the user's input. The search results may be generated based on information searchable on the web. The search system 320 may also provide a search service by including information to be provided by the multiple information providers 340 (instances for the information provider's content) in the search results. Here, the information provided by the multiple information providers 340 may be advertising information, but is not limited to this. The search service itself that provides such search results is well known, and therefore a detailed description will be omitted.

[0036] Meanwhile, the search system 320 according to the present embodiment may provide a search service by including an answer based on artificial intelligence such as Large Language Models (LLM) in the search results. For example, assume that the search system 320 receives a natural language-based prompt from a specific user among the multiple users 330. In this case, the search system 320 may input the received prompt into the LLM, generate a first answer suitable for the prompt as an LLM result, and provide the user with a search result including the first answer. At this time, the search result may include at least a part of various existing search results in addition to the first answer. In addition, the search system 320 may provide a search service based on a conversation between an LLM-based artificial intelligence and a user. A search service may be provided to a user by switching between a first mode in which a first answer is provided as an LLM result based on a general search service and a second mode in which a first answer is provided as an LLM result based on a conversation between an LLM-based artificial intelligence and a user. At this time, a user interface for switching to the second mode may be provided in the first mode, and a user interface for switching to the first mode may be provided in the second mode. Additionally, in each of the first and second modes, at least a portion of the first response may further include providing an instance for the information provider's content as a second response.

[0037] Furthermore, the search results may include a second answer generated by at least a part of the information registered by the multiple information providers 340. For example, assume that the search system 320 dynamically generates an AI-based second answer onto which a message of an information provider selected from the multiple information providers 340 is projected. In this case, the search system 320 may provide the user with search results including not only the first answer generated using the LLM but also the AI-based second answer.

[0038] The information provider selection system 310 may select from among a plurality of information providers 340 an information provider to provide an instance of its content as a second response to the user's prompt.

[0039] First, the information provider selection system 310 may perform a stability verification process to verify whether a natural language-based prompt received from a user is a prompt for which the information provider's content may be provided and / or whether the information provider's content may be provided in association with an answer prompt to the user's prompt (for example, a first answer generated through the LLM to the user's prompt). For example, if the information provider is an advertiser who wishes to publish his / her advertisement, the advertiser may not want his / her advertisement to be published in response to a prompt requesting preset illegal information or preset non-advertising information. In addition, the answer to the user's prompt may include an industry or keyword that the information provider does not want. Therefore, the information provider selection system 310 may proactively verify whether the user's prompt or an answer prompt to the user's prompt is a safe prompt for providing the information provider's content. In this case, the information provider selection system 310 may not provide the information provider's content in response to a legally questionable prompt such as a prompt requesting illegal information. Furthermore, the information provider selection system 310 may exclude an information provider from the information provider selection process for a prompt that does not comply with the policy of the information provider. For example, if the policy of information provider a prohibits providing content of information provider a for a prompt related to a specific business type b or a prompt including a keyword c, information provider a may be excluded from selection targets for a prompt related to the business type b or a prompt including a keyword c.

[0040] In addition, since there may be a plurality of information providers who wish to make their own content public, the information provider selection system 310 may select an information provider to provide content from among the plurality of information providers who have passed the above-mentioned verification process. For example, the information provider may be selected by an auction. As the auction method, one of well-known methods may be used. For example, a GSP (Generalized Second Price) auction method may be used.

[0041] At this time, the ranking of the auction may be determined by a quality index (QI) and a bid amount (BA).

[0042] Here, the quality index may be determined based on at least one of 1) a first relevance between the user's prompt and the information provider's content, 2) a second relevance between the LLM result generated based on the LLM for the user's prompt and the information provider's content, and 3) a third relevance between the information provider's asset. For example, the quality index may be calculated by a weighted sum of the first relevance, the second relevance, and the third relevance. In this case, each of the first relevance, the second relevance, and the third relevance may be obtained through the LLM. For example, the information provider selection system 310 may generate a prompt to ask about the relevance between the user's prompt and the information provider's content and input it to the LLM, and the relevance presented by the LLM may be used as the first relevance. As another example, the information provider selection system 310 may generate a prompt to ask about the relevance between the LLM result and the information provider's content and input it to the LLM, and the relevance presented by the LLM may be used as the second relevance. In this case, the LLM may be trained to calculate the relevance based on the degree of overlap between the contents of the two data, the matching of themes, and / or to calculate the relevance by analyzing the structure, vocabulary, and relationships between sentences using natural language processing technology. Meanwhile, the third relevance may mean the relevance between at least one of the user's prompt, the result of the LLM, and the recommendation query generated by the LLM, and an analysis dimension (e.g., content format, content, style, tone and manner, etc.) derived from the information provider's asset (e.g., an advertiser's advertising material). Such a third relevance may also be measured and calculated by inputting related data into the LLM. For example, the LLM may calculate a relatively high third relevance for a particular type of user prompt, the more each analysis dimension is predicted to perform well (e.g., advertising performance).

[0043] Additionally, the third relevance may be calculated by comprehensively reflecting the LLM advertising index and the analysis content based on the user's demographic / interest information. Here, the LLM advertising index may be obtained by the advertising performance of an existing query and an existing advertisement. For example, when an advertisement for the query "What sneakers do you recommend?" is published, if the advertisement that published "a large-sized image that emphasizes sneakers" has previously performed well, a relatively high relevance score can be obtained when material including this analysis dimension is registered. Also, as an example of utilizing the user's demographic / interest information, when an advertisement is published for a query of a user who is interested in fashion, if the LLM determines that "video content with fast screen changes and people displayed together" has a good performance, a high relevance score can be obtained when material including this analysis dimension is registered.

[0044] Also, a bid amount may be input by each information provider, and the information provider selection system 310 may then select an information provider to provide content in response to a user prompt through an auction based on the quality index and bid amount of each information provider.

[0045] When an information provider is selected, the search system 320 may dynamically generate an instance for the content of the selected information provider and provide it to the user as a second answer. For example, the search system 320 may provide the user with a search result including not only the first answer generated using the LLM but also the second answer described above.

[0046] In this case, the search system 320 may dynamically generate the AI-based second answer using a user's prompt, the first answer generated using the LLM, assets registered by the information provider, and / or a prompt registered by the information provider, instead of providing the information provided by the information provider as is. Here, the assets may include, for example, a uniform resource locator (URL) related to the content to be provided by the information provider, a title or identification name of the content, a content category, multimedia related to the content, the content, and the content of an article related to the content. Here, the multimedia related to the content may include an image or a video related to the content. For example, if the information provider is an advertiser who intends to advertise a specific product or service, the assets may include a URL related to the product or service, a product or service name, a product or service category, product information or service information, and the content of an article related to the product or service. In addition, the prompt registered by the information provider may include information such as a phrase or keyword to be emphasized in relation to the content to be provided by the information provider, and a tone or format of an information message to be provided as the second answer. In this way, the search system 320 may provide the user with search results that include not only a first answer generated using the LLM in response to a user's natural language-based prompt, but also a second answer that is dynamically generated by taking into account the registered assets and prompts of an information provider who wishes to provide their own information, as well as the first answer.

[0047] In addition, according to an embodiment, the search system 320 may further utilize information about the user to generate the second answer, where the information about the user may include the user's demos, interests, purchase information, etc., and may be used to customize the second answer to the user.

[0048] In this manner, when the search system 320 provides an answer to a natural language-based prompt from a user, the information provider selection system 310 selects a specific information provider from among a plurality of information providers 340, and the search system 320 dynamically generates an answer based on the assets and prompts of the selected information provider, i.e., an AI-based second answer in which the information provider's message is projected. Thus, the search system 320 can provide the user with an answer that is dynamically generated so that the information provider's message is projected in association with the natural language-based prompt received from the user.

[0049] In some embodiments, the content of the user's prompt may be insufficient to match the information of a particular information provider. In this case, the search system 320 may provide the user with a question to guide the user to include sufficient information in the user's prompt for the matching. Such a question may also be generated through the LLM, and information obtained as the user's answer to the question may also be included in the user's prompt.

[0050] FIG. 4 is a flowchart showing an example of an information provider selection method according to an embodiment of the present invention. The information provider selection method according to this embodiment may be executed by a computer device 200 that realizes the above-mentioned information provider selection system 310. In this case, the processor 220 of the computer device 200 may be realized to execute control instructions according to the operating system code and at least one computer program code included in the memory 210. Here, the processor 220 may control the computer device 200 so that the computer device 200 performs steps 410 to 430 included in the method of FIG. 4 according to the control instructions provided by the code recorded in the computer device 200.

[0051] In step 410, the computer device 200 may check LLM results generated based on Large Language Models (LLMs) in response to a user's prompt. Here, the user's prompt may include, but is not limited to, a user's query input through a search service provided by the search system 320. As an example, the search system 320 may provide a conversation function between an artificial intelligence and a user, and information input by the user through such a conversation function may be used as the user's prompt. In this case, the LLM results generated by the search system 320 using the LLM in response to the information input by the user may be checked by the computer device 200 in step 410. Such a conversation function may be realized in the form of a sub-service included in the search service, or may be realized in the form of a service separate from the search service and linked to the search service.

[0052] In step 420, the computer device 200 may select at least one information provider from among the plurality of information providers to provide an instance of content for the user's prompt, by reflecting a relevance between each of the plurality of information provider's content and the LLM result. Here, the above-mentioned relevance may correspond to the above-mentioned second relevance through 1) to 3). According to an embodiment, the computer device 200 may select an information provider by further reflecting at least one of a first relevance between the user's prompt and the information provider's content and a third relevance between the information provider's asset and the user's prompt.

[0053] As described above, the computer device 200 may select at least one information provider from among the plurality of information providers through an auction using a quality index determined based on at least one of the first relevance, the second relevance, and the third relevance and the bids of each of the plurality of information providers. In this case, the computer device 200 may include only information providers who have passed the above-mentioned stability confirmation process as targets of the auction.

[0054] In step 430, the computing device 200 may provide information about the at least one selected information provider. For example, the computing device 200 may communicate an identifier of the selected information provider to the search system 320. In this case, the search system 320 may generate a second answer to the user's prompt using the assets and prompts of the selected information provider, the user's prompt and the LLM results, etc., and may provide the user with search results including the first answer and the second answer, including the LLM results.

[0055] FIG. 5 is a diagram showing an example of an advertisement instance dynamically created in an embodiment of the present invention. FIG. 5 shows an example of an advertisement instance 540 generated by considering all of an asset 510 for an advertisement, an LLM result 520 generated through an LLM in response to a user's prompt, and an advertiser's prompt 530. In FIG. 5, "AAA" means an advertiser's product name, and "BBB" means an advertiser's brand name. In addition, in the prompt 530, "ad" means an advertisement, "SEO" means search engine optimization, and "organic" means an LLM result 520. In this way, the information provider selection system 310 can generate an advertisement instance 540 based on the LLM while considering all of the LLM result 520 generated through an LLM in response to a user's prompt, and the asset 510 and the prompt 530 registered by the advertiser as an information provider. As a more specific example, the information provider selection system 310 may extract multiple prompts for the LLM from the LLM results 520, the assets 510, and the prompts 530, respectively, and input the extracted prompts into the LLM to generate the advertisement instance 540.

[0056] In the embodiment of FIG. 5, an example is described in which the assets 510 include only text and a text-based advertisement instance 540 is generated. However, when an asset set including various multimedia such as images and videos is used, the information provider selection system 310 can provide advertisement instances in more various formats including images and videos.

[0057] 6 is a diagram illustrating an example of a process of providing an answer to a user's prompt in an embodiment of the present invention. In the embodiment of FIG. 6, an example is described in which an information provider wishes to publish an advertisement for an advertiser's product or service as information to be provided.

[0058] The search system 320 may input a user prompt 601 from a user terminal connected via the network 170. As an example, the prompt may correspond to a natural language-based search term input by the user. The user may input the search term into a user interface of a search service provided through the user's terminal, and the search system 320 may receive the search term input into the user interface as the user prompt 601.

[0059] At this time, the search system 320 may select the prompt to be actually used by analyzing the user prompt 601 and extracting and summarizing the user intent in the process of extracting and summarizing the user intent (User Intent Extracting & Summarizing 602).

[0060] Meanwhile, the search system 320 may induce the user to provide sufficient information to provide an answer that reflects the marketing message of the advertiser. As an example, the content of the user prompt 601 may be insufficient to match the marketing message of a particular advertiser. In this case, the search system 320 may generate a question to induce additional information for selecting a particular advertiser, and may provide the generated question to the user via the search system 320. Thereafter, when an answer to the question is received from the user, the content of the answer may be used to complete the prompt. The question specification prompt 603 may include a prompt derived from the user's answer.

[0061] At this time, the search system 320 may select a specified user prompt 604 as a prompt for providing the marketing message. That is, the specified user prompt 604 may be specified based on a prompt obtained from the extraction and summary 602 of the user intent for the user prompt 601 and a question specification prompt 603.

[0062] Prompt Ads Safe Check 605 may be an example of a process that verifies whether a specified user prompt 604 is a prompt that is OK for exposing an advertiser's marketing message to a user. As an example, the search system 320 may generate and provide an answer if the specified user prompt 604 is not a prompt requesting pre-defined illegal information or pre-defined non-advertising information.

[0063] The search system 320 may also input specified user prompts 604 into the LLM to generate LLM results. Figure 6 shows an example of an Organic LLM Result Memory 606 that stores such LLM results.

[0064] The search system 320 may initially select an advertiser associated with the LLM result based on the LLM result recorded in the organic LLM result memory 606. In this case, the advertiser associated with the LLM result may be an advertiser who has registered a marketing message that may be published together with the LLM result. The marketing message that may be published together with the LLM result may be selected based on the relevance between the information registered by the advertiser and the LLM result. In addition, according to an embodiment, the search system 320 may use at least one of the user prompt 604, the LLM result, and a recommendation query generated by a large-scale language model when initially selecting an advertiser. In this case, the information provider selection system 310 may initially select an advertiser based on the relevance between at least one of the user prompt 604, the LLM result, and the recommendation query and the information registered by the advertiser. In this case, the information provider selection system 310 may select a specific advertiser from among the advertisers initially selected by the Ad Prompt Auction 607.

[0065] When the advertiser is selected, the search system 320 acquires the advertisement assets 608 registered by the selected advertiser and the advertiser prompts 609 registered by the selected advertiser. In this case, the search system 320 may generate an answer prompt 610 reflecting the marketing message of the advertiser by using at least one of the user prompt 601 and the LLM results recorded in the organic LLM result memory 606, and at least one of the advertisement assets 608 and the advertiser prompt 609. In some embodiments, the advertiser may want to provide an answer in a specific format according to the characteristics of the user. For this reason, the search system 320 may generate the answer prompt 610 by further reflecting information about the user. As an example, the information about the user may include at least one of the demo, interests, and purchase information of the user. For example, the search system 320 may analyze the advertiser prompt 609 and find out that the advertiser wants a relatively more detailed answer to be provided to female users than to male users. In this case, the search system 320 may determine the gender of the user from the user's demo, and then generate the answer prompt 610 taking into consideration the determined gender of the user.

[0066] After the answer prompt 610 is generated, the search system 320 may check whether the generated answer prompt 610 is generated to suit the tone and / or format identified from the advertiser prompt 609. If the generated answer prompt 610 does not match the tone and / or format desired by the advertiser, the answer prompt 610 may be modified to match the tone and / or format desired by the advertiser. In addition, depending on the embodiment, the search system 320 may additionally check whether the generated answer prompt 610 is safe to be made public, etc.

[0067] The search system 320 may then provide the user with the final generated answer 612. As an example, the search system 320 may add the answer 612 to the search results and provide them to the user through a search service. In addition, depending on the embodiment, user information (such as gender, age, and interests) recorded in a DMP (Data Management Platform) 613 may be further utilized in generating the answer prompt 610. By utilizing such user information, the search system 320 can generate an answer 612 optimized for the user.

[0068] Meanwhile, in order to efficiently utilize the DMP 613, the advertiser prompt 609 may further include information on the characteristics and weightings of the target. As an example, the characteristics of the target may include demographics (gender, age (or age group)), interests, and / or purchase information. As an example, the weightings may include the weightings of each characteristic of the target and / or the weightings of each content of the characteristics. As an example, the weightings of each characteristic may indicate how much weighting is given to which characteristic among the gender, age, and interests of the target. For example, if the gender is female, a weighting of 5 may be given, if the age is 20 years old, a weighting of 3 may be given, and if the interest is exercise, a weighting of 8 may be given. In addition, the weightings of each content may indicate how much weighting is given to which content of the same characteristic. As an example, if the interests set by the advertiser are exercise, fashion, and games, the advertiser may give a weighting of 8 to exercise, 6 to fashion, and 2 to games through the advertiser prompt 609. In this case, the search system 320 may further use information about the advertiser's desired target characteristics and weightings included in the advertiser prompt 609 to generate the answer prompt 610. Such information about the target characteristics and weightings may be optionally used if the information about the target characteristics and weightings is available for use in the LLM.

[0069] FIG. 7 is a diagram showing an example of providing search results in an embodiment of the present invention. FIG. 7 shows an example of a screen of a search page 700 provided to a user through a search service. The search page 700 may include a user interface 710 for inputting a user prompt. The search page 700 may also include a search result area 720 for displaying search results. In this case, the search result area 720 may include an LLM result area 730 for displaying LLM results generated based on Large Language Models (LLM) in response to the user prompt.

[0070] The search result area 720 also shows an example of an answer area 740 that displays an answer generated by the search system 320 in response to a user's prompt. In this embodiment, an example is shown in which multiple answers are displayed in the answer area 740. In this manner, multiple answers can be generated and displayed in response to one prompt. It is also possible to generate and display answers for each of two or more information providers. For this reason, the information provider selection system 310 may select two or more information providers.

[0071] On the other hand, the embodiment of FIG. 7 illustrates an example in which an answer of an information provider selected based on a user prompt input to the user interface 710 and / or an LLM result displayed in the LLM result area 730 is displayed in the expansion area 750. For example, the expansion area 750 may further display an answer of an information provider selected based on at least one of a user prompt, an LLM result, and a recommendation query generated by a large-scale language model. If the information provider is an advertiser, an advertisement of the advertiser selected based on at least one of a user prompt, an LLM result, and a recommendation query generated by a large-scale language model may further be displayed in the expansion area 750.

[0072] Also, the dotted frame 760 may display a question for requesting an additional prompt from the user in relation to the answer displayed in the expansion area 750. When the user selects a specific question, the question may be recognized as the user's additional prompt. When providing a conversational search service, the user's additional prompt may be recognized as the user's next conversation. In this case, the search system 320 may generate LLM results and / or answers while taking into account the entire conversation with the user.

[0073] In addition, in the embodiment of FIG. 7, an example in which a user's prompt is input to the user interface 710 of the search service to dynamically generate an answer has been described. However, depending on the embodiment, the search results may also include an interface for generating and providing a dynamic answer. For example, a function for dynamically generating and providing an answer by inputting a user's prompt through each and / or some of various vertical services provided in the existing search ecosystem may be provided to the user. Here, the vertical service may refer to a service for various collections that classify search results, such as shopping search, knowledge search, local search, UGC (User Generated Contents) search, language search, image search, video search, news search, etc. For example, when a shopping search service is separately provided as a vertical service of the integrated search service, a function for dynamically generating and providing an answer by inputting a user's prompt through the shopping search service may be provided to the user. When a plurality of different advertising services are provided as vertical services of the integrated search service, a function for dynamically generating and providing an answer by inputting a user's prompt through each of the plurality of advertising services may be provided to the user.

[0074] FIG. 8 is a diagram showing an example of calculating a first relevance through an LLM in an embodiment of the present invention. FIG. 8 shows an example in which the information provider selection system 310 generates a question 830 and inputs it to the LLM to obtain a first relevance between the advertisement copy 810 and the user's prompt 820 based on the advertisement copy 810 of the advertiser and the user's prompt 820. In this case, the output 840 of the LLM may include the first relevance between the advertisement copy 810 and the user's prompt 820 (60% in the embodiment of FIG. 8). The embodiment of FIG. 8 merely visually displays the output 840 of the LLM to facilitate understanding of the present invention, and the output 840 of the LLM does not need to be actually visually displayed. As an example, the information provider selection system 310 may extract and utilize the first relevance from the output 840 of the LLM.

[0075] FIG. 9 is a diagram showing an example of calculating a second relevance through an LLM in an embodiment of the present invention. FIG. 9 shows an example in which the information provider selection system 310 generates a question 930 and inputs it to the LLM to obtain a second relevance between an advertisement copy 910 of an advertiser and an LLM result 920 generated based on the LLM in response to a user's prompt. In this case, the output 940 of the LLM may include the second relevance between the advertisement copy 910 and the LLM result 920 (40% in the embodiment of FIG. 9). The embodiment of FIG. 9 merely visually displays the output 940 of the LLM to facilitate understanding of the present invention, and the output 940 of the LLM does not need to be actually visually displayed. As an example, the information provider selection system 310 may extract and utilize the second relevance from the output 940 of the LLM.

[0076] 10 to 14 show an example of a chat mode in which an answer is provided as a result of the LLM through a conversation between an LLM-based artificial intelligence and a user in one embodiment of the present invention. Such a chat mode may correspond to the second mode described above.

[0077] In the embodiment of FIG. 10, an input interface 1010 for inputting a user's prompt is shown. The input interface 1010 may be associated with a virtual keyboard function for a user to input text and / or a function for transmitting the text input to the input interface 1010 to the search system 320. In addition, a session initialization interface 1011 for initializing a current conversation session may be provided. The session initialization interface 1011 may be associated with a function for requesting the search system 320 to initialize the current conversation session and start a new conversation session. In the embodiment of FIG. 10, an example is shown in which the session initialization interface 1011 having a specific icon shape is disposed on the left side of the input interface 1010, but the shape, type (icon, button, link, etc.), position, etc. of the session initialization interface 1011 may be variously set according to the embodiment. In addition, a multimedia input interface 1012 for inputting multimedia such as an image or video in addition to text as a user's prompt may be provided. The multimedia input interface 1012 may be associated with a function for selecting and transmitting multimedia data stored in the user's terminal, or transmitting multimedia data generated by a camera provided in the user's terminal.

[0078] Meanwhile, in the embodiment of FIG. 10, a first area 1020 is shown in which a user's prompt transmitted to the search system 320 through the input interface 1010 is displayed in the form of a message for conversation. Also, a second area 1030 is shown in association with the first area 1020 for displaying a process of generating an answer to the user's prompt. The process of generating an answer may include, for example, a "search" process, a "search result analysis" process, a "consider whether further search is necessary" process, and an "answer generation completion" process, but is not limited thereto. In the embodiment of FIG. 10, the second area 1030 shows the "answer generation completion" process. Also, the embodiment of FIG. 10 shows a third area 1030 in which an answer generated in response to the user's prompt is displayed. At this time, an icon 1031 for indicating an entity that provided the answer may be displayed in association with the third area 1030. For example, the icon 1031 may include information that can be used to identify the search system 320.

[0079] The search system 320 may further provide the user with a first recommendation prompt 1040. In this case, the user only needs to select the first recommendation prompt 1040, and the first recommendation prompt 1040 is used as the user's prompt, so that the user can continue the next conversation with the artificial intelligence in the current conversation session. The search system 320 may further provide the user with a second recommendation prompt 1050 for a conversation with a specific information provider. In this case, an icon 1051 for indicating the information provider associated with the second recommendation prompt 1050 may be displayed in association with the second recommendation prompt 1050. For example, the icon 1051 may include information such as an image or text associated with the information provider.

[0080] The embodiment of FIG. 11 illustrates an example in which a conversation with a user is conducted through an artificial intelligence specialized for a specific information provider by the user selecting the second recommendation prompt 1050 in FIG. 10. In this case, the artificial intelligence specialized for a specific information provider may also be an artificial intelligence provided by the search system 320 based on the LLM. In some embodiments, the artificial intelligence specialized for a specific information provider may be registered in the search system 320 by a specific information provider, or may be provided by a specific information provider from the beginning. In this case, the answers 1110 and 1120 provided by the artificial intelligence specialized for a specific information provider may display information 1130 for notifying that the answers 1110 and 1120 are provided by a specific information provider, and an icon 1051 for the information provider may be further displayed. Such answers 1110 and 1120 may be dynamically generated by reflecting information (e.g., assets, prompts, etc.) registered in relation to the information provider, as described above. Each of the advertisement cards (advertisement card 1 (1121), advertisement card 2 (1122), and advertisement card 3 (1123)) included in the answer 1120 may be produced in a form including a product image and a product description (product identifier, price, etc.).

[0081] The embodiment of Fig. 12 shows an example in which a dynamically generated answer for a specific information provider's app advertisement is provided in response to a user's prompt for an app recommendation, in which a specific brand content answer is dynamically generated and provided as a search generative experience (SGE) style.

[0082] The embodiment of Figure 13 illustrates an example where information provider responses are dynamically generated and provided in other SGE style formats in response to the same user prompts as in Figure 12. In this manner, information provider responses may be dynamically generated and provided in a variety of formats and content in response to prompts.

[0083] The embodiment of Fig. 14 shows an example in which, during a conversation with an artificial intelligence, the information provider's answers are provided as if they were other users in a group chat room. That is, by providing the information provider's answers 1420 in a conversational format in addition to the artificial intelligence's answers 1410, the user can have an experience as if he or she were having a conversation with two or more other users in a group chat room.

[0084] Thus, according to an embodiment of the present invention, a method and system for selecting information providers for a user's query can be provided.

[0085] The above-described systems or devices may be realized by hardware components or a combination of hardware and software components. For example, the devices and components described in the embodiments may be realized using one or more general-purpose or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or various devices capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications that run on the OS. The processing device may also respond to the execution of the software to access, record, manipulate, process, and generate data. For convenience of understanding, a single processing device may be described as being used, but those skilled in the art will understand that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, the processing device may include multiple processors or one processor and one controller. Other processing configurations, such as parallel processors, are also possible.

[0086] The software may include computer programs, codes, instructions, or a combination of one or more of these, and may configure a processing device to operate as desired or instruct the processing device, either independently or collectively. The software and / or data may be embodied in any type of machine, component, physical device, virtual device, computer storage medium, or device to be interpreted based on the processing device or to provide instructions or data to the processing device. The software may be distributed and stored or executed in a distributed manner on computer systems connected by a network. The software and data may be stored on one or more computer-readable storage media.

[0087] The method according to the embodiment may be realized in the form of program instructions executable by various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The medium may continuously record a program executable by a computer, or may temporarily record the program for execution or download. The medium may be various recording or storage means in the form of a single or multiple hardware devices combined, and may be a medium directly connected to a certain computer system, or may be distributed on a network. Examples of the medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and media configured to record program instructions such as ROMs, RAMs, flash memories, and the like. Other examples of the medium include recording media and storage media managed by app stores that distribute applications, sites that supply or distribute various other software, and servers. Examples of the program instructions include not only machine language codes such as those generated by compilers, but also high-level language codes executed by a computer using an interpreter, etc.

[0088] Although the embodiments have been described based on the limited embodiments and drawings as above, those skilled in the art will be able to make various modifications and variations from the above description. For example, the described techniques may be performed in an order different from that described, and / or the components of the described systems, structures, devices, circuits, etc. may be combined or combined in a manner different from that described, or may be counterbalanced or replaced by other components or equivalents, and still achieve suitable results.

[0089] Therefore, different embodiments are within the scope of the appended claims provided that they are equivalent to the claims. [Explanation of symbols]

[0090] 310: Information provider selection system 320: Search system 330: User 340:Informant

Claims

1. 1. A method for selecting an information provider in a computer device including at least one processor, comprising: identifying, by the at least one processor, Large Language Models (LLM) results generated based on the LLM in response to user prompts; and selecting, by the at least one processor, from among the plurality of information providers, at least one information provider to provide an instance of content for the user prompt, the at least one information provider reflecting a degree of relevance of each of the plurality of information providers' content to the LLM results. and methods for selecting informants, including:

2. The step of selecting at least one information provider includes: selecting, from among the plurality of information providers, an information provider associated with at least one of the user prompt, the LLM result, and a recommendation query generated by the large-scale language model; and dynamically conducting an auction among the preselected information providers and selecting at least one of the information providers as a final information provider.

2. The information provider selection method according to claim 1, further comprising:

3. The step of selecting a final information provider includes: selecting the at least one information provider according to a ranking determined based on a quality index (QI) and a bid amount (BA) of each of the information providers; The quality index is determined using at least a degree of relevance between each of the content of the plurality of information providers and the LLM result.

3. The information provider selection method according to claim 2, further comprising:

4. The step of selecting at least one information provider includes: selecting the at least one information provider based on a further reflection of a relevance of each of the contents of the plurality of information providers to the user's prompt; 2. The information provider selection method according to claim 1,

5. The step of selecting at least one information provider includes: selecting the at least one information provider further reflecting a degree of relevance of at least one of the user prompts, the LLM results, and the recommendation queries generated by the LLM to an analytical dimension derived from the assets of the plurality of information providers; 2. The information provider selection method according to claim 1,

6. The analysis dimensions include at least one of the following: form, substance, style, and tone and manner of the content.

6. The information provider selection method according to claim 5,

7. The assets include at least one of a URL (Uniform Resource Locator) related to the content that the information provider intends to provide, a title of the content, an identification name of the content, a category of the content, multimedia related to the content, content details, and content details of an article related to the content.

6. The information provider selection method according to claim 5,

8. An instance of the content of the at least one selected information provider is dynamically generated using the LLM results and pre-registered assets of the at least one selected information provider, and provided to the user.

8. The information provider selection method according to claim 1, further comprising:

9. and dynamically generating an instance of the content of the at least one selected information provider by further utilizing at least one of the user's prompts, the pre-registered prompts of the at least one selected information provider, and information about the user.

9. The information provider selection method according to claim 8, further comprising:

10. the pre-registered prompts include at least one of a phrase or keyword entered by the at least one selected information provider to be highlighted in association with the content to be provided by the at least one selected information provider, and a tone or format of an informational message to be provided through the instance of the content; The information about the user includes at least one of demos, interests, and purchasing information of the user.

10. The information provider selection method according to claim 9,

11. A computer program for causing a computer device to execute the information provider selection method according to any one of claims 1 to 7.

12. At least one processor implemented to execute computer readable instructions. Including, by said at least one processor, Reviewing Large Language Models (LLM) results generated based on LLMs in response to user prompts; selecting at least one information provider from among the plurality of information providers to provide an instance of content for the user prompt, the at least one information provider being selected based on a relevance of each of the plurality of information providers' content to the LLM result; A computer device comprising:

13. selecting the at least one information provider, by the at least one processor; Primary selecting an information provider from the plurality of information providers that is associated with at least one of the user prompt, the LLM result, and a recommendation query generated by the large-scale language model; dynamically conducting an auction among the initially selected information providers to select the at least one information provider as a final information provider; 13. The computer device of claim 12.

14. by said at least one processor for selecting said final information provider; selecting the at least one information provider according to a ranking determined based on a quality index (QI) and a bid amount (BA) of each of the information providers; The quality index is determined using at least a degree of relevance between each of the content of the plurality of information providers and the LLM result.

14. The computer device of claim 13.

15. selecting the at least one information provider, by the at least one processor; selecting the at least one information provider based on a further reflection of a relevance of each of the contents of the plurality of information providers to the user's prompt; 13. The computer device of claim 12.

16. selecting the at least one information provider, by the at least one processor; selecting the at least one information provider further reflecting a degree of relevance of at least one of the user prompts, the LLM results, and the recommendation queries generated by the LLM to an analytical dimension derived from the assets of the plurality of information providers; 13. The computer device of claim 12.

Images