Method and system for selecting information providers for user queries

Abstract

To provide a method and a system for selecting an information provider for a query of a user.SOLUTION: The method for selecting an information provider according to an embodiment includes the steps of: confirming an LLM result generated based on an LLM (Large Language Models: LLM) to a prompt of a user; and selecting at least one information provider who provides an instance of a content for a prompt of the user from the information providers by reflecting the degree of relevance of the content of each information provider and the LLM result.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The following description relates to a method and system for selecting an information provider for a user query.

Background Art

[0002] A large language model (LLM) is a type of artificial intelligence trained on large-scale text data so that it can respond in human language to natural language inputs. It is a language model composed of an artificial neural network with many parameters (usually billions or more). Such an LLM is trained on large amounts of text without labels using self-supervised learning or semi-self-supervised learning.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] A method and system for selecting an information provider for a user query are provided.

Means for Solving the Problems

[0005] A method for selecting an information provider for a computer device including at least one processor, comprising the steps of: confirming LLM results generated by the at least one processor based on Large Language Models (LLM) in response to a user prompt; and selecting at least one information provider from among the multiple information providers to provide an instance of content for the user prompt, reflecting the degree of relevance between each of the multiple information providers' contents and the LLM results, by the at least one processor.

[0006] In one aspect, the step of selecting the at least one information provider may be characterized by including a step of first selecting an information provider from the plurality of information providers that is associated with at least one of the user prompts, the LLM results, and the recommendation queries generated by the large-scale language model, and a step of dynamically conducting an auction among the first selected information providers to select the at least one information provider as the final information provider.

[0007] In other respects, the stage of selecting the final information provider may be characterized by selecting at least one information provider based on a ranking determined on each information provider's Quality Index (QI) and Bid Amount (BA), wherein the Quality Index is determined using at least the degree of relevance between each of the multiple information providers' contents and the LLM results.

[0008] From another perspective, the step of selecting the at least one information provider may be characterized by selecting the at least one information provider by further reflecting the degree of relevance between each of the multiple information provider contents and the user prompt.

[0009] In other respects, the step of selecting the at least one information provider may be characterized by selecting the at least one information provider by further reflecting the relationship between the user prompt, the LLM results, and at least one of the recommendation queries generated by the LLM and the analytical dimensions derived from the assets of the multiple information providers.

[0010] From another perspective, the aforementioned analytical dimension may be characterized by including at least one of the following: content format, content, style, and tone and manner.

[0011] In other respects, the asset may be characterized by including at least one of the following: a URL (Uniform Resource Locator) associated with the content that the information provider intends to provide, the title of the content, the identifier of the content, the category of the content, the multimedia associated with the content, the content of the content, and the content of the article associated with the content.

[0012] Another aspect may be that instances of content from the selected at least one information provider are dynamically generated using the LLM results and the already registered assets of the selected at least one information provider and provided to the user.

[0013] Another aspect may be that an instance of the content of the selected at least one information provider is dynamically generated by further utilizing at least one of the following: the prompt, the pre-registered prompt of the selected at least one information provider, and the information of the user.

[0014] In another aspect, the pre-registered prompt may be characterized by including at least one of the following: a phrase or keyword entered by the selected at least one information provider, and the tone and format of the informational message provided through the instance for the content, in order to be emphasized in relation to the content that the selected at least one information provider intends to provide; and the information about the user may be characterized by including at least one of the user's demos, interests, and purchase information.

[0015] The present invention provides a computer program for causing the computer device to execute the aforementioned method.

[0016] The present invention provides a computer-readable recording medium on which a program for causing a computer device to execute the aforementioned method is recorded.

[0017] The present invention provides a computer device comprising at least one processor implemented to execute instructions readable by the computer device, wherein the at least one processor checks LLM results generated based on Large Language Models (LLM) in response to a user prompt, and selects at least one information provider from among the multiple information providers to provide an instance of content for the user prompt, reflecting the degree of relevance between each of the information provider's contents and the LLM results. [Effects of the Invention]

[0018] We can provide a method and system for selecting information providers for user queries. [Brief explanation of the drawing]

[0019] [Figure 1] This diagram shows an example of a network environment in one embodiment of the present invention. [Figure 2]A block diagram showing an example of a computer device in one embodiment of the present invention. [Figure 3] An exemplary diagram generally showing an information provider selection system in one embodiment of the present invention. [Figure 4] A flowchart showing an example of an information provider selection method in one embodiment of the present invention. [Figure 5] A diagram showing an example of a dynamically created advertisement instance in one embodiment of the present invention. [Figure 6] A diagram showing an example for explaining the process of providing an answer to a user's prompt in one embodiment of the present invention. [Figure 7] A diagram showing an example of providing search results in one embodiment of the present invention. [Figure 8] A diagram showing an example of selecting a first relevance level through an LLM in one embodiment of the present invention. [Figure 9] A diagram showing an example of selecting a second relevance level through an LLM in one embodiment of the present invention. [Figure 10] A diagram showing an example of a chat mode for providing an answer as an LLM result through a conversation between an LLM-based artificial intelligence and a user in one embodiment of the present invention. [Figure 11] A diagram showing an example of a chat mode for providing an answer as an LLM result through a conversation between an LLM-based artificial intelligence and a user in one embodiment of the present invention. [Figure 12] A diagram showing an example of a chat mode for providing an answer as an LLM result through a conversation between an LLM-based artificial intelligence and a user in one embodiment of the present invention. [Figure 13] A diagram showing an example of a chat mode for providing an answer as an LLM result through a conversation between an LLM-based artificial intelligence and a user in one embodiment of the present invention. [Figure 14] A diagram showing an example of a chat mode for providing an answer as an LLM result through a conversation between an LLM-based artificial intelligence and a user in one embodiment of the present invention. [Modes for carrying out the invention]

[0020] The embodiments will be described in detail below with reference to the attached drawings.

[0021] An information provider selection system according to an embodiment of the present invention may be implemented by at least one computer device. In this case, a computer program according to one embodiment of the present invention may be installed and executed on the computer device that implements the information provider selection system, and the computer device may execute an information provider selection method according to the embodiment in accordance with the control of the executed computer program. The above-described computer program may be recorded on a computer-readable recording medium in conjunction with the computer device to allow the computer to execute the information provider selection method.

[0022] Figure 1 is a diagram showing an example of a network environment in one embodiment of the present invention. The network environment in Figure 1 shows an example that includes multiple electronic devices 110, 120, 130, 140, multiple servers 150, 160, and a network 170. Figure 1 is merely an example for the purpose of explaining the invention, and the number of electronic devices and servers is not limited to what is shown in Figure 1.

[0023] The multiple electronic devices 110, 120, 130, and 140 may be fixed or mobile terminals implemented by a computer system. Examples of the multiple electronic devices 110, 120, 130, and 140 include smartphones, mobile phones, navigation systems, personal computers (PCs), notebook PCs, digital broadcasting terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), tablets, game consoles, wearable devices, internet of things (IoT) devices, virtual reality (VR) devices, and augmented reality (AR) devices. As an example, Figure 1 shows a smartphone as an example of electronic device 110, but in embodiments of the present invention, electronic device 110 may mean one of a variety of physical computer systems that can communicate with other electronic devices 120, 130, 140 and / or servers 150, 160 via a network 170 using substantially wireless or wired communication methods.

[0024] The communication method is not limited, and it may include not only communication methods that utilize communication networks that can be included in network 170 (for example, mobile communication networks, wired internet, wireless internet, broadcasting networks, phase networks, etc.), but also short-range wireless communication between devices. For example, network 170 may include one or more arbitrary networks such as PAN (personal area network), LAN (local area network), CAN (campus area network), MAN (metropolitan area network), WAN (wide area network), BBN (broadband network), and the Internet. Furthermore, network 170 may include, but is not limited to, one or more network topologies, including bus networks, star networks, ring networks, mesh networks, star-bus networks, tree or hierarchical networks.

[0025] Servers 150 and 160 may each be implemented by one or more computer devices that communicate with multiple electronic devices 110, 120, 130, and 140 via a network 170 to provide commands, code, files, content, services, etc. For example, server 150 may be a system that provides a first service to multiple electronic devices 110, 120, 130, and 140 connected via the network 170, and server 160 may be a system that provides a second service to multiple electronic devices 110, 120, 130, and 140 connected via the network 170. As a more specific example, server 150 may provide the multiple electronic devices 110, 120, 130, and 140 as a first service through an application, which is a computer program installed and executed on the multiple electronic devices 110, 120, 130, and 140, with the service targeted by that application (for example, a search service). As another example, server 160 may provide a second service that distributes files for installing and running the aforementioned application to multiple electronic devices 110, 120, 130, and 140.

[0026] Figure 2 is a block diagram showing an example of a computer device in one embodiment of the present invention. Each of the aforementioned electronic devices 110, 120, 130, and 140, as well as each of the servers 150 and 160, may be implemented by the computer device 200 shown in Figure 2.

[0027] Such a computer device 200 may include a memory 210, a processor 220, a communication interface 230, and an input / output interface 240, as shown in Figure 2. The memory 210 is a computer-readable recording medium and may include RAM (random access memory), ROM (read-only memory), and persistent mass storage devices such as disk drives. Here, persistent mass storage devices such as ROM and disk drives may be included in the computer device 200 as separate persistent storage devices distinct from the memory 210. The memory 210 may also store an operating system and at least one program code. Such software components may be loaded into the memory 210 from a computer-readable recording medium separate from the memory 210. Such a separate computer-readable recording medium may include computer-readable recording media such as floppy disks, disks, tapes, DVD / CD-ROM drives, and memory cards. In other embodiments, the software components may be loaded into the memory 210 through a communication interface 230 which is not a computer-readable recording medium. For example, software components may be loaded into the memory 210 of the computer device 200 based on a computer program installed by a file received via the network 170.

[0028] The processor 220 may be configured to process computer program instructions by performing basic arithmetic, logic, and input / output operations. Instructions may be provided to the processor 220 by memory 210 or a communication interface 230. For example, the processor 220 may be configured to execute instructions received according to program code stored in a recording device such as memory 210.

[0029] The communication interface 230 may provide a function for the computer device 200 to communicate with other devices (for example, the recording device described above) via the network 170. For example, requests, instructions, data, files, etc., generated by the processor 220 of the computer device 200 according to program code recorded in a recording device such as memory 210 may be transmitted to other devices via the network 170 under the control of the communication interface 230. Conversely, signals, instructions, data, files, etc., from other devices may be received by the computer device 200 via the network 170 through the communication interface 230 of the computer device 200. Signals, instructions, data, etc., received via the communication interface 230 may be transmitted to the processor 220 or memory 210, and files, etc., may be recorded on a recording medium (the persistent recording device described above) that the computer device 200 may further include.

[0030] The input / output interface 240 may be a means for interface with the input / output device 250. For example, the input device may include a microphone, keyboard, or mouse, and the output device may include a display or speaker. In another example, the input / output interface 240 may be a means for interface with a device that integrates input and output functions into one, such as a touchscreen. The input / output device 250 may consist of a single device together with the computer device 200.

[0031] In other embodiments, the computer device 200 may include fewer or more components than those shown in Figure 2. However, it is not necessary to explicitly show most of the conventional components in the figure. For example, the computer device 200 may be implemented to include at least some of the input / output devices 250 described above, and may further include other components such as transceivers and databases.

[0032] Figure 3 is a schematic diagram illustrating an information provider selection system in one embodiment of the present invention. Figure 3 shows an information provider selection system 310, a search system 320, multiple users 330, and multiple information providers 340.

[0033] The search system 320 may correspond to a server (for example, server 150) that provides search services to multiple users 330, and may be implemented by at least one computer device 200. Here, each of the multiple users 330 may be a physical device of a user that connects to the search system 320 via the network 170 to receive search services, and such a physical device may be implemented by the computer device 200 described above.

[0034] The information provider selection system 310 according to this embodiment may be included in the search system 320, or it may be implemented in a manner that is linked with the search system 320 via the network 170. The embodiment in Figure 3 shows an example in which the information provider selection system 310 is implemented in a manner that is included in the search system 320. In this case, the information provider selection system 310 may be implemented on at least one physical device that implements the search system 320. Depending on the embodiment, the information provider selection system 310 may be implemented on a physical device separate from the physical device that implements the search system 320, and may be implemented in a manner that communicates with the search system 320 via the network 170.

[0035] The search service provided by the search system 320 to multiple users 330 may include search results corresponding to user input. The search results may, in principle, be generated based on information searchable on the web. Furthermore, the search system 320 may include information intended to be provided by multiple information providers 340 (instances for the information providers' content) in the search results. Here, the information provided by the multiple information providers 340 may, but is not limited to, advertising information. Since the search service itself that provides such search results is well-known, a detailed explanation will be omitted.

[0036] On the other hand, the search system 320 according to this embodiment may provide a search service that includes answers based on artificial intelligence, such as Large Language Models (LLMs), in the search results. For example, suppose the search system 320 receives a natural language-based prompt from a specific user among a group of users 330. In this case, the search system 320 may input the received prompt into the LLM, generate a first answer suitable for the prompt as an LLM result, and provide the user with search results including the first answer. In this case, the search results may include at least a portion of existing diverse search results in addition to the first answer. Furthermore, the search system 320 may provide a search service based on a conversation between the LLM-based artificial intelligence and the user. The search service may be provided to the user by switching between a first mode that provides a first answer as an LLM result based on a general search service and a second mode that provides a first answer as an LLM result based on a conversation between the LLM-based artificial intelligence and the user. In this case, a user interface for switching to the second mode may be provided in the first mode, and a user interface for switching to the first mode may be provided in the second mode. Furthermore, in both the first and second modes, at least a portion of the first responses may be further provided as second responses, containing instances of the information provider's content.

[0037] Furthermore, the search results may include a second response generated from at least some of the information registered by multiple information providers 340. For example, suppose the search system 320 dynamically generates a second response based on artificial intelligence, onto which the message of an information provider selected from among the multiple information providers 340 is projected. In this case, the search system 320 may provide the user with search results that include not only the first response generated using LLM, but also the second response based on artificial intelligence.

[0038] The information provider selection system 310 may select an information provider from among multiple information providers 340 to provide an instance of its own content as a second response to the user's prompt.

[0039] First, the information provider selection system 310 may process a stability check process to verify whether a natural language-based prompt received from a user is a prompt that may provide the information provider's content, and / or whether it may provide the information provider's content in conjunction with a response prompt to the user's prompt (for example, a first response generated through LLM to the user's prompt). For example, if the information provider is an advertiser who intends to publish their advertisement, the advertiser may not want their advertisement to be published in response to prompts that request pre-configured illegal information or pre-configured non-advertising information. Also, the response to the user's prompt may include industries or keywords that the information provider does not want. Therefore, the information provider selection system 310 may preemptively verify whether the user's prompt and the response prompt to the user's prompt are prompts that are safe for providing the information provider's content. In this case, the information provider selection system 310 does not have to provide the information provider's content to prompts that are legally problematic, such as prompts that request illegal information. Furthermore, the information provider selection system 310 may exclude information providers from the selection process if the prompt does not conform to the information provider's policy. For example, if information provider a's policy prohibits providing information provider a's content to prompts related to a specific industry b or prompts containing keyword c, then information provider a may be excluded from the selection process for prompts related to industry b or prompts containing keyword c.

[0040] Furthermore, since there may be multiple information providers who wish to publish their content, the information provider selection system 310 may select information providers who will provide content from among the multiple information providers who have passed the verification process described above. For example, information providers may be selected by auction. One of the well-known auction methods may be used. For example, the GSP (Generalized Second Price) auction method may be used.

[0041] In this case, the auction ranking may be determined by the Quality Index (QI) and the Bid Amount (BA).

[0042] Here, the quality index may be determined based on at least one of the following relevances: 1) the first relevance between the user prompt and the information provider's content, 2) the second relevance between the LLM result generated based on the LLM in response to the user prompt and the information provider's content, and 3) the third relevance with the information provider's assets. For example, the quality index may be calculated as a weighted sum of the first, second, and third relevances. In this case, the first, second, and third relevances may each be obtained through the LLM. For example, the information provider selection system 310 may generate a prompt to ask about the relevance between the user prompt and the information provider's content and input it into the LLM, and the relevance presented by the LLM may be used as the first relevance. As another example, the information provider selection system 310 may generate a prompt to ask about the relevance between the LLM result and the information provider's content and input it into the LLM, and the relevance presented by the LLM may be used as the second relevance. In this case, LLM may be trained to calculate relevance based on the degree of overlap in content between two sets of data, the degree of subject agreement, etc., and / or to calculate relevance by analyzing sentence structure, vocabulary, relationships between sentences, etc. using natural language processing techniques. On the other hand, the third relevance may represent the degree of relevance between at least one of the user prompt, LLM results, and recommendation queries generated by LLM, and analytical dimensions derived from the information provider's assets (for example, the advertiser's advertising materials) (for example, content format, content, style, tone and manner, etc.). Such a third relevance may also be measured and calculated by inputting relevant data into LLM. For example, LLM may calculate a relatively high third relevance for a particular type of user prompt, as each analytical dimension is predicted to have good results (for example, advertising results).

[0043] Additionally, the third relevance score may be calculated by comprehensively reflecting LLM advertising metrics and analysis based on user demo / interest information. Here, LLM advertising metrics may be obtained from the advertising performance of existing queries and existing ads. For example, when an ad is published for the query "Tell me your recommended sneakers," if ads that previously published "large-sized images emphasizing sneakers" performed well, then a relatively high relevance score can be obtained when material including this analysis dimension is registered. Also, as an example of utilizing user demo / interest information, when an ad is published for a query from a user interested in fashion, if LLM determines that "video content with fast screen transitions and people displayed together" performs well, then a high relevance score can be obtained when material including this analysis dimension is registered.

[0044] Furthermore, the bid amount may be entered by each information provider. In this case, the information provider selection system 310 may select information providers to provide content in response to user prompts through an auction based on each information provider's quality index and bid amount.

[0045] Once an information provider is selected, the search system 320 may dynamically generate an instance of the selected information provider's content and provide it to the user as a second response. For example, the search system 320 may provide the user with search results that include not only the first response generated using LLM, but also the second response described above.

[0046] In this case, when the search system 320 generates a second response based on artificial intelligence, it may not simply provide the information provided by the information provider as is, but may dynamically generate the second response based on artificial intelligence by using the user's prompt, the first response generated using LLM, the assets registered by the information provider, and / or the prompts registered by the information provider. Here, assets may include, for example, a URL (Uniform Resource Locator) associated with the content that the information provider intends to provide, the title or identifier of the content, the category of the content, multimedia associated with the content, the content of the content, and the content of articles associated with the content. Here, multimedia associated with the content may include images and videos associated with the content. For example, if the information provider is an advertiser intending to advertise a specific product or service, the assets may include a URL associated with the product or service, the product or service name, the category of the product or service, product or service information, and the content of articles associated with the product or service. In addition, the prompts registered by the information provider may include phrases or keywords that the information provider intends to emphasize in relation to the content that the information provider intends to provide, and information such as the tone and format of the information message provided as the second response. Thus, the search system 320 may provide the user with search results that include not only a first response generated using LLM in response to the user's natural language-based prompt, but also a second response dynamically generated by considering the registered assets and prompts of information providers who intend to provide their own information, as well as the first response.

[0047] Furthermore, according to the embodiment, the search system 320 may generate a second response by further utilizing information about the user. Here, the information about the user may include the user's demos, interests, purchase information, etc., and may be used to customize the second response for the user.

[0048] Thus, in providing a response to a natural language-based prompt from a user, the search system 320 selects a specific information provider from among multiple information providers 340, and the search system 320 can dynamically generate a response based on the selected information provider's assets and prompt, i.e., a second AI-based response onto which the information provider's message is projected. Therefore, the search system 320 can provide the user with a dynamically generated response such that the information provider's message is projected in relation to the natural language-based prompt received from the user.

[0049] Furthermore, depending on the embodiment, the content of the user prompt may be insufficient to match with information from a specific information provider. In this case, the search system 320 may provide the user with questions to guide the user so that the user prompt contains sufficient information for the matching. Such questions may also be generated through the LLM, and the information obtained as the user's answer to the questions may also be included in the user prompt.

[0050] Figure 4 is a flowchart showing an example of an information provider selection method in one embodiment of the present invention. The information provider selection method according to this embodiment may be executed by a computer device 200 that implements the information provider selection system 310 described above. In this case, the processor 220 of the computer device 200 may be implemented to execute control instructions from the operating system code contained in the memory 210 and the code of at least one computer program. Here, the processor 220 may control the computer device 200 so that it performs steps 410 to 430 included in the method of Figure 4 in accordance with the control instructions provided by the code recorded in the computer device 200.

[0051] In step 410, the computer device 200 may verify the LLM results generated based on the Large Language Model (LLM) in response to the user prompt. Here, the user prompt may include, but is not limited to, the user's query entered through the search service provided by the search system 320. For example, the search system 320 may provide a conversational function between artificial intelligence and the user, and information entered by the user through such a conversational function may be used as the user prompt. In this case, the LLM results generated by the search system 320 using the LLM as a response to the information entered by the user may be verified by the computer device 200 in step 410. Such a conversational function may be implemented as a sub-service included in the search service, or as a separate service that works in conjunction with the search service.

[0052] In step 420, the computer device 200 may select at least one information provider from among the multiple information providers to provide an instance of content for the user prompt, reflecting the degree of relevance between each of the information providers' contents and the LLM results. Here, the degree of relevance described above may correspond to the second degree of relevance described through 1) to 3). Depending on the embodiment, the computer device 200 may further select the information provider by reflecting at least one of the first degree of relevance between the user prompt and the information provider's content and the third degree of relevance with the information provider's assets.

[0053] As described above, the computer device 200 may select at least one information provider from among multiple information providers by an auction using a quality index determined based on at least one of the first, second, and third relevances and the bid amounts of each of the multiple information providers. In this case, the computer device 200 may include only information providers who have passed the stability verification process described above as subjects of the auction.

[0054] In step 430, the computer device 200 may provide information about at least one selected information provider. For example, the computer device 200 may transmit the identifier of the selected information provider to the search system 320. In this case, the search system 320 may use the assets and prompts of the selected information provider, the user's prompt and LLM results, etc., to generate a second response to the user's prompt, and may provide the user with search results including a first response including the LLM results and a second response.

[0055] Figure 5 shows an example of a dynamically generated ad instance in one embodiment of the present invention. Figure 5 shows an example of an ad instance 540 generated by taking into account the ad asset 510, the LLM result 520 generated through LLM in response to the user prompt, and the advertiser's prompt 530. In Figure 5, "AAA" means the advertiser's product name and "BBB" means the advertiser's brand name. In the prompt 530, "ad" means ad, "SEO" means Search Engine Optimization, and "organic" means LLM result 520. In this way, the information provider selection system 310 can generate an LLM-based ad instance 540 by taking into account the LLM result 520 generated through LLM in response to the user prompt, and the assets 510 and prompt 530 registered by the advertiser as an information provider. As a more specific example, the information provider selection system 310 may extract multiple prompts for LLM from the LLM result 520, the asset 510, and the prompt 530, respectively, and input the extracted prompts into LLM to generate an advertising instance 540.

[0056] In the embodiment shown in Figure 5, an example was described in which asset 510 contains only text and a text-based advertising instance 540 is generated. However, when using an asset set that includes various multimedia such as images and videos, the information provider selection system 310 can provide advertising instances in a wider variety of formats that include images and videos.

[0057] Figure 6 is a diagram illustrating an example of the process of providing a response to a user prompt in one embodiment of the present invention. In the embodiment of Figure 6, an example is described in which the information provider intends to provide information that includes an advertisement for an advertiser's product or service.

[0058] The search system 320 may receive a user prompt 601 from a user's terminal connected via the network 170. For example, the prompt may correspond to a natural language-based search term entered by the user. The user may enter the search term into the user interface of the search service provided through the user's terminal, and the search system 320 may receive the search term entered into the user interface as a user prompt 601.

[0059] In this case, the search system 320 may select the prompt to actually use by analyzing the user prompt 601, extracting and summarizing the user intent during the User Intent Extracting & Summarizing 602 process.

[0060] On the other hand, the search system 320 may guide the user to provide sufficient information to provide an answer that reflects the advertiser's marketing message. For example, the content of the user prompt 601 may be insufficient to match the marketing message of a particular advertiser. In this case, the search system 320 may generate a question to guide the user to provide additional information to select a particular advertiser, and may provide the user with the question generated via the search system 320. After the user provides an answer to the question, the system may use the content of this answer to supplement the prompt. The Question Specification Prompt 603 may include a prompt derived from the user's answer.

[0061] At this time, the search system 320 may select a Specified User Prompt 604 that has been designated as a prompt for providing a marketing message. That is, the designated user prompt 604 may be specified based on the prompt obtained from the extraction and summarization of user intent 602 for the user prompt 601 and the question specification prompt 603.

[0062] A Prompt Ads Safe Check 605 may be an example of a process for verifying whether a given user prompt 604 is a prompt that may disclose an advertiser's marketing message to the user. For example, the search system 320 may generate and provide a response if the given user prompt 604 is not a prompt that requests pre-configured illegal information or pre-configured non-advertising information.

[0063] The search system 320 may also input the specified user prompt 604 into the LLM to generate LLM results. Figure 6 shows an example of an Organic LLM Result Memory 606 that records such LLM results.

[0064] The search system 320 may primarily select advertisers associated with LLM results based on the LLM results recorded in the organic LLM result memory 606. In this case, advertisers associated with LLM results may be advertisers who have registered marketing messages that may be published together with the LLM results. Marketing messages that may be published together with the LLM results may be selected based on the relevance between the information registered by the advertiser and the LLM results. In addition, depending on the embodiment, the search system 320 may use at least one of the user prompt 604, LLM results, and recommendation queries generated by the large language model when primarily selecting advertisers. In this case, the information provider selection system 310 may primarily select advertisers based on the relevance between at least one of the user prompt 604, LLM results, and recommendation queries and the information registered by the advertiser. In this case, the information provider selection system 310 may select a specific advertiser from among the advertisers primarily selected by the Ad Prompt Auction 607.

[0065] Once an advertiser is selected, the search system 320 retrieves the advertising assets 608 and advertiser prompts 609 registered by the selected advertiser. In this case, the search system 320 may use the user prompt 601 and at least one of the LLM results recorded in the organic LLM result memory 606, along with at least one of the advertising assets 608 and advertiser prompts 609, to generate an Answer Prompt 610 that reflects the advertiser's marketing message. In some embodiments, the advertiser may want to provide answers in a specific format depending on the user's characteristics. For this purpose, the search system 320 may generate the Answer Prompt 610 that further reflects information about the user. For example, information about the user may include at least one of the user's demos, interests, and purchase information. For example, the search system 320 may analyze the advertiser prompt 609 and understand that the advertiser wants to provide more detailed answers to female users than to male users. In this case, the search system 320 may determine the user's gender from the user's demo and then generate a response prompt 610 taking into account the determined user's gender.

[0066] After the response prompt 610 is generated, the search system 320 may verify whether the generated response prompt 610 is generated in a manner that conforms to the tone and / or format confirmed from the advertiser prompt 609. If the generated response prompt 610 does not conform to the tone and / or format desired by the advertiser, the search system 320 may modify the response prompt 610 to conform to the tone and / or format desired by the advertiser. Depending on the embodiment, the search system 320 may also perform additional verifications, such as whether the generated response prompt 610 is safe to publish.

[0067] After this, the search system 320 may provide the user with the finally generated answer 612. For example, the search system 320 may add the answer 612 to the search results and provide it to the user through the search service. Furthermore, depending on the embodiment, user information recorded in the DMP (Data Management Platform) 613 (for example, gender, age, interests, etc.) may be used to generate the response prompt 610. By utilizing such user information, the search system 320 can generate a response 612 optimized for the user.

[0068] On the other hand, in order to make efficient use of DMP613, advertiser prompt 609 may include further information regarding target characteristics and weightings. For example, target characteristics may include demo graphic (Gender, age (or age group)) ;Also referred to as “demo” in this application ), interests, and / or purchase information may be included. The weights may include, for example, weights for each target characteristic and / or weights for each content of a characteristic. For example, the weights for each characteristic may indicate how much weight is assigned to each of the target's gender, age, and interests. For example, a weight of 5 may be assigned to a female, a weight of 3 to a 20-year-old, and a weight of 8 to an interest in exercise. The weights for each content may indicate how much weight is assigned to each of the content of the same characteristic. For example, if the advertiser's interests are exercise, fashion, and games, the advertiser may assign weights of 8 to exercise, 6 to fashion, and 2 to games through the advertiser prompt 609. In this case, the search system 320 may further use the information about the target characteristics and weights desired by the advertiser included in the advertiser prompt 609 to generate the response prompt 610. Such information regarding target characteristics and weights may be optionally used in LLM if it is possible to utilize this information in LLM.

[0069] Figure 7 is a diagram illustrating an example of providing search results in one embodiment of the present invention. Figure 7 shows an example of a search page 700 screen provided to a user through a search service. The search page 700 may include a user interface 710 for entering user prompts. The search page 700 may also include a search results area 720 for displaying search results. In this case, the search results area 720 may include an LLM results area 730 for displaying LLM results generated based on Large Language Models (LLM) in response to user prompts.

[0070] Furthermore, the search results area 720 shows an example of an answer area 740 that displays the answers generated by the search system 320 in response to the user prompt. In this embodiment, an example is shown in which multiple answers are displayed in the answer area 740. Thus, it is possible to generate and display multiple answers in response to a single prompt. It is also possible to generate and display answers for two or more information providers. For this reason, the information provider selection system 310 may select two or more information providers.

[0071] On the other hand, the embodiment of Figure 7 shows an example in which the responses of an information provider selected based on user prompts entered into the user interface 710 and / or LLM results displayed in the LLM results area 730 are displayed in the extended area 750. For example, the extended area 750 may further display the responses of an information provider selected based on at least one of the user prompts, LLM results, and recommendation queries generated by the large language model. If the information provider is an advertiser, the extended area 750 may further display advertisements from the advertiser selected based on at least one of the user prompts, LLM results, and recommendation queries generated by the large language model.

[0072] Furthermore, the dotted frame 760 may display a question to request additional user prompts in relation to the answer displayed in the extended area 750. If the user selects a specific question, this question may be recognized as an additional user prompt. When providing a conversational search service, the additional user prompt may be recognized as the user's next conversation. In this case, the search system 320 may generate LLM results and / or answers while considering the entire conversation with the user.

[0073] Furthermore, while the embodiment in Figure 7 describes an example of dynamically generating answers by inputting user prompts into the user interface 710 of the search service, depending on the embodiment, the search results may also include an interface for generating and providing dynamic answers. For example, the user may be provided with a function to dynamically generate and provide answers by inputting user prompts through each and / or some of the various vertical services provided in the existing search ecosystem. Here, vertical services may mean services for various collections that classify search results, such as shopping search, knowledge search, local search, UGC (User Generated Contents) search, language search, image search, video search, and news search. For example, if a shopping search service is provided separately as a vertical service of the integrated search service, the user may be provided with a function to dynamically generate and provide answers by inputting user prompts through the shopping search service. If multiple different advertising services are provided as vertical services of the integrated search service, the user may also be provided with a function to dynamically generate and provide answers by inputting user prompts through each of these multiple advertising services.

[0074] Figure 8 shows an example of calculating the first relevance through LLM in one embodiment of the present invention. Figure 8 shows an example in which the information provider selection system 310 generates a question 830 and inputs it into LLM in order to obtain the first relevance between the advertiser's ad text 810 and the user's prompt 820 based on the advertiser's ad text 810 and the user's prompt 820. In this case, the LLM output 840 may include the first relevance between the ad text 810 and the user's prompt 820 (60% in the embodiment of Figure 8). Such an embodiment in Figure 8 is merely a visual representation of the LLM output 840 to aid in understanding the present invention, and it is not necessary for the LLM output 840 to actually be visually displayed. As an example, the information provider selection system 310 may extract and utilize the first relevance from the LLM output 840.

[0075] Figure 9 shows an example of calculating the second relevance through LLM in one embodiment of the present invention. Figure 9 shows an example in which the information provider selection system 310 generates a question 930 and inputs it into LLM in order to obtain the second relevance between the advertiser's ad copy 910 and the LLM result 920 generated based on LLM in response to the user prompt. In this case, the LLM output 940 may include the second relevance between the ad copy 910 and the LLM result 920 (40% in the embodiment of Figure 9). This embodiment of Figure 9 is merely a visual representation of the LLM output 940 to aid in understanding the present invention, and it is not necessary for the LLM output 940 to actually be visually displayed. As an example, the information provider selection system 310 may extract and utilize the second relevance from the LLM output 940.

[0076] Figures 10-14 show an example of a chat mode in one embodiment of the present invention, in which an LLM-based artificial intelligence provides an answer as a result of LLM through a conversation between the user and the AI. Such a chat mode may correspond to the second mode described above.

[0077] In the embodiment shown in Figure 10, an input interface 1010 is shown for receiving user prompts. The input interface 1010 may be linked to a virtual keyboard function for the user to enter text and / or a function for transmitting the text entered into the input interface 1010 to the search system 320. A session initialization interface 1011 may also be provided for initializing the current conversation session. The session initialization interface 1011 may be linked to a function for requesting the search system 320 to initialize the current conversation session and start a new conversation session. In the embodiment of Figure 10, an example is shown in which a session initialization interface 1011 with a specific icon shape is located to the left of the input interface 1010, but the shape, type (icon, button, link, etc.), position, etc. of the session initialization interface 1011 may be set in various ways depending on the embodiment. In addition, a multimedia input interface 1012 may be provided for receiving multimedia such as images and videos as user prompts, in addition to text. The multimedia input interface 1012 may be linked to functions for selecting and transmitting multimedia data stored on the user's terminal, or for transmitting multimedia data generated by a camera installed on the user's terminal.

[0078] On the other hand, the embodiment of Figure 10 shows a first area 1020 in which a user prompt transmitted to the search system 320 through the input interface 1010 is displayed in the form of a message for conversation. In conjunction with the first area 1020, a second area 1030 is shown for displaying the process of generating a response to the user prompt. The response generation process may include, but is not limited to, a "search" process, a "search result analysis" process, a "consider whether further search is needed" process, and a "response generation complete" process. In the embodiment of Figure 10, the second area 1030 shows the "response generation complete" process. The embodiment of Figure 10 also shows a third area 1030 in which the response generated in response to the user prompt is displayed. At this time, an icon 1031 to indicate the entity that provided the response may be displayed in conjunction with the third area 1030. For example, the icon 1031 may contain information that can be used to identify the search system 320.

[0079] The search system 320 may also provide the user with a first recommendation prompt 1040. In this case, the user can simply select the first recommendation prompt 1040, and it will be used as the user's prompt, allowing the user to continue the next conversation with the artificial intelligence in the current conversation session. The search system 320 may also provide the user with a second recommendation prompt 1050 for conversation with a specific information provider. In this case, an icon 1051 indicating the information provider in conjunction with the second recommendation prompt 1050 may be displayed in conjunction with the second recommendation prompt 1050. For example, the icon 1051 may include information such as images and text related to the information provider.

[0080] The embodiment in Figure 11 shows an example in which, in Figure 10, a second recommendation prompt 1050 is selected by the user, and a conversation with the user takes place through artificial intelligence specialized for a specific information provider. In this case, the artificial intelligence specialized for a specific information provider may also be artificial intelligence provided by the search system 320 based on LLM. Depending on the embodiment, it may be considered that the artificial intelligence specialized for a specific information provider is registered with the search system 320 by the specific information provider, or is provided by the specific information provider from the outset. In this case, the answers 1110 and 1120 provided by the artificial intelligence specialized for a specific information provider may display information 1130 to indicate that the answers 1110 and 1120 are provided by the specific information provider, and an icon 1051 for the information provider may be further displayed. Such answers 1110 and 1120 may be dynamically generated answers that reflect information (e.g., assets and prompts) registered in association with the information provider, as described above. The advertising cards included in response 1120 (advertising card 1 (1121), advertising card 2 (1122), and advertising card 3 (1123)) may each be produced in a form that includes a product image and a product description (product identifier, price, etc.).

[0081] The embodiment in Figure 12 shows an example where a user prompt for app recommendation is provided with a dynamically generated response for an app advertisement from a specific information provider. In this embodiment, Figure 12 shows an example where a response for specific branded content is dynamically generated and provided in the SGE (search Generative Experience) style.

[0082] The embodiment in Figure 13 shows an example in which the information provider's response is dynamically generated and provided in a different SGE-style format in response to a user prompt, which is the same as in Figure 12. Thus, the information provider's response may be dynamically generated and provided in a variety of formats and content in response to a prompt.

[0083] The embodiment in Figure 14 shows an example where, during a conversation with artificial intelligence, the information provider's response is provided as if it were another user in a group chat room. That is, by providing the information provider's response 1420 in conversational format, separate from the artificial intelligence's response 1410, the user can have an experience as if they were conversing with two or more other users in a group chat room.

[0084] Thus, according to embodiments of the present invention, a method and system for selecting information providers in response to user queries can be provided.

[0085] The systems or devices described above may be implemented by hardware components, or by a combination of hardware and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as processors, controllers, ALUs (arithmetic logic units), digital signal processors, microcomputers, FPGAs (field programmable gate arrays), PLUs (programmable logic units), microprocessors, or various devices capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications running on the OS. The processing unit may also respond to software execution, access data, record, manipulate, process, and generate data. For convenience of understanding, it may be described as if a single processing unit is used, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. Other processing configurations, such as parallel processors, are also possible.

[0086] Software may include computer programs, code, instructions, or a combination of one or more of these, which may configure a processing unit to operate as desired, or which may instruct the processing unit independently or collectively. Software and / or data may be embodied in any kind of machine, component, physical device, virtual equipment, computer recording medium, or device for interpretation based on the processing unit or for providing instructions or data to the processing unit. Software may be distributed across a network of computer systems, and may be recorded or executed in a distributed manner. Software and data may be recorded on one or more computer-readable recording media.

[0087] The methods according to the embodiment may be implemented in the form of program instructions executable by various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., individually or in combination. The medium may continuously record computer-executable programs or may temporarily record them for execution or download. The medium may also be a variety of recording or storage means in the form of a combination of one or more hardware components, and may be a medium directly connected to a computer system or distributed on a network. Examples of media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and media configured to record program instructions such as ROMs, RAMs, and flash memories. Other examples of media include recording media and storage media managed by app stores that distribute applications, and sites and servers that supply and distribute various other software. Examples of program instructions include not only machine code such as that generated by a compiler, but also high-level language code that is executed by a computer using an interpreter or the like.

[0088] As described above, embodiments have been explained based on limited embodiments and drawings, but those skilled in the art will be able to make various modifications and variations from the above description. For example, the described technique may be performed in a different order than described, and / or the components of the described system, structure, apparatus, circuit, etc. may be combined or assembled in a different manner than described, or opposed or replaced by other components or equivalents, and still achieve suitable results.

[0089] Therefore, even if the embodiment is different, if it is equivalent to the claims, it falls within the scope of the attached claims. [Explanation of Symbols]

[0090] 310: Information Provider Selection System 320: Search System 330: User 340:Informant

Claims

1. A method for selecting an information provider for a computer device including at least one processor, The steps include: verifying the LLM response result generated by the at least one processor based on a Large Language Model (LLM) in response to a user's question prompt; and The step of the at least one processor selecting at least one information provider from among the multiple information providers to provide the user with an instance of content related to the user's question prompt, reflecting the degree of relevance between each of the information provider's contents and the LLM response result. A method for selecting information providers, including the method itself.

2. The step of selecting at least one information provider is, At least the stage of selecting an information provider related to the LLM response results from among the multiple information providers, and The step of dynamically conducting an auction among the information providers selected in the first stage to select at least one information provider as the final information provider. The method for selecting an information provider according to claim 1, characterized by including the following:

3. The first selection step is: The user's question prompt and the LLM response result are to be primarily selected from among the multiple information providers. The method for selecting an information provider according to claim 2, characterized by the above.

4. The LLM is configured to further generate recommendation question prompts in response to the user's question prompts, The aforementioned recommendation question prompt can be selected by the user as the prompt for the next question. The method for selecting an information provider according to claim 2, characterized in that it is a method for selecting an information provider according to claim 2.

5. The stage of selecting the final information provider is as follows: Based on the ranking determined by the Quality Index (QI) and Bid Amount (BA) of each of the information providers selected in the first stage, at least one information provider is selected. The quality index is determined by at least the degree of correlation between the content of each of the multiple information providers and the LLM response results. The method for selecting an information provider according to claim 2, characterized by the above.

6. The step of selecting at least one information provider is as follows: To further reflect the degree of relevance between the content of each of the aforementioned multiple information providers and the prompts of the user's questions, to select at least one information provider. The method for selecting an information provider according to claim 1, characterized by the above.

7. The step of selecting at least one information provider is as follows: The at least one information provider is selected, taking into account the degree of relevance between at least one of the user's question prompts and the LLM response results and the analytical dimensions derived from the assets of the multiple information providers. The aforementioned analysis dimension includes at least one of the following: format, content, style, and tone and manner. The method for selecting an information provider according to claim 1, characterized by the above.

8. The aforementioned asset includes at least one of the following: a URL (Uniform Resource Locator) associated with the content that the information provider intends to provide, the title of the content, the identifier of the content, the category of the content, the multimedia associated with the content, the content of the content, and the content of the article associated with the content. The method for selecting an information provider according to claim 7, characterized by the above.

9. An instance of the content of the selected at least one information provider is dynamically generated using the LLM response results and the pre-registered assets of the selected at least one information provider, and provided to the user. A method for selecting an information provider according to any one of claims 1 to 8, characterized by the above.

10. An instance of content from the selected at least one information provider is dynamically generated by further utilizing at least one of the user's question prompts, the pre-registered prompts of the selected at least one information provider, and information about the user. The method for selecting an information provider according to claim 9, characterized by the above.

11. The pre-registered prompt includes, in order to be emphasized in relation to the content that the selected at least one information provider intends to provide, at least one of the following: a phrase or keyword entered by the selected at least one information provider, and the tone or format of the information message provided through the instance of the content. The information relating to the user includes at least one of the user's demographics, interests, and purchase information. The method for selecting an information provider according to claim 10, characterized by the above.

12. A computer program for causing the computer device to execute the information provider selection method described in any one of claims 1 to 8.

13. At least one processor implemented to execute instructions readable by a computer device Includes, With the aforementioned at least one processor, Review the LLM (Large Language Model) response generated based on the Large Language Model (LLM) in response to the user's question prompt. Select at least one information provider from among the multiple information providers to provide the user with an instance of content related to the user's question prompt, taking into account the degree of relevance between the content of each information provider and the LLM response results. A computer device characterized by the following:

14. In order to select the at least one information provider, the at least one processor is used to: At least the information provider related to the LLM response results is selected first from among the multiple information providers, Dynamically conduct an auction among the information providers selected in the first stage, and select at least one information provider as the final information provider. The computer device according to claim 13, characterized by the above.

15. In order to select the aforementioned final information provider, at least one processor is used, Based on the ranking determined by the Quality Index (QI) and Bid Amount (BA) of each of the information providers selected in the first stage, at least one information provider is selected. The quality index is determined by at least the degree of correlation between the content of each of the multiple information providers and the LLM response results. The computer device according to claim 14, characterized by the above.

16. In order to select the at least one information provider, the at least one processor is used to: To further reflect the degree of relevance between the content of each of the aforementioned multiple information providers and the prompts of the user's questions, to select at least one information provider. The computer device according to claim 13, characterized by the above.

17. In order to select the at least one information provider, the at least one processor is used to: The at least one information provider is selected, taking into account the degree of relevance between at least one of the user's question prompts and the LLM response results and the analytical dimensions derived from the assets of the multiple information providers. The aforementioned analysis dimension includes at least one of the following: format, content, style, and tone and manner. The computer device according to claim 13, characterized by the above.

Images