Method and system for recommending question for linkage with content of information provider as answer to user query

Abstract

To provide a method and a system for recommending a question for linkage with a content of an information provider as an answer to a question of a user.SOLUTION: The method for recommending a question according to an embodiment includes the steps of: generating an LLM result based on an LLM (Large Language Models: LLM) to a prompt of a user; and providing the generated question as a response to the prompt of the user along with the generated LLM result.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The following description relates to a method and system for recommending questions for collaboration with information provider content as answers to user queries. [Background technology]

[0002] A large language model (LLM) is a type of artificial intelligence trained on large amounts of text data to respond in human language to natural language input. It is a language model consisting of an artificial neural network with many parameters (usually billions or more). Such LLMs are trained on large amounts of unlabeled text using self-supervised or semi-self-supervised learning. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Korean Patent No. 10-2551531 Summary of the Invention [Problem to be solved by the invention]

[0004] A method and system are provided for recommending questions for collaboration with information provider content as answers to user queries. [Means for solving the problem]

[0005] A question recommendation method for a computer device including at least one processor is provided, the method including the steps of: generating, by the at least one processor, Large Language Models (LLM) results based on LLMs in response to a user prompt; generating, by the at least one processor, questions for linking with content of an information provider based on the LLM results; and providing, by the at least one processor, the generated questions together with the generated LLM results as responses to the user prompt.

[0006] According to one aspect, the step of generating questions may include a step of generating prompts for question generation related to the LLM results and the information provider content and inputting the prompts into a large-scale language model to generate a plurality of candidate questions, and a step of selecting at least one question to be provided to the user based on the relevance of the plurality of candidate questions to the LLM results.

[0007] According to another aspect, the relevance may be calculated based on at least one of: (1) the presence or amount of information provider content that can be provided by each of the multiple question candidates; (2) the quality or expected charge of the information provider content that can be provided by each of the multiple question candidates; (3) the compatibility of each of the multiple question candidates with the conversation context; and (4) the compatibility of each of the multiple question candidates with user characteristics.

[0008] According to another aspect, at least one of the relevance of (3) and the relevance of (4) may include at least one of a first relevance calculated based on the degree of overlap between the texts, thematic similarity, etc., and a second relevance calculated by analyzing sentence structure, vocabulary, relationships between sentences, etc. using natural language processing technology.

[0009] According to yet another aspect, the step of providing in response to the user prompt may be characterized as providing search results including the LLM results and the generated question via a search service in which the user prompt is entered.

[0010] According to another aspect, the step of providing as a response to the user prompt may be characterized by providing an answer including the LLM result and the generated question as an answer of the artificial intelligence through a conversation session between the user and the LLM-based artificial intelligence.

[0011] According to yet another aspect, the question recommendation method may further include dynamically generating, by at least one processor, an instance of information provider content associated with the question when the question provided as a response is selected by the user, and providing the instance to the user.

[0012] According to another aspect, instances for content may be dynamically generated for selected questions using LLM results generated based on a large-scale language model and pre-registered assets of information providers.

[0013] According to yet another aspect, the instances for the content may be characterized as being dynamically generated further utilizing at least one of questions, pre-registered prompts of the information provider, and information about the user.

[0014] A computer program is provided for causing a computer device to carry out the method.

[0015] A computer-readable recording medium is provided on which a program for causing a computer device to execute the method is recorded.

[0016] A computer device is provided, comprising at least one processor implemented to execute computer device-readable instructions, the at least one processor generating Large Language Models (LLM) results based on LLMs in response to user prompts, generating questions for linking with information provider content based on the LLM results, and providing the generated questions together with the generated LLM results as responses to the user prompts. Effect of the Invention

[0017] A method and system can be provided that recommends questions for collaboration with information provider content as answers to user queries. [Brief description of the drawings]

[0018] [Figure 1] FIG. 1 illustrates an example of a network environment in accordance with an embodiment of the present invention. [Diagram 2] FIG. 2 is a block diagram illustrating an example of a computing device in accordance with an embodiment of the present invention. [Diagram 3] FIG. 1 is a schematic diagram illustrating a question recommendation system according to an embodiment of the present invention. [Figure 4] 4 is a flow chart illustrating an example of a method for recommending questions in accordance with an embodiment of the present invention. [Diagram 5] FIG. 2 is an illustrative diagram showing a process of generating candidate questions using LLM in one embodiment of the present invention. [Figure 6] 1 is an exemplary diagram illustrating a process of providing an answer to a user prompt in accordance with an embodiment of the present invention. [Figure 7] FIG. 10 illustrates an example of providing search results in accordance with an embodiment of the present invention. [Figure 8] FIG. 13 is a diagram illustrating an example of calculating a first relevance through LLM in one embodiment of the present invention. [Figure 9]FIG. 13 is a diagram illustrating an example of calculating a second relevance through LLM in one embodiment of the present invention. [Figure 10] 1 is an illustrative diagram showing a chat mode in which an answer is provided as an LLM result through a conversation between an LLM-based artificial intelligence and a user in one embodiment of the present invention. [Figure 11] 1 is an illustrative diagram showing a chat mode in which an answer is provided as an LLM result through a conversation between an LLM-based artificial intelligence and a user in one embodiment of the present invention. [Figure 12] 1 is an illustrative diagram showing a chat mode in which an answer is provided as an LLM result through a conversation between an LLM-based artificial intelligence and a user in one embodiment of the present invention. [Figure 13] 1 is an illustrative diagram showing a chat mode in which an answer is provided as an LLM result through a conversation between an LLM-based artificial intelligence and a user in one embodiment of the present invention. [Figure 14] 1 is an illustrative diagram showing a chat mode in which an answer is provided as an LLM result through a conversation between an LLM-based artificial intelligence and a user in one embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings.

[0020] A question recommendation system according to an embodiment of the present invention may be realized by at least one computer device. In this case, a computer program according to an embodiment of the present invention may be installed and executed in the computer device realizing the question recommendation system, and the computer device may execute a question recommendation method according to an embodiment according to the control of the executed computer program. The above-mentioned computer program may be recorded in a computer-readable recording medium in combination with the computer device to cause the computer to execute the question recommendation method.

[0021] Fig. 1 is a diagram showing an example of a network environment in one embodiment of the present invention. The network environment in Fig. 1 shows an example including a plurality of electronic devices 110, 120, 130, 140, a plurality of servers 150, 160, and a network 170. Fig. 1 is merely an example for explaining the invention, and the number of electronic devices and the number of servers are not limited to those shown in Fig. 1.

[0022] The electronic devices 110, 120, 130, and 140 may be fixed terminals or mobile terminals realized by computer systems. Examples of the electronic devices 110, 120, 130, and 140 include smartphones, mobile phones, navigation systems, personal computers (PCs), notebook PCs, digital broadcasting terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), tablets, game consoles, wearable devices, internet of things (IoT) devices, virtual reality (VR) devices, and augmented reality (AR) devices. As an example, FIG. 1 shows a smartphone as an example of the electronic device 110, but in the embodiment of the present invention, the electronic device 110 may mean one of various physical computer systems that can communicate with other electronic devices 120, 130, and 140 and / or servers 150 and 160 via a network 170 using a substantially wireless or wired communication method.

[0023] The communication method is not limited, and may include not only a communication method utilizing a communication network (for example, a mobile communication network, a wired Internet, a wireless Internet, a broadcast network, a phase network, etc.) that the network 170 can include, but also a short-range wireless communication between devices. For example, the network 170 may include any one or more of a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), a broadband network (BBN), the Internet, etc. Furthermore, the network 170 may include any one or more of a network topology including, but not limited to, a bus network, a star network, a ring network, a mesh network, a star-bus network, a tree or a hierarchical network, etc.

[0024] Each of the servers 150 and 160 may be realized by one or more computer devices that communicate with the multiple electronic devices 110, 120, 130, and 140 via the network 170 to provide instructions, codes, files, contents, services, and the like. For example, the server 150 may be a system that provides a first service to the multiple electronic devices 110, 120, 130, and 140 connected via the network 170, and the server 160 may be a system that provides a second service to the multiple electronic devices 110, 120, 130, and 140 connected via the network 170. As a more specific example, the server 150 may provide a service (such as a search service, for example) targeted by an application, which is a computer program installed and executed in the multiple electronic devices 110, 120, 130, and 140, as a first service to the multiple electronic devices 110, 120, 130, and 140. As another example, the server 160 may provide, as a second service, a service of distributing files for installing and executing the above-mentioned applications to the multiple electronic devices 110, 120, 130, and 140.

[0025] 2 is a block diagram showing an example of a computer device according to an embodiment of the present invention. Each of the electronic devices 110, 120, 130, and 140 and the servers 150 and 160 described above may be realized by a computer device 200 shown in FIG.

[0026] Such a computer device 200 may include a memory 210, a processor 220, a communication interface 230, and an input / output interface 240, as shown in FIG. 2. The memory 210 is a computer-readable recording medium and may include a random access memory (RAM), a read only memory (ROM), and a permanent mass storage device such as a disk drive. Here, a permanent mass storage device such as a ROM or a disk drive may be included in the computer device 200 as a separate permanent storage device separate from the memory 210. The memory 210 may also store an operating system and at least one program code. Such software components may be loaded into the memory 210 from a computer-readable recording medium separate from the memory 210. Such separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD / CD-ROM drive, a memory card, etc. In another embodiment, the software components may be loaded into the memory 210 through a communication interface 230 that is not a computer-readable recording medium. For example, the software components may be loaded into the memory 210 of the computing device 200 based on a computer program installed by a file received over the network 170 .

[0027] The processor 220 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. The instructions may be provided to the processor 220 by the memory 210 or the communication interface 230. For example, the processor 220 may be configured to execute instructions received according to program code stored in a storage device such as the memory 210.

[0028] The communication interface 230 may provide a function for the computer device 200 to communicate with other devices (for example, the above-mentioned recording device) via the network 170. For example, requests, instructions, data, files, etc. generated by the processor 220 of the computer device 200 according to program codes recorded in a recording device such as the memory 210 may be transmitted to other devices via the network 170 under the control of the communication interface 230. Conversely, signals, instructions, data, files, etc. from other devices may be received by the computer device 200 via the communication interface 230 of the computer device 200 via the network 170. The signals, instructions, data, etc. received via the communication interface 230 may be transmitted to the processor 220 or the memory 210, and the files, etc. may be recorded on a recording medium (the above-mentioned permanent recording device) that the computer device 200 may further include.

[0029] The input / output interface 240 may be a means for interfacing with the input / output device 250. For example, the input device may include a device such as a microphone, a keyboard, or a mouse, and the output device may include a device such as a display and a speaker. As another example, the input / output interface 240 may be a means for interfacing with a device in which functions for input and output are integrated into one, such as a touch screen. The input / output device 250 may be configured as a single device together with the computer device 200.

[0030] Also, in other embodiments, computing device 200 may include fewer or more components than those of Figure 2. However, most prior art components need not be explicitly illustrated. For example, computing device 200 may be implemented to include at least some of the input / output devices 250 described above, and may further include other components such as a transceiver, a database, etc.

[0031] 3 is a schematic diagram of a question recommendation system according to an embodiment of the present invention, showing a search system 320, a number of users 330, and a number of information providers 340.

[0032] The search system 320 may correspond to a server (for example, the server 150) that provides a search service to a plurality of users 330, and may be realized by at least one computer device 200. Here, each of the plurality of users 330 may be a physical device of the user that connects to the search system 320 via the network 170 to receive the search service, and such a physical device may be realized by the computer device 200 described above.

[0033] The question recommendation system 310 according to the present embodiment may be included in the search system 320, or may be implemented in a form linked with the search system 320 via the network 170. The embodiment of FIG. 3 illustrates an example in which the question recommendation system 310 is implemented in a form included in the search system 320. In this case, the question recommendation system 310 may be implemented on at least one physical device for implementing the search system 320. Depending on the embodiment, the question recommendation system 310 may be implemented in a physical device separate from the physical device for implementing the search system 320, and may be implemented in a form communicating with the search system 320 via the network 170.

[0034] The search service provided by the search system 320 to the multiple users 330 may include search results corresponding to the user's input. The search results may be generated based on information searchable on the web. The search system 320 may also provide a search service by including information to be provided by the multiple information providers 340 (instances for the information provider's content) in the search results. Here, the information provided by the multiple information providers 340 may be advertising information, but is not limited to this. The search service itself that provides such search results is well known, and therefore a detailed description will be omitted.

[0035] Meanwhile, the search system 320 according to the present embodiment may provide a search service by including an answer based on artificial intelligence such as a large language model (LLM) in the search results. For example, assume that the search system 320 receives a natural language-based prompt from a specific user among the multiple users 330. In this case, the search system 320 may input the received prompt into the LLM, generate a first answer suitable for the prompt as an LLM result, and provide the user with a search result including the first answer. At this time, the search result may include at least a part of various existing search results in addition to the first answer. In addition, the search system 320 may provide a search service by a conversation between an LLM-based artificial intelligence and a user. A search service may be provided to a user by switching between a first mode in which a first answer is provided as an LLM result based on a general search service and a second mode in which a first answer is provided as an LLM result based on a conversation between an LLM-based artificial intelligence and a user. At this time, a user interface for switching to the second mode may be provided in the first mode, and a user interface for switching to the first mode may be provided in the second mode. Additionally, in each of the first and second modes, at least a portion of the first response may further include an instance for the information provider's content being provided as a second response.

[0036] The search system 320 may also select from among a number of information providers 340 an information provider to provide an instance of its content as a second response to the user's prompt.

[0037] First, the search system 320 may process a stability verification process to verify whether the natural language based prompt received from the user is a prompt for which the information provider's content may be provided and / or whether the information provider's content may be provided in conjunction with an answer prompt to the user's prompt (for example, a first answer generated by the LLM to the user's prompt). For example, if the information provider is an advertiser who wishes to publish his / her advertisement, the advertiser may not want his / her advertisement to be published in response to a prompt requesting pre-set illegal information or pre-set non-advertising information. Also, the answer to the user's prompt may include an industry or keyword that the information provider does not want. Therefore, the search system 320 pre-verifies whether the user's prompt or the answer prompt to the user's prompt is a safe prompt for providing the information provider's content. In this case, the search system 320 may prevent the information provider's content from being provided in response to a legally questionable prompt such as a prompt requesting illegal information. Also, the search system 320 may exclude the information provider in the process of recommending questions in response to a prompt that does not comply with the information provider's policy. For example, if information provider A's policy prohibits it from providing its content in response to a prompt related to a specific industry b or a prompt containing keyword c, information provider A may be excluded from selection for a prompt related to that industry b or a prompt containing keyword c.

[0038] Furthermore, when there are multiple information providers who wish to make their information public, the search system 320 may select an information provider to provide content from among the multiple information providers who have passed the above-mentioned verification process. As an example, the information provider may be selected by an auction. As the auction method, one of well-known methods may be used, for example, a generalized second price (GSP) auction method. In this case, the ranking of the auction may be determined by a quality second price and a bid amount (BA).

[0039] When an information provider is selected, the search system 320 may dynamically generate an instance for the content of the selected information provider and provide it to the user as a second answer. For example, the search system 320 may provide the user with a search result that includes not only the first answer generated using the LLM but also the second answer described above.

[0040] In this case, the search system 320 may dynamically generate the AI-based second answer using the user's prompt, the first answer generated using the LLM, assets registered by the information provider, and / or prompts registered by the information provider, instead of providing the information provided by the information provider as is. Here, the assets may include, for example, a uniform resource locator (URL) related to the content to be provided by the information provider, a title or identification name of the content, a content category, multimedia related to the content, the content, and the content of an article related to the content. Here, the multimedia related to the content may include an image or a video related to the content. For example, if the information provider is an advertiser who wants to advertise a specific product or service, the assets may include a URL related to the product or service, a product name or a service name, a product or service category, product information or service information, and the content of an article related to the product or service. In addition, the prompt registered by the information provider may include information such as a phrase or keyword to be emphasized in relation to the content to be provided by the information provider, and a tone or format of an information message to be provided as the second answer. In this way, the search system 320 can provide the user with search results that include not only a first answer generated using the LLM in response to a user's natural language-based prompt, but also a second answer that is dynamically generated by taking into account the registered assets and prompts of an information provider who wishes to provide their own information, as well as the first answer.

[0041] In addition, depending on the embodiment, the search system 320 may generate the second answer by further using information about the user. Here, the information about the user may include the user's demo, interests, purchasing information, etc., and may be used to customize the second answer for the user.

[0042] In this manner, when the search system 320 provides an answer to a natural language-based prompt from a user, the question recommendation system 310 selects a specific information provider from among the multiple information providers 340, and the search system 320 dynamically generates an answer based on the assets and prompt of the selected information provider, i.e., an AI-based second answer in which the message of the information provider is projected. Thus, the search system 320 can provide the user with an answer that is dynamically generated so that the message of the information provider is projected in association with the natural language-based prompt received from the user.

[0043] In some embodiments, the content of a user prompt may be insufficient to match a marketing message of a particular advertiser. In this case, the search system 320 may provide the user with a question to induce the user to include sufficient information in the user prompt for the matching. Such a question may also be generated by the LLM, and information obtained as the user's answer to the question may also be included in the user prompt.

[0044] The search system 320 may further provide the user with a question generated by the question recommendation system 310 to be recommended to the user as a search result. Such a question may be a question for linking with the information provider's content and may include information for inducing the user's interest in the information provider's content. When the user selects a question included in the search result, an instance of the information provider's content linked to the selected question may be provided to the user. In this case, the provided instance may also be dynamically generated like the above-mentioned second answer.

[0045] 4 is a flowchart showing an example of a question recommendation method according to an embodiment of the present invention. The question recommendation method according to this embodiment may be executed by a computer device 200 that realizes the above-mentioned question recommendation system 310. In this case, the processor 220 of the computer device 200 may be realized to execute control instructions according to the operating system code and at least one computer program code included in the memory 210. Here, the processor 220 may control the computer device 200 to execute steps 410 and 440 included in the method of FIG. 4 according to the control instructions provided by the code recorded in the computer device 200.

[0046] In step 410, the computer device 200 may generate an LLM result based on Large Language Models (LLM) in response to a user prompt. Here, the user prompt may include, but is not limited to, a user query input through a search service provided by the search system 320. As an example, the search system 320 may provide a conversation function between an artificial intelligence and a user, and information input by the user through such a conversation function may be used as a user prompt. In this case, an LLM result generated by the search system 320 using the LLM in response to the information input by the user may be confirmed by the computer device 200 in step 410. In the above, a first mode in which a first answer is provided as an LLM result through a general search service and a second mode in which a first answer is provided as an LLM result through a conversation between an LLM-based artificial intelligence and a user have been described, and a search service is provided to a user by switching between the first and second modes. The conversation function realized by the second mode may be realized in the form of a sub-service included in the search service, or may be realized in the form of a service separate from the search service that is linked to the search service.

[0047] In step 420, the computer device 200 may generate questions for linking with the information provider's content based on the LLM results. As an example, the computer device 200 may generate question candidates by generating a prompt for requesting the creation of a question related to the LLM results and the information provider's content and inputting the LLM. After this, the computer device 200 may select at least one question to be provided to the user from among the question candidates based on the relevance of the generated question candidates to the LLM results. In this case, the relevance of the question candidates to the LLM results may be calculated based on at least one of (1) the presence or absence or number of content from the information provider that can be provided by each of the question candidates, (2) the quality or expected charge of the content from the information provider that can be provided by each of the question candidates, (3) the suitability of each of the question candidates to the conversation context, and (4) the suitability of each of the question candidates to the user characteristics (such as the user's demographics and interests). The relevance in (3) and (4) may include relevance calculated based on the degree of overlap between texts, topic matching, etc., and / or relevance calculated using natural language processing techniques to analyze sentence structure, vocabulary, relationships between sentences, etc.

[0048] In step 430, the computing device 200 may provide the generated question along with the generated LLM result as a response to the user's prompt. As an example, in the first mode described above, when providing a search result including the first answer and the second answer, the generated question may be provided in conjunction with the first answer. As another example, in the second mode, the generated question may be provided to the user in a conversation session in conjunction with an answer to the user's prompt.

[0049] In step 440, the computer device 200 may provide an instance of the information provider's content to the user when a question is selected by the user. The generated question may include information for inducing the user's interest in the information provider's content associated with the user's prompt and / or LLM result. In this case, when the user selects the provided question, the computer device 200 may dynamically generate an instance of the information provider's content associated with the selected question and provide it to the user. The instance provided as described above may also be dynamically generated and provided to the user like the second answer described above.

[0050] FIG. 5 is a diagram showing an example of an advertisement instance dynamically created in an embodiment of the present invention. FIG. 5 shows an example of an advertisement instance 540 generated by considering all of an asset 510 for an advertisement, an LLM result 520 generated by the LLM in response to a user's prompt, and an advertiser's prompt 530. In FIG. 5, "AAA" means the advertiser's product name, and "BBB" means the advertiser's brand name. In addition, in the prompt 530, "ad" means an advertisement, "SEO" means search engine optimization, and "organic" means the LLM result 520. In this way, the search system 320 can generate an advertisement instance 540 based on the LLM by considering both the LLM result 520 generated by the LLM in response to a user's prompt and the asset 510 and the prompt 530 registered by the advertiser as an information provider. As a more specific example, the search system 320 may extract multiple prompts for the LLM from each of the LLM results 520, assets 510, and prompts 530, and input the extracted prompts into the LLM to generate the advertisement instance 540.

[0051] In the embodiment of FIG. 5, an example is described in which the assets 510 contain only text and a text-based advertisement instance 540 is generated, but if an asset collection including various multimedia such as images and videos is used, the search system 320 can provide advertisement instances in more diverse formats including images, videos, etc.

[0052] 6 is an exemplary diagram illustrating a process of providing an answer to a user's prompt in an embodiment of the present invention. In the embodiment of FIG. 6, an example is described in which an information provider wishes to publish an advertisement for an advertiser's product or service as information to be provided.

[0053] The search system 320 may receive a user prompt 601 from a user terminal connected via the network 170. As an example, the prompt may correspond to a natural language-based search term input by the user. The user may input the search term into a user interface of a search service provided through the user's terminal, and the search system 320 may receive the search term input into the user interface as the user prompt 601.

[0054] At this time, the search system 320 may select the prompt to be actually used by analyzing the user prompt 601 and extracting and summarizing the user intent in the process of extracting and summarizing the user intent (User Intent Extracting & Summarizing 602).

[0055] Meanwhile, the search system 320 may guide the user to provide sufficient information to provide an answer that reflects the marketing message of the advertiser. As an example, the content of the user prompt 601 may be insufficient to match the marketing message of a particular advertiser. In this case, the search system 320 may generate a question to induce additional information for selecting a particular advertiser, and may provide the generated question to the user through the search system 320. Thereafter, when an answer to the question is received from the user, the content of the answer may be used to complete the prompt. The question specification prompt 603 may include a prompt derived from the user's answer.

[0056] At this time, the search system 320 may select a specified user prompt 604 as a prompt for providing the marketing message. That is, the specified user prompt 604 may be specified based on a prompt obtained from the extraction and summary 602 of the user intent for the user prompt 601 and a question specification prompt 603.

[0057] Prompt Ads Safe Check 605 may be an example of a process that verifies whether a specified user prompt 604 is a prompt that is OK for exposing an advertiser's marketing message to a user. As an example, the search system 320 may generate and provide an answer if the specified user prompt 604 is not a prompt requesting pre-defined illegal information or pre-defined non-advertising information.

[0058] The search system 320 may also input specified user prompts 604 into the LLM to generate LLM results. Figure 6 shows an example of an Organic LLM Result Memory 606 that stores such LLM results.

[0059] The search system 320 may initially select an advertiser associated with the LLM result based on the LLM result recorded in the organic LLM result memory 606. In this case, the advertiser associated with the LLM result may be an advertiser who has registered a marketing message that may be published together with the LLM result. The marketing message that may be published together with the LLM result may be selected based on the relevance between the information registered by the advertiser and the LLM result. In addition, according to an embodiment, the search system 320 may use at least one of the user prompt 604, the LLM result, and a recommendation query generated by a large-scale language model when initially selecting an advertiser. In this case, the search system 320 may initially select an advertiser based on the relevance between at least one of the user prompt 604, the LLM result, and the recommendation query and the information registered by the advertiser. In this case, the search system 320 may select a specific advertiser from among the initially selected advertisers through an Ad Prompt Auction 607.

[0060] Once an advertiser is selected, the search system 320 may obtain an advertisement asset 608 registered by the selected advertiser and an advertiser prompt 609 registered by the selected advertiser. In this case, the search system 320 may generate an answer prompt 610 reflecting the advertiser's marketing message by using the user prompt 601, at least one of the LLM results recorded in the organic LLM result memory 606, and at least one of the advertisement asset 608 and the advertiser prompt 609. In some embodiments, the advertiser may want to provide an answer in a specific format according to the characteristics of the user. For this reason, the search system 320 may generate the answer prompt 610 by further reflecting information about the user. As an example, the information about the user may include at least one of the user's demo, interests, and purchase information. For example, the search system 320 may analyze the advertiser prompt 609 and find out that the advertiser wants a relatively more detailed answer to be provided to female users than to male users. In this case, the search system 320 may determine the gender of the user from the user's demo, and then generate the answer prompt 610 taking into consideration the determined gender of the user.

[0061] After the answer prompt 610 is generated, the search system 320 may check (611) whether the generated answer prompt 610 is generated to suit the tone and / or format identified from the advertiser prompt 609. If the generated answer prompt 610 does not match the tone and / or format desired by the advertiser, it is also possible to modify the answer prompt 610 to match the tone and / or format desired by the advertiser. In addition, depending on the embodiment, the search system 320 may additionally check whether the generated answer prompt 610 is safe to be made public, etc.

[0062] Thereafter, the search system 320 may provide the finally generated answer 612 to the user through the search system 320. As an example, the search system 320 may add the answer 612 to the search results and provide it to the user through the search service. In addition, depending on the embodiment, user information (e.g., gender, age, interests, etc.) recorded in a DMP (Data Management Platform) 613 may be further used to generate the answer prompt 610. By utilizing such user information, the search system 320 can generate an answer 612 optimized for the user.

[0063] Meanwhile, in order to efficiently utilize the DMP 613, the advertiser prompt 609 may further include information regarding the characteristics and weightings of the target. For example, the characteristics of the target may include demographics (gender, age (or age group)), interests, and / or purchase information. For example, the weightings may include the weightings of each characteristic of the target and / or the weightings of each content of the characteristics. For example, the weightings of each characteristic may indicate how much weighting is given to which characteristic among the gender, age, and interests of the target. For example, if the gender is female, a weighting of 5 may be given, if the age is in the 20s, a weighting of 3 may be given, and if the interest is sports, a weighting of 8 may be given. In addition, the weightings of each content may indicate how much weighting is given to which content among the contents of the same characteristic. For example, if the interests set by the advertiser are sports, fashion, and games, the advertiser may give a weighting of 8 to sports, 6 to fashion, and 2 to games through the advertiser prompt 609. In this case, the search system 320 may further utilize information regarding advertiser desired target characteristics and weightings contained in the advertiser prompt 609 to generate the answer prompt 610. Such information regarding target characteristics and weightings may optionally be utilized if available in the LLM.

[0064] Also, when a question generated and provided by the question recommendation system 310 is selected by the user, the selected question may be used as a user prompt 601 and the process of Fig. 6 may be executed again to dynamically generate an instance for the information provider's content. At this time, a specific information provider may be associated with the question, and in this case, processes such as security check and auction for selecting an information provider may be omitted.

[0065] FIG. 7 is a diagram showing an example of providing search results in an embodiment of the present invention. FIG. 7 shows an example of a screen of a search page 700 provided to a user through a search service. The search page 700 may include a user interface 710 for inputting a user prompt. The search page 700 may also include a search result area 720 for displaying search results. In this case, the search result area 720 may include an LLM result area 730 for displaying LLM results generated based on the LLM in response to the user prompt.

[0066] The search result area 720 also shows an example of an answer area 740 that displays an answer generated by the search system 320 in response to a user's prompt. In this embodiment, an example is shown in which a plurality of answers are displayed in the answer area 740. In this manner, a plurality of answers can be generated and displayed in response to one prompt. It is also possible to generate and display answers for each of two or more information providers. For this reason, the search system 320 may select two or more information providers.

[0067] On the other hand, the embodiment of FIG. 7 illustrates an example in which answers of information providers selected based on the user's prompt input to the user interface 710 and / or the LLM results displayed in the LLM result area 730 are displayed in the expansion area 750. For example, the expansion area 750 may further display answers of information providers selected based on at least one of the user's prompt, the LLM result, and the recommendation query generated by the large-scale language model. If the information provider is an advertiser, the expansion area 750 may further display advertisements of the advertisers selected based on at least one of the user's prompt, the LLM result, and the recommendation query generated by the large-scale language model.

[0068] Also, the dotted frame 760 displays a question for requesting an additional prompt from the user in relation to the answer displayed in the expansion area 750. When the user selects a specific question, this question may be recognized as the user's additional prompt. When providing a conversational search service, the user's additional prompt may be recognized as the user's next conversation. In this case, the search system 320 may generate LLM results and / or answers by taking into account the entire conversation with the user.

[0069] In addition, in the embodiment of FIG. 7, an example in which a user's prompt is input to the user interface 710 of the search service to dynamically generate an answer has been described. However, depending on the embodiment, the search results may also include an interface for dynamically generating and providing an answer. For example, a function for dynamically generating and providing an answer by inputting a user's prompt through each and / or some of various vertical services provided in the existing search ecosystem may be provided to the user. Here, the vertical service refers to a service for each of various collections that classify search results, such as shopping search, knowledge search, local search, UGC (User Generated Contents) search, language search, image search, video search, and news search. For example, when a shopping search service is provided separately as a vertical service of the integrated search service, a function for dynamically generating and providing an answer may be provided to the user by inputting a user's prompt in the shopping search service. When a plurality of different advertising services are provided as vertical services of the integrated search service, a function for dynamically generating and providing an answer may be provided to the user by inputting a user's prompt in each of the plurality of advertising services.

[0070] In addition, the embodiment of FIG. 7 illustrates an example of recommending an additional question to the user. In this case, the dotted frame 770 illustrates an example of recommending a question generated by the question recommendation system 310 in association with the content of an information provider. If the user selects a question illustrated in the dotted frame 770, the process of providing an answer described with reference to FIG. 6 may be executed using this question as a user prompt. In this case, if a question is generated for the content of a specific information provider, the process of selecting an information provider may be omitted as described above.

[0071] FIG. 8 is a diagram showing an example of calculating the first relevance using the LLM in one embodiment of the present invention. FIG. 8 shows an example in which the search system 320 generates a question 830 and inputs it to the LLM to obtain the first relevance between the advertisement copy 810 and the user's prompt 820 based on the advertisement copy 810 of the advertiser and the user's prompt 820. In this case, the output 840 of the LLM may include the first relevance between the advertisement copy 810 and the user's prompt 820 (60% in the embodiment of FIG. 8). Such an embodiment of FIG. 8 merely visually displays the output 840 of the LLM to help understand the present invention, and the output 840 of the LLM does not need to be actually visually displayed. For example, the search system 320 may simply extract and utilize the first relevance from the output 840 of the LLM.

[0072] FIG. 9 is a diagram showing an example of calculating the second relevance using the LLM in an embodiment of the present invention. FIG. 9 shows an example in which the search system 320 generates a question 930 and inputs it to the LLM to obtain a second relevance between the advertisement copy 910 of the advertiser and the LLM result 920 generated based on the LLM in response to the user's prompt. In this case, the output 940 of the LLM may include the second relevance between the advertisement copy 910 and the LLM result 920 (40% in the embodiment of FIG. 9). The embodiment of FIG. 9 is merely a visual display of the output 940 of the LLM to help understand the present invention, and the output 940 of the LLM does not need to be actually visually displayed. As an example, the search system 320 may simply extract and use the second relevance from the output 940 of the LLM.

[0073] 10 to 14 show an example of a chat mode in which an answer that is an LLM result is provided through a conversation between an LLM-based artificial intelligence and a user in one embodiment of the present invention. Such a chat mode may correspond to the second mode described above.

[0074] In the embodiment of FIG. 10, an input interface 1010 for inputting a user's prompt is shown. The input interface 1010 may be associated with a virtual keyboard function for the user to input text and / or a function for transmitting the text input to the input interface 1010 to the search system 320. In addition, a session initialization interface 1011 for initializing a current conversation session may be provided. The session initialization interface 1011 may be associated with a function for requesting the search system 320 to initialize the current conversation session and start a new conversation session. In the embodiment of FIG. 10, an example is shown in which the session initialization interface 1011 in a specific icon shape is arranged on the left side of the input interface 1010, but the shape, type (icon, button, link, etc.), position, etc. of the session initialization interface 1011 can be set in various ways depending on the embodiment. In addition, a multimedia input interface 1012 for inputting multimedia such as images and videos in addition to text as a user's prompt may be provided. The multimedia input interface 1012 may be associated with a function for selecting and transmitting multimedia data stored in the user's terminal, or transmitting multimedia data generated through a camera provided in the user's terminal.

[0075] Meanwhile, in the embodiment of FIG. 10, a first area 1020 is shown in which a user's prompt transmitted to the search system 320 through the input interface 1010 is displayed in the form of a message for conversation. Also, a second area 1021 is shown in association with the first area 1020 for displaying a process of generating an answer to the user's prompt. The process of generating an answer may include, for example, a "search" process, a "search result analysis" process, a "consider whether further search is necessary" process, and an "answer generation completion" process, but is not limited thereto. In the embodiment of FIG. 10, the second area 1021 shows the "answer generation completion" process. Also, the embodiment of FIG. 10 shows a third area 1030 in which an answer generated in response to the user's prompt is displayed. At this time, an icon 1031 for indicating an entity that provided the answer may be displayed in association with the third area 1030. For example, the icon 1031 may include information that can be used to identify the search system 320.

[0076] The search system 320 may further provide the user with a first recommendation prompt 1040. In this case, the user can continue the next conversation with the artificial intelligence in the current conversation session by simply selecting the first recommendation prompt 1040, with the first recommendation prompt 1040 being used as the user's prompt. The search system 320 may further provide the user with a second recommendation prompt 1050 for a conversation with a specific information provider. In this case, an icon 1051 for indicating the information provider may be displayed in association with the second recommendation prompt 1050. For example, the icon 1051 may include information such as an image or text related to the information provider.

[0077] The embodiment of FIG. 11 illustrates an example in which a conversation with a user is conducted through an artificial intelligence specialized for a specific information provider by the user selecting the second recommendation prompt 1050 in FIG. 10. In this case, the artificial intelligence specialized for a specific information provider may also be an artificial intelligence provided by the search system 320 based on the LLM. In some embodiments, the artificial intelligence specialized for a specific information provider may be registered in the search system 320 by the specific information provider, or may be provided by the specific information provider from the beginning. In this case, the answers 1110 and 1120 provided by the artificial intelligence specialized for a specific information provider may display information 1130 for notifying that the answers 1110 and 1120 are provided by a specific information provider, and an icon 1051 for the information provider may further be displayed. Such answers 1110 and 1120 may be dynamically generated by reflecting information (for example, assets, prompts, etc.) associated with the information provider and registered, as described above. Each of the advertisement cards (advertisement card 1 (1121), advertisement card 2 (1122), and advertisement card 3 (1123)) included in the answer 1120 may be produced in a form including a product image and a product description (such as a product identifier and price).

[0078] The embodiment of Fig. 12 illustrates an example where a dynamically generated answer is provided for a specific information provider's app advertisement in response to a user's prompt for an app recommendation, where a specific brand content answer is dynamically generated and provided in a search generative experience (SGE) style.

[0079] The embodiment of Figure 13 illustrates an example where information provider responses are dynamically generated and provided in other SGE style formats in response to the same user prompts as those shown in Figure 12. In this manner, information provider responses can be dynamically generated and provided in a variety of formats and content in response to prompts.

[0080] The embodiment of Fig. 14 shows an example in which an information provider's response is provided during a conversation with an artificial intelligence as if it were another user in a group chat room. That is, the information provider's response 1420 is provided in a conversational format in addition to the artificial intelligence's response 1410, allowing the user to experience an experience as if they were having a conversation with two or more other users in a group chat room.

[0081] Thus, embodiments of the present invention provide a method and system for recommending questions for linking with information provider content as answers to user queries.

[0082] The above-described systems or devices may be realized by hardware components or a combination of hardware and software components. For example, the devices and components described in the embodiments may be realized using one or more general-purpose or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or various devices capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications that run on the OS. The processing device may also respond to the execution of the software to access, record, manipulate, process, and generate data. For convenience of understanding, a single processing device may be described as being used, but those skilled in the art will understand that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, the processing device may include multiple processors or one processor and one controller. Other processing configurations, such as parallel processors, are also possible.

[0083] The software may include computer programs, codes, instructions, or a combination of one or more of these, and may configure a processing device to operate as desired or instruct the processing device, either independently or collectively. The software and / or data may be embodied in any type of machine, component, physical device, virtual device, computer storage medium, or device to be interpreted based on the processing device or to provide instructions or data to the processing device. The software may be distributed and stored or executed in a distributed manner on computer systems connected by a network. The software and data may be stored on one or more computer-readable storage media.

[0084] The method according to the embodiment may be realized in the form of program instructions executable by various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The medium may be a medium that continuously records a program executable by a computer, or a medium that temporarily records the program for execution or download. The medium may be a variety of recording or storage means in the form of a single or multiple hardware devices combined, and may be a medium directly connected to a certain computer system, or may be distributed on a network. Examples of the medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and media configured to record program instructions such as ROMs, RAMs, flash memories, and the like. Other examples of the medium include recording media and storage media managed by app stores that distribute applications, sites that supply or distribute various other software, and servers. Examples of the program instructions include not only machine language codes such as those generated by compilers, but also high-level language codes executed by a computer using an interpreter, etc.

[0085] Although the embodiments have been described based on the limited embodiments and drawings as above, those skilled in the art will be able to make various modifications and variations from the above description. For example, the described techniques may be performed in an order different from that described, and / or the components of the described systems, structures, devices, circuits, etc. may be combined or combined in a manner different from that described, or may be counterbalanced or replaced by other components or equivalents, and still achieve suitable results.

[0086] Therefore, different embodiments are within the scope of the appended claims provided that they are equivalent to the claims. [Explanation of symbols]

[0087] 310: Question recommendation system 320: Search system 330: User 340:Informant

Claims

1. 1. A method for recommending questions in a computer device including at least one processor, comprising: generating, by the at least one processor, a large language model (LLM) result based on the LLM in response to a user prompt; generating, by the at least one processor, a query for linking with information provider content based on the LLM results; and providing, by the at least one processor, the generated question along with the generated LLM results as a response to a prompt from the user. Question recommendation method.

2. The step of generating a query comprises: generating prompts for generating questions related to the LLM results and the information provider content and inputting the prompts into the large-scale language model to generate a plurality of candidate questions; and selecting at least one question to be provided to the user based on a relevance of the plurality of candidate questions to the LLM result, The method for recommending questions according to claim 1 .

3. 3. The question recommendation method according to claim 2, characterized in that the relevance is calculated based on at least one of: (1) the presence or absence or number of content from an information provider that can be provided by each of the multiple question candidates; (2) the quality or expected charge of the content from an information provider that can be provided by each of the multiple question candidates; (3) the suitability of each of the multiple question candidates with a conversation context; and (4) the suitability of each of the multiple question candidates with user characteristics.

4. The question recommendation method of claim 3, characterized in that at least one of the relevance of (3) and the relevance of (4) includes at least one of a first relevance calculated based on the degree of overlap between texts, a matching situation of themes, etc., and a second relevance calculated by analyzing sentence structure, vocabulary, relationships between sentences, etc. using natural language processing technology.

5. providing in response to the user prompt, providing a search result including the LLM results and the generated question through a search service that inputs the user's prompts. The method for recommending questions according to any one of claims 1 to 4.

6. providing in response to the user prompt, The LLM result and an answer including the generated question are provided as an answer of the artificial intelligence through a conversation session between the user and the LLM-based artificial intelligence. The method for recommending questions according to any one of claims 1 to 4.

7. The method for recommending a question according to any one of claims 1 to 4, further comprising: when the question provided as the response is selected by the user, dynamically generating, by the at least one processor, an instance of content of an information provider related to the question and providing the content to the user.

8. The question recommendation method according to claim 7, characterized in that the instance for the content is dynamically generated for the selected question by utilizing LLM results generated based on the large-scale language model and already registered assets of information providers.

9. The method for recommending questions according to claim 8, characterized in that the instance for the content is dynamically generated by further utilizing at least one of the question, a pre-registered prompt of the information provider, and information about the user.

10. A computer program product for causing said computer device to carry out the method according to any one of claims 1 to 4.

11. 1. A computing device comprising: at least one processor implemented to execute computer readable instructions; by said at least one processor, generating a large language model (LLM) result based on the LLM in response to a user prompt; generating a query for linking with information provider content based on the LLM results; providing the generated question along with the generated LLM result as a response to a prompt from the user. Computer equipment.

12. generating the query by the at least one processor; generating prompts for generating questions related to the LLM results and the information provider content and inputting the prompts into the large-scale language model to generate a plurality of candidate questions; At least one question to be provided to the user is selected based on a relevance of the plurality of question candidates to the LLM result.

12. The computer device of claim 11.

13. The computer device of claim 12, characterized in that the relevance is calculated based on at least one of: (1) the presence or number of content from an information provider that can be provided by each of the multiple question candidates; (2) the quality or expected charge of the content from an information provider that can be provided by each of the multiple question candidates; (3) the suitability of each of the multiple question candidates with a conversation context; and (4) the suitability of each of the multiple question candidates with user characteristics.

14. The computer device of claim 13, characterized in that at least one of the relevance of (3) and the relevance of (4) includes at least one of a first relevance calculated based on the degree of overlap between texts, a matching status of themes, etc., and a second relevance calculated by analyzing sentence structure, vocabulary, relationships between sentences, etc. using natural language processing technology.

15. by said at least one processor, When the question provided as the response is selected by the user, an instance of content of an information provider related to the question is dynamically generated and provided to the user. A computer device according to any one of claims 11 to 14.

Images