Service providing server that selects and provides a response sentence that corresponds to the user's query sentence among multiple response sentences, and the operating method thereof
The service providing server uses multiple LLMs to select and provide answer sentences that closely match user intent, addressing the challenge of response diversity and relevance in chatbot services.
Patent Information
- Application Number
- KR1020250063254
- Authority / Receiving Office
- KR · KR
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2025-05-15
- Publication Date
- 2026-11-11
- Estimated Expiration
- 2045-05-15
AI Technical Summary
Existing chatbot services using Large Language Models (LLMs) struggle to provide answer sentences that accurately match user intent, as they often generate multiple responses that lack appropriate selection and correlation criteria.
A service providing server that utilizes multiple pre-built LLMs to generate answer sentences, selects one with maximum similarity to the query, calculates an association index to check correlation, and optionally provides a second distinct answer sentence based on correlation strength, ensuring diverse and intent-matching responses.
Enhances the quality of chatbot responses by providing answer sentences that closely match user intent while offering diversity, thereby improving user satisfaction and response relevance.
Smart Images

Figure 112025054489641-PAT00020_ABST
Abstract
Description
Technology Field
[0001] The present invention relates to a service providing server that selects and provides an answer sentence corresponding to a user's query sentence from among a plurality of answer sentences, and a method of operation thereof. Background Technology
[0002] Recently, the use of generative artificial intelligence utilizing Large Language Models (LM) is increasing across various fields.
[0003] In particular, the use of LLM-based generative artificial intelligence is increasing in chatbot services, and these chatbot services are operated in a manner where, when a user inputs a specific query, an answer corresponding to that query is generated and provided to the user.
[0004] Although the quality of such response services is steadily improving as artificial intelligence models become more sophisticated, to further enhance the quality of response services using LLM, it is necessary to introduce a method in which, when a user inputs a specific query, the query is fed into multiple pre-built response generation models to generate response sentences, and then appropriate response sentences that match the user's intent are selected from among the generated sentences and provided to the user.
[0005] Specifically, when a query sentence is received from a user's electronic terminal and a command is received instructing to generate and provide an answer sentence corresponding to the query sentence, the query sentence is input into multiple LLM-based answer generation models to generate multiple answer sentences, then one answer sentence with the maximum similarity to the query sentence is selected from among the generated multiple answer sentences, another answer sentence is selected from among the multiple answer sentences based on correlation information between the multiple answer sentences, and then the selected answer sentences are provided to the user. The problem to be solved
[0006] The present invention aims to support the provision of appropriate answer sentences that match the user's intent to the user when the user inputs a query sentence by presenting a service providing server and a method of operation thereof that selects and provides an answer sentence corresponding to the user's query sentence from among a plurality of answer sentences. means of solving the problem
[0007] A service providing server according to an embodiment of the present invention, when a first query sentence is received from a user’s electronic terminal and a request command for providing an answer corresponding to the first query sentence is received, comprises: an acquisition unit that inputs the first query sentence as an input to each of a plurality of pre-established Large Language Models (LLMs)—each of which is a language model built based on different text data—and obtains a plurality of answer sentences produced through each of the plurality of LLMs; a first selection unit that selects one of the plurality of answer sentences that has the maximum similarity to the first query sentence as a first-type answer sentence; a verification unit that calculates an association index representing the degree of association between the plurality of answer sentences and checks whether the association index exceeds a preset threshold; and if, as a result of the verification by the verification unit, it is confirmed that the association index exceeds the threshold, among the plurality of answer sentences, an answer sentence that has the maximum difference from other answer sentences within the plurality of answer sentences is selected as a second-type answer sentence, and if it is confirmed that the association index does not exceed the threshold, among the plurality of answer sentences, It includes a second selection unit that selects an answer sentence having minimal differentiation from other answer sentences among the plurality of answer sentences as a second-type answer sentence, and a transmission unit that designates the first-type answer sentence and the second-type answer sentence as answer sentences corresponding to the first query sentence and transmits them to the electronic terminal, wherein if the first-type answer sentence and the second-type answer sentence are identical, only the first-type answer sentence is designated as the answer sentence corresponding to the first query sentence and transmits it to the electronic terminal.
[0008] In addition, a method of operation of a service providing server according to an embodiment of the present invention comprises the steps of: receiving a first query sentence from a user’s electronic terminal and receiving a request command for providing an answer corresponding to the first query sentence, applying the first query sentence as an input to each of a plurality of pre-established Large Language Models (LLMs)—each of which is a language model built based on different text data—to obtain a plurality of answer sentences produced through each of the plurality of LLMs; selecting one of the plurality of answer sentences that has the maximum similarity to the first query sentence as a first-type answer sentence; calculating an association index representing the degree of association between the plurality of answer sentences and checking whether the association index exceeds a preset threshold; and, as a result of the checking step, if it is confirmed that the association index exceeds the threshold, selecting the answer sentence among the plurality of answer sentences that has the maximum difference from other answer sentences within the plurality of answer sentences as a second-type answer sentence, and if it is confirmed that the association index does not exceed the threshold, among the plurality of answer sentences, the The method includes the step of selecting a response sentence that has minimal differentiation from other response sentences among multiple response sentences as a second-type response sentence, and the step of designating the first-type response sentence and the second-type response sentence as response sentences corresponding to the first query sentence and transmitting them to the electronic terminal, wherein if the first-type response sentence and the second-type response sentence are identical, only the first-type response sentence is designated as the response sentence corresponding to the first query sentence and transmitted to the electronic terminal. Effects of the invention
[0009] The present invention provides a service providing server and a method of operation thereof that selects and provides an answer sentence corresponding to a user's query sentence from among a plurality of answer sentences, thereby supporting the provision of appropriate answer sentences that match the user's intent to the user when the user inputs a query sentence. Brief explanation of the drawing
[0010] FIG. 1 is a diagram illustrating the structure of a service providing server according to an embodiment of the present invention. FIG. 2 is a flowchart illustrating the operation method of a service providing server according to an embodiment of the present invention. Specific details for implementing the invention
[0011] Embodiments according to the present invention will be described in detail below with reference to the accompanying drawings. This description is not intended to limit the present invention to specific embodiments and should be understood to include all modifications, equivalents, and substitutions that fall within the spirit and scope of the present invention. Similar reference numerals have been used for similar components in describing each drawing, and unless otherwise defined, all terms used in this specification, including technical or scientific terms, have the same meaning as generally understood by a person skilled in the art to which the present invention pertains.
[0012] In this document, when a part is described as "including" a component, it means that, unless specifically stated otherwise, it does not exclude other components but may include additional components. Furthermore, in various embodiments of the present invention, each component, functional block, or means may be composed of one or more sub-components, and the electrical, electronic, or mechanical functions performed by each component may be implemented by various known devices or mechanical elements, such as electronic circuits, integrated circuits, and ASICs (Application Specific Integrated Circuits), and may be implemented separately or two or more may be integrated into one.
[0013] Meanwhile, the blocks in the attached block diagram or the steps in the flowchart may be interpreted as computer program instructions that perform designated functions by being loaded into the processor or memory of data-processing equipment, such as general-purpose computers, specialized computers, portable notebook computers, and network computers. Since these computer program instructions may be stored in memory provided in a computer device or in memory readable by a computer, the functions described in the blocks in the block diagram or the steps in the flowchart may be produced as manufactured products containing means of instruction to perform them. Furthermore, each block or each step may represent a module, segment, or part of code containing one or more executable instructions for executing a specific logical function(s). Also, it should be noted that in some alternative embodiments, the functions mentioned in the blocks or steps may be executed in a different order than the prescribed order. For example, two blocks or steps shown in succession may be performed substantially simultaneously or in reverse order, and in some cases, some blocks or steps may be omitted.
[0014] FIG. 1 is a diagram illustrating the structure of a service providing server according to an embodiment of the present invention.
[0015] Referring to FIG. 1, a service providing server (110) according to an embodiment of the present invention includes an acquisition unit (111), a first selection unit (112), a confirmation unit (113), a second selection unit (114), and a transmission unit (115).
[0016] When the acquisition unit (111) receives a first query sentence from the user's electronic terminal (10) and receives a request command to provide an answer corresponding to the first query sentence, it applies the first query sentence as an input to each of the multiple pre-built LLMs (Large Language Models) and acquires multiple answer sentences produced through each of the multiple LLMs.
[0017] Here, each of the above multiple LLMs refers to a language model built based on different text data. For example, if the above multiple LLMs are configured as 'LLM 1, LLM 2, LLM 3', the acquisition unit (111) can acquire 'Answer Sentence 1' which is produced by inputting the first query sentence to 'LLM 1', acquire 'Answer Sentence 2' which is produced by inputting the first query sentence to 'LLM 2', and acquire 'Answer Sentence 3' which is produced by inputting the first query sentence to 'LLM 3'. In this way, the acquisition unit (111) can acquire multiple answer sentences produced through each of the above multiple LLMs as 'Answer Sentence 1, Answer Sentence 2, Answer Sentence 3'.
[0018] The first selection unit (112) selects one of the multiple answer sentences that has the maximum similarity to the first query sentence as the first type of answer sentence.
[0019] In this case, according to one embodiment of the present invention, the service providing server (110) may further include a word vector database (116).
[0020] The word vector database (116) stores word vectors corresponding to each of the multiple words.
[0021] Here, the word vector corresponding to each of the aforementioned multiple words is a pre-embedded vector based on the semantic similarity between words, and is configured such that a higher vector similarity is calculated as the semantic similarity between words increases. For example, if 'Word A' and 'Word B' have similar meanings, word vectors corresponding to 'Word A' and 'Word B' may be assigned so that a high vector similarity is calculated between the two words. Such word vectors may be assigned to each word according to the Word2Vec embedding technique.
[0022] In this case, according to one embodiment of the present invention, the first selection unit (112) may include a calculation unit (117) and a first selection processing unit (118).
[0023] The calculation unit (117) performs the process of calculating the similarity between each of the plurality of answer sentences sequentially and the first query sentence, and when the order is to perform the process of calculating the similarity between the first answer sentence and the first query sentence, which is one of the plurality of answer sentences, it refers to the word vector database (116) to identify the word vectors of the first words that appear at least once in the first answer sentence and the word vectors of the second words that appear at least once in the first query sentence, calculates all the vector similarities between each of the first words and each of the second words, and calculates the first overall average, which is the overall average of the calculated vector similarities, as the similarity corresponding to the first answer sentence.
[0024] In this regard, as in the example described above, when the plurality of answer sentences are referred to as 'answer sentence 1, answer sentence 2, answer sentence 3', the calculation unit (117) can sequentially perform the process of calculating the similarity with the first query sentence for each of the plurality of answer sentences, 'answer sentence 1, answer sentence 2, answer sentence 3'.
[0025] At this time, it is assumed that the process of calculating the similarity between 'answer sentence 1', which is one of the plurality of answer sentences, and the first query sentence is to be performed. Assuming the first words are 'word 1, word 2, word 3, word 4, word 5' and the second words are 'word A, word B, word C', the process of the calculation unit (117) calculating the similarity between 'answer sentence 1' and the first query sentence is as follows.
[0026] First, the output unit (117) can check the word vectors of ‘word 1, word 2, word 3, word 4, word 5’, which appear at least once in ‘answer sentence 1’, and the word vectors of ‘word A, word B, word C’, which appear at least once in the first query sentence, by referring to the word vector database (116).
[0027] Then, the output unit (117) calculates all vector similarities between each of the first words and each of the second words, and calculates the overall average of the calculated vector similarities as the similarity corresponding to the first answer sentence.
[0028] First, the output unit (117) can calculate all vector similarities between each of the first words, 'word 1, word 2, word 3, word 4, word 5', and each of the second words, 'word A, word B, word C'.
[0029] Specifically, the calculation unit (117) calculates the vector similarity between 'Word 1' and 'Word A', the vector similarity between 'Word 1' and 'Word B', the vector similarity between 'Word 1' and 'Word C', the vector similarity between 'Word 2' and 'Word A', the vector similarity between 'Word 2' and 'Word B', the vector similarity between 'Word 2' and 'Word C', the vector similarity between 'Word 3' and 'Word A', the vector similarity between 'Word 3' and 'Word B', the vector similarity between 'Word 3' and 'Word C', the vector similarity between 'Word 4' and 'Word A', the vector similarity between 'Word 4' and 'Word C', the vector similarity between 'Word 5' and 'Word A', and the vector similarity between 'Word 5' and 'Word B'. Similarity, vector similarity between 'word 5' and 'word C' can be calculated.
[0030] After that, the output unit (117) can calculate the overall average of the calculated vector similarities as the similarity corresponding to ‘answer sentence 1’.
[0031] At this time, when the calculated vector similarities between each of the first words and each of the second words are called 'V1, V2, V3, V4, V5, V6, V7, V8, V9, V10, V11, V12, V13, V14, V15', the calculation unit (117) can calculate the overall average of 'V1, V2, V3, V4, V5, V6, V7, V8, V9, V10, V11, V12, V13, V14, V15' as the similarity corresponding to 'answer sentence 1'.
[0032] In this way, the output unit (117) can calculate the similarity for each of the plurality of answer sentences, 'answer sentence 1, answer sentence 2, answer sentence 3'.
[0033] The first selection processing unit (118) selects one of the multiple answer sentences with the maximum similarity as the first type of answer sentence.
[0034] In this regard, if the above multiple answer sentences are referred to as 'answer sentence 1, answer sentence 2, answer sentence 3' and the answer sentence with the maximum similarity among 'answer sentence 1, answer sentence 2, answer sentence 3' is referred to as 'answer sentence 2', the first selection processing unit (118) can select 'answer sentence 2', which has the maximum similarity among the above multiple answer sentences, as the first type of answer sentence.
[0035] The verification unit (113) calculates an association index representing the degree of association between the plurality of answer sentences and checks whether the association index exceeds a preset threshold.
[0036] At this time, according to one embodiment of the present invention, the verification unit (113) extracts all words that appear at least once from the plurality of answer sentences, calculates the existence ratio of words that appear commonly in the plurality of answer sentences among the extracted words, checks the word vector of each of the extracted words by referring to the word vector database (116), calculates the concentration of each of the extracted words according to the operation of mathematical formula 1 below, sorts the calculated concentrations in ascending order to calculate the quartiles, calculates the second overall average which is the overall average of the remaining concentrations excluding the concentrations corresponding to the first quartile or lower, and then calculates the sum between the existence ratio and the second overall average as an association index representing the association between the plurality of answer sentences.
[0038]
[0040] Here is the i-th word among the words extracted from the above multiple answer sentences ( With the concentration of ), is the i-th word among the above extracted words ( ) and the j-th word( It refers to the cosine similarity between ). This concentration can be used as a measure representing how many other words with high vector similarity are clustered around a specific word.
[0041] Let us assume that, as in the example above, the above multiple answer sentences are called 'Answer Sentence 1, Answer Sentence 2, Answer Sentence 3', and that as a result of extracting all words that appear at least once in 'Answer Sentence 1, Answer Sentence 2, Answer Sentence 3', a total of 15 words are extracted, such as 'Word 1, Word 2, Word 3, ..., Word 15'.
[0042] Then, the verification unit (113) can calculate the existence ratio of words that appear commonly in 'answer sentence 1, answer sentence 2, answer sentence 3' among 15 words. In this regard, if three words, 'word 1, word 2, word 5' among 15 words, appear commonly in 'answer sentence 1, answer sentence 2, answer sentence 3', the verification unit (113) can calculate an existence ratio of '0.2'.
[0043] Then, the verification unit (113) can check the word vector of each of the 15 words by referring to the word vector database (116), and then calculate the concentration of each of the 15 words based on this word vector.
[0044] Specifically, if we explain the process of the verification unit (113) calculating the concentration of 'word 1', which is one of 15 words, as an example, the verification unit (113) can calculate the concentration of 'word 1' according to the calculation process of the following mathematical formula 2 based on the formula of the above mathematical formula 1.
[0046]
[0048] In this way, the verification unit (113) can calculate the concentration of each of the 15 words.
[0049] After that, the verification unit (113) can calculate the quartiles by sorting the concentration of each of the 15 words in ascending order.
[0050] Here, quartiles refer to data sorted in ascending order divided into four equal parts, where the first quartile is the variable value corresponding to the 1 / 4 place and the third quartile is the variable value corresponding to the 3 / 4 place.
[0051] At this time, if the concentration of each of the 15 words is 'C1, C2, C3, C4, C5, C6, C7, C8, C9, C10, C11, C12, C13, C14, C15' and the concentration corresponding to the first quartile or lower is 'C4, C7', the verification unit (113) can calculate the overall average of the remaining concentrations, 'C1, C2, C3, C5, C6, C8, C9, C10, C11, C12, C13, C14, C15', excluding the concentration corresponding to the first quartile or lower among the concentrations.
[0052] At this time, if the above-mentioned overall average is calculated as 'M', the verification unit (113) can calculate the value '0.2+M', which is the sum of the above-mentioned existence ratio '0.2' and the above-mentioned overall average 'M', as an association index representing the association between the above-mentioned multiple answer sentences, 'answer sentence 1, answer sentence 2, answer sentence 3'.
[0053] Once the calculation of the above correlation index is completed, the verification unit (113) can check whether the above correlation index exceeds a preset threshold.
[0054] The second selection unit (114) selects the answer sentence with the maximum difference from other answer sentences among the plurality of answer sentences as the second type of answer sentence if it is confirmed that the correlation index exceeds the threshold value as a result of confirmation by the confirmation unit (113), and selects the answer sentence with the minimum difference from other answer sentences among the plurality of answer sentences as the second type of answer sentence if it is confirmed that the correlation index does not exceed the threshold value.
[0055] In this case, according to one embodiment of the present invention, the second selection unit (114) may include a first word set generation unit (119), a second word set generation unit (120), and a second selection processing unit (121).
[0056] The first word set generation unit (119) extracts all words that appear at least once from the plurality of answer sentences and generates a first word set having the extracted words as elements.
[0057] Let us assume that, as in the example above, the above multiple answer sentences are called 'Answer Sentence 1, Answer Sentence 2, Answer Sentence 3', and that as a result of extracting all words that appear at least once in 'Answer Sentence 1, Answer Sentence 2, Answer Sentence 3', a total of 15 words are extracted, such as 'Word 1, Word 2, Word 3, ..., Word 15'.
[0058] Then, the first word set generating unit (119) can generate a first word set having the extracted words 'word 1, word 2, word 3, ..., word 15' as elements as '{word 1, word 2, word 3, ..., word 15}'.
[0059] The second word set generation unit (120) generates a word set corresponding to each of the plurality of answer sentences by extracting all words that appear at least once in each of the plurality of answer sentences and forming a word set having the extracted words as elements.
[0060] Let us assume that, as in the example above, the above multiple answer sentences are referred to as 'Answer Sentence 1, Answer Sentence 2, Answer Sentence 3', and that for each of the above multiple answer sentences, all words appearing at least once in each answer sentence are extracted, and that words for each of 'Answer Sentence 1, Answer Sentence 2, Answer Sentence 3' are extracted as shown in Table 1 below.
[0062] Multiple answer sentences extracted words Answer sentence 1 Word 1, Word 2, Word 3, Word 4, Word 5 Answer sentence 2 Word 1, Word 2, Word 5, Word 6, Word 7, Word 8 Answer sentence 3 Word 1, Word 2, Word 5, Word 9, Word 10, Word 11, Word 12, Word 13, Word 14, Word 15
[0064] Then, the second word set generating unit (120) can generate a word set having elements 'word 1, word 2, word 3, word 4, word 5', which are words extracted for 'answer sentence 1', as '{word 1, word 2, word 3, word 4, word 5}', and can generate a word set having elements 'word 1, word 2, word 5, word 6, word 7, word 8', which are words extracted for 'answer sentence 2', as '{word 1, word 2, word 5, word 6, word 7, word 8}', and can generate a word set having elements 'word 1, word 2, word 5, word 9, word 10, word 11, word 12, word 13, word 14, word 15', which are words extracted for 'answer sentence 3', as '{word 1, word 2, word It can be generated like {5, word 9, word 10, word 11, word 12, word 13, word 14, word 15}.
[0065] In this way, the second word set generating unit (120) can generate word sets corresponding to each of ‘answer sentence 1, answer sentence 2, answer sentence 3’ as shown in Table 2 below.
[0067] Multiple answer sentences word set Answer sentence 1 {Word 1, Word 2, Word 3, Word 4, Word 5} Answer sentence 2 {Word 1, Word 2, Word 5, Word 6, Word 7, Word 8} Answer sentence 3 {Word 1, Word 2, Word 5, Word 9, Word 10, Word 11, Word 12, Word 13, Word 14, Word 15}
[0069] The second selection processing unit (121) selects, among the plurality of answer sentences, one answer sentence having the word set with the maximum Jaccard distance from the first word set as the second type of answer sentence when it is confirmed that the correlation index exceeds the threshold value as a result of confirmation by the confirmation unit (113), and, among the plurality of answer sentences, one answer sentence having the word set with the minimum Jaccard distance from the first word set as the second type of answer sentence when it is confirmed that the correlation index does not exceed the threshold value.
[0070] Here, Jaccard distance is a measure of dissimilarity between two sets and refers to the value obtained by subtracting the Jaccard index, which is the value obtained by dividing the number of elements in the intersection of the two sets by the number of elements in the union, from 1.
[0071] In this regard, let us assume that the above multiple answer sentences are referred to as 'answer sentence 1, answer sentence 2, answer sentence 3' as in the example above, that the above first word set is generated as '{word 1, word 2, word 3, ..., word 15}', and that the word set corresponding to each of the above multiple answer sentences is generated as shown in Table 2 above.
[0072] Then, if the second selection processing unit (121) confirms that the correlation index exceeds the reference value as a result of the confirmation by the confirmation unit (113), it may select one of the multiple answer sentences, 'answer sentence 1, answer sentence 2, answer sentence 3', which has the maximum Jaccard distance from the first word set '{word 1, word 2, word 3, ..., word 15}', as the second type of answer sentence.
[0073] For example, if an answer sentence having a word set with the maximum Jaccard distance from the first word set is called 'answer sentence 1', the second selection processing unit (121) can select 'answer sentence 1' as the second type of answer sentence.
[0074] Meanwhile, unlike the example described above, if the verification result of the verification unit (113) confirms that the correlation index does not exceed the reference value, the second selection processing unit (121) may select one of the multiple answer sentences, 'answer sentence 1, answer sentence 2, answer sentence 3', which has a word set with the minimum Jaccard distance from the first word set '{word 1, word 2, word 3, ..., word 15}', as the second type of answer sentence.
[0075] For example, if an answer sentence having a word set with the minimum Jaccard distance from the first word set is called 'answer sentence 3', the second selection processing unit (121) can select 'answer sentence 3' as the second type of answer sentence.
[0076] Thus, when the first type of answer sentence is selected by the first selection unit (112) and the second type of answer sentence is selected by the second selection unit (114), the transmission unit (115) designates the first type of answer sentence and the second type of answer sentence as answer sentences corresponding to the first question sentence and transmits them to the electronic terminal (10); however, if the first type of answer sentence and the second type of answer sentence are identical to each other, only the first type of answer sentence is designated as the answer sentence corresponding to the first question sentence and transmitted to the electronic terminal (10).
[0077] In this regard, if 'answer sentence 2' is selected as the first type of answer sentence and 'answer sentence 1' is selected as the second type of answer sentence, the transmission unit (115) can designate 'answer sentence 2' and 'answer sentence 1' as answer sentences corresponding to the first question sentence and transmit them to the electronic terminal (10).
[0078] Meanwhile, unlike the example described above, if the first type of answer sentence and the second type of answer sentence are selected as the same thing as 'answer sentence 2', the transmission unit (115) can designate only the first type of answer sentence, 'answer sentence 2', as the answer sentence corresponding to the first question sentence and transmit it to the electronic terminal (10).
[0079] Through this, the user of the electronic terminal (10) can obtain appropriate answer sentences corresponding to the first query sentence entered by the user.
[0080] Ultimately, the service providing server (110) of the present invention generates multiple answer sentences according to the first query sentence entered by the user using multiple LLMs, and then provides the user with one of the multiple answer sentences that is most similar to the first query sentence, and at the same time checks the correlation between the multiple answer sentences and if the multiple answer sentences are highly correlated with each other, it additionally designates the answer sentence that is most distinctive among the multiple answer sentences as a second answer sentence and provides it to the user, unlike the first answer sentence provided to the user, thereby enabling the user to obtain the answer sentence most similar to the first query sentence and at the same time to obtain an answer sentence with great diversity. At this time, if the correlation between the multiple answer sentences is somewhat weak during the process of selecting the second type of answer sentence, and if the answer sentence with the greatest differentiation among the multiple answer sentences is designated as the second type of answer sentence, there is a possibility that the second type of answer sentence may contain content that is too far removed from the first query sentence. Therefore, when selecting the second type of answer sentence, the service providing server (110) of the present invention performs the process of selecting the answer sentence with the lowest differentiation from other answer sentences among the multiple answer sentences as the second type of answer sentence when the correlation between the multiple answer sentences is weak.
[0081] FIG. 2 is a flowchart illustrating the operation method of a service providing server according to an embodiment of the present invention.
[0082] In step (S211), when a first query sentence is received from a user's electronic terminal and a request command to provide an answer corresponding to the first query sentence is received, the first query sentence is input to each of the multiple LLMs that are pre-built (each of the multiple LLMs is a language model built based on different text data) and multiple answer sentences produced through each of the multiple LLMs are obtained.
[0083] In step (S212), one of the multiple answer sentences that has the maximum similarity to the first query sentence is selected as the first type of answer sentence.
[0084] In step (S213), an association index representing the degree of association between the plurality of answer sentences is calculated, and it is checked whether the association index exceeds a preset threshold.
[0085] In step (S214), if the correlation index is confirmed to exceed the threshold value as a result of the verification in step (S213), the answer sentence among the plurality of answer sentences that has the maximum differentiation from other answer sentences is selected as the second-type answer sentence, and if the correlation index is confirmed not to exceed the threshold value, the answer sentence among the plurality of answer sentences that has the minimum differentiation from other answer sentences is selected as the second-type answer sentence.
[0086] In step (S215), the first type of answer sentence and the second type of answer sentence are designated as answer sentences corresponding to the first query sentence and transmitted to the electronic terminal, provided that if the first type of answer sentence and the second type of answer sentence are identical, only the first type of answer sentence is designated as the answer sentence corresponding to the first query sentence and transmitted to the electronic terminal.
[0087] At this time, according to one embodiment of the present invention, the method of operation of the service providing server may further include the step of maintaining a word vector database in which word vectors corresponding to each of a plurality of words are stored (the word vectors corresponding to each of the plurality of words are pre-embedded vectors based on semantic similarity between words, and are specified so that a higher vector similarity is calculated as the semantic similarity between words is higher).
[0088] At this time, in step (S212), the process of calculating the similarity with the first query sentence is performed sequentially for each of the plurality of answer sentences, and when the order of performing the process of calculating the similarity between the first answer sentence, which is one of the plurality of answer sentences, and the first query sentence is to be performed, the word vector of first words that appear at least once in the first answer sentence and the word vector of second words that appear at least once in the first query sentence are identified by referring to the word vector database, and then all vector similarities between each of the first words and each of the second words are calculated, and the first overall average, which is the overall average of the calculated vector similarities, is calculated as the similarity corresponding to the first answer sentence, and the step of selecting one of the answer sentences with the maximum similarity among the plurality of answer sentences as the first type of answer sentence.
[0089] At this time, according to one embodiment of the present invention, in step (S213), all words that appear at least once in the plurality of answer sentences are extracted, and among the extracted words, the existence ratio of words that appear commonly in the plurality of answer sentences is calculated, then the word vector of each of the extracted words is checked by referring to the word vector database, and the concentration of each of the extracted words is calculated according to the operation of mathematical formula 1, and then the calculated concentrations are sorted in ascending order to calculate the quartiles, and then the second overall average, which is the overall average of the remaining concentrations excluding the concentrations corresponding to the first quartile or lower, is calculated, and then the sum between the existence ratio and the second overall average can be calculated as an association index representing the association between the plurality of answer sentences.
[0090] At this time, according to one embodiment of the present invention, step (S214) may include: a step of generating a first word set having the extracted words as elements by extracting all words that appear at least once from the plurality of answer sentences; a step of generating a word set corresponding to each of the plurality of answer sentences by configuring a word set having the extracted words as elements by extracting all words that appear at least once in each of the plurality of answer sentences for each of the plurality of answer sentences; and a step of selecting, as a result of verification in step (S213), one of the answer sentences having a word set with the maximum Jaccard distance from the first word set as the second type of answer sentence if it is confirmed that the correlation index exceeds the reference value, and one of the answer sentences having a word set with the minimum Jaccard distance from the first word set as the second type of answer sentence if it is confirmed that the correlation index does not exceed the reference value.
[0091] For the above, a method of operation of a service providing server according to an embodiment of the present invention has been described with reference to FIG. 2. Here, since the method of operation of a service providing server according to an embodiment of the present invention may correspond to the configuration of the operation of a service providing server (110) described using FIG. 1, a more detailed description thereof will be omitted.
[0092] A method of operation of a service providing server according to one embodiment of the present invention may be implemented as a computer program stored in a storage medium for execution through combination with a computer.
[0093] In addition, the method of operation of a service providing server according to an embodiment of the present invention may be implemented in the form of computer program instructions for execution through combination with a computer and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either individually or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the present invention, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROMs and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.
[0094] As described above, the present invention has been explained by specific details such as specific components, limited embodiments, and drawings; however, this is provided merely to aid in a more comprehensive understanding of the invention, and the invention is not limited to the above embodiments. A person skilled in the art can make various modifications and variations from this description.
[0095] Therefore, the scope of the present invention is not limited to the described embodiments, and all things equivalent to or having equivalent variations to the claims set forth below, as well as the claims set forth below, shall be considered to fall within the scope of the concept of the present invention. Explanation of the symbols
[0096] 110: Service Provider Server 111: Acquisition Section 112: First Selection Section 113: Confirmation section 114: Second selection section 115: Transmission section 116: Word vector database 117: Output unit 118: First selection processing unit 119: 1st word set generator 120: 2nd word set generator 121: Second selection processing unit 10: Electronic terminal
Claims
Claim 1 When a first query sentence is received from a user's electronic terminal and a request command for providing an answer corresponding to the first query sentence is received, the first query sentence is input to each of a plurality of pre-established Large Language Models (LLMs)—each of which is a language model built based on different text data—and a plurality of answer sentences produced through each of the plurality of LLMs are obtained; a first selection unit that selects one of the plurality of answer sentences that has the maximum similarity to the first query sentence as a first-type answer sentence; a verification unit that calculates an association index representing the degree of association between the plurality of answer sentences and checks whether the association index exceeds a preset threshold; if, as a result of the verification by the verification unit, it is confirmed that the association index exceeds the threshold, the answer sentence among the plurality of answer sentences that has the maximum distinctiveness from other answer sentences within the plurality of answer sentences is selected as a second-type answer sentence, and if it is confirmed that the association index does not exceed the threshold, among the plurality of answer sentences, the answer sentence within the plurality of answer sentences and the other answer sentences A service providing server comprising: a second selection unit for selecting an answer sentence with minimal differentiation as a second-type answer sentence; and a transmission unit for designating the first-type answer sentence and the second-type answer sentence as answer sentences corresponding to the first query sentence and transmitting them to the electronic terminal, wherein if the first-type answer sentence and the second-type answer sentence are identical to each other, only the first-type answer sentence is designated as the answer sentence corresponding to the first query sentence and transmitted to the electronic terminal. Claim 2 In claim 1, the invention further includes a word vector database in which word vectors corresponding to each of the plurality of words are stored—the word vectors corresponding to each of the plurality of words are pre-embedded vectors based on semantic similarity between words, and are specified such that a higher vector similarity is calculated as the semantic similarity between words increases—and the first selection unit performs a process of sequentially calculating the similarity with the first query sentence for each of the plurality of answer sentences, wherein when it is the order in which the process of calculating the similarity between the first answer sentence, which is one of the plurality of answer sentences, and the first query sentence is to be performed, the calculation unit refers to the word vector database to identify the word vectors of the first words, which are words appearing one or more times in the first answer sentence, and the word vectors of the second words, which are words appearing one or more times in the first query sentence, calculates all vector similarities between each of the first words and each of the second words, and calculates the first overall average, which is the overall average of the calculated vector similarities, as the similarity corresponding to the first answer sentence; A service providing server comprising a first selection processing unit that selects one of the plurality of answer sentences with the maximum similarity as the first type of answer sentence. Claim 3 In paragraph 2, the verification unit extracts all words that appear at least once from the plurality of answer sentences, calculates the existence ratio of words that appear commonly in the plurality of answer sentences among the extracted words, verifies the word vector of each of the extracted words by referring to the word vector database, calculates the concentration of each of the extracted words according to the operation of Mathematical Formula 1 below, sorts the calculated concentrations in ascending order to calculate quartiles, calculates a second overall average which is the overall average of the remaining concentrations excluding concentrations corresponding to the first quartile or lower, and calculates the sum between the existence ratio and the second overall average as an association index representing the association between the plurality of answer sentences. [Mathical Formula 1] Here is the i-th word among the words extracted from the above multiple answer sentences ( With the concentration of ), is the i-th word among the above extracted words ( ) and the j-th word( It refers to the cosine similarity between ). Claim 4 In paragraph 3, the service providing server comprises: a first word set generating unit that generates a first word set having the extracted words as elements by extracting all words that appear at least once in the plurality of answer sentences; a second word set generating unit that generates a word set corresponding to each of the plurality of answer sentences by configuring a word set having the extracted words as elements by extracting all words that appear at least once in each of the plurality of answer sentences for each of the plurality of answer sentences; and a second selection processing unit that, if the correlation index is confirmed to exceed the reference value as a result of the verification unit, selects one of the plurality of answer sentences having a word set with the maximum Jaccard distance from the first word set as the second type of answer sentence, and if the correlation index is confirmed not to exceed the reference value, selects one of the plurality of answer sentences having a word set with the minimum Jaccard distance from the first word set as the second type of answer sentence. Claim 5 When a first query sentence is received from a user's electronic terminal and a request command for providing an answer corresponding to the first query sentence is received, the first query sentence is input to each of a plurality of pre-established Large Language Models (LLMs)—each of which is a language model built based on different text data—to obtain a plurality of answer sentences produced through each of the plurality of LLMs; a step of selecting one answer sentence among the plurality of answer sentences that has the maximum similarity to the first query sentence as a first-type answer sentence; a step of calculating an association index representing the degree of association between the plurality of answer sentences and checking whether the association index exceeds a preset threshold; if, as a result of the checking step, it is confirmed that the association index exceeds the threshold, the answer sentence among the plurality of answer sentences that has the maximum distinctiveness from other answer sentences within the plurality of answer sentences is selected as a second-type answer sentence, and if it is confirmed that the association index does not exceed the threshold, the answer among the plurality of answer sentences that has the minimum distinctiveness from other answer sentences within the plurality of answer sentences A method of operation of a service providing server comprising: a step of selecting a sentence as a second-type answer sentence; and a step of designating the first-type answer sentence and the second-type answer sentence as answer sentences corresponding to the first-type question sentence and transmitting them to the electronic terminal, wherein if the first-type answer sentence and the second-type answer sentence are identical, only the first-type answer sentence is designated as the answer sentence corresponding to the first-type question sentence and transmitted to the electronic terminal. Claim 6 In claim 5, the method further includes the step of maintaining a word vector database in which word vectors corresponding to each of the plurality of words are stored—the word vectors corresponding to each of the plurality of words are pre-embedded vectors based on semantic similarity between words, and are specified such that a higher vector similarity is calculated as the semantic similarity between words increases—and the step of selecting the first type of answer sentence performs a process of sequentially calculating the similarity with the first query sentence for each of the plurality of answer sentences, wherein when it is the order in which the process of calculating the similarity between the first answer sentence, which is one of the plurality of answer sentences, and the first query sentence is to be performed, the method refers to the word vector database to identify the word vectors of the first words, which are words appearing one or more times in the first answer sentence, and the word vectors of the second words, which are words appearing one or more times in the first query sentence, and then calculates all vector similarities between each of the first words and each of the second words, and calculates the first overall average, which is the overall average of the calculated vector similarities, as the similarity corresponding to the first answer sentence; A method of operation of a service providing server comprising the step of selecting one answer sentence with the maximum similarity among the plurality of answer sentences as the first type of answer sentence. Claim 7 In claim 6, the verification step comprises extracting all words that appear at least once from the plurality of answer sentences, calculating the existence ratio of words that appear commonly in the plurality of answer sentences among the extracted words, verifying the word vector of each of the extracted words by referring to the word vector database, calculating the concentration of each of the extracted words according to the operation of Mathematical Formula 1 below, sorting the calculated concentrations in ascending order to calculate quartiles, calculating a second overall average which is the overall average of the remaining concentrations excluding concentrations corresponding to the first quartile or lower, and calculating the sum between the existence ratio and the second overall average as an association index representing the association between the plurality of answer sentences. [Mathical Formula 1] Here is the i-th word among the words extracted from the above multiple answer sentences ( With the concentration of ), is the i-th word among the above extracted words ( ) and the j-th word( It refers to the cosine similarity between ). Claim 8 In claim 7, the step of selecting the second type of answer sentence comprises: a step of generating a first word set having the extracted words as elements by extracting all words that appear at least once from the plurality of answer sentences; a step of generating a word set corresponding to each of the plurality of answer sentences by configuring a word set having the extracted words as elements by extracting all words that appear at least once in each of the plurality of answer sentences for each of the plurality of answer sentences; and a step of selecting, as a result of the verification step, one of the answer sentences having the word set with the maximum Jaccard distance from the first word set as the second type of answer sentence when it is confirmed that the correlation index exceeds the threshold value, and one of the answer sentences having the word set with the minimum Jaccard distance from the first word set as the second type of answer sentence when it is confirmed that the correlation index does not exceed the threshold value. Claim 9 A computer-readable recording medium having a computer program for executing the method of any one of paragraphs 5 through 8 through combination with a computer. Claim 10 A computer program stored on a storage medium for executing the method of any one of paragraphs 5 through 8 through combination with a computer.
Citation Information
Patent Citations
Parallel scoring of an ensemble model
US10902005B2
Dynamic selection from among multiple candidate generative models with differing computational efficiencies
US20240311405A1
Hierarchical language model system for optimized query response generation
WO2025093339A1
