Service providing apparatus that generates and provides answers based on references corresponding to a user's inquiry and the operating method thereof

The service providing device and method improve answer accuracy in chatbot services by using a document database, keyword extraction, and Jaccard Index-based subject field selection to generate precise responses through LLM-based answer generation.

KR103045578B1Active Publication Date: 2026-11-11TJHE HANCOMM INC
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
KR1020250063259
Authority / Receiving Office
KR · KR
Patent Type
Patents
Current Assignee / Owner
Filing Date
2025-05-15
Publication Date
2026-11-11
Estimated Expiration
2045-05-15

AI Technical Summary

Technical Problem

Existing chatbot services using Large Language Models (LLM) struggle to provide accurate answers to user queries due to the lack of a method to select relevant references and generate answers based on those references effectively.

Method used

A service providing device and method that includes a document database, main keyword extraction, keyword set generation, union calculation, Jaccard Index-based subject field selection, and an LLM-based answer generation to provide accurate answers by selecting relevant documents and generating prompts for answer sentences.

Benefits of technology

Enhances the accuracy of answers provided to user queries by identifying relevant subject fields and documents, ensuring more precise and relevant responses.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 112025054491330-PAT00023_ABST
    Figure 112025054491330-PAT00023_ABST
Patent Text Reader

Abstract

The present invention provides a service providing device and a method of operation thereof that generate and provide answers based on reference literature corresponding to a user's query, thereby supporting the provision of more accurate answers to the user.
Need to check novelty before this filing date? Find Prior Art

Description

Technology Field

[0001] The present invention relates to a service providing device that generates and provides an answer based on reference literature corresponding to a user's query, and a method of operation thereof. Background Technology

[0002] Recently, the use of generative artificial intelligence utilizing Large Language Models (LM) is increasing across various fields.

[0003] In particular, the use of LLM-based generative artificial intelligence is increasing in chatbot services, and these chatbot services are operated in a manner where, when a user inputs a specific query, an answer corresponding to that query is generated and provided to the user.

[0004] Although the quality of such answer services is steadily improving as artificial intelligence models become more sophisticated, to further enhance the quality of answer services using LLM, it is necessary to introduce a method of inputting a prompt into an answer generation model that instructs the model to find appropriate references corresponding to a given query when a user inputs a query, and to generate an answer based on those references.

[0005] Specifically, when a query is received from a user's electronic terminal and a command is received instructing the generation and provision of an answer sentence corresponding to the query, one may consider introducing a method in which, among the references corresponding to each of multiple subject fields, one reference belonging to the subject field most relevant to the user's query is selected, a prompt is generated instructing the generation of an answer sentence corresponding to the user's query based on the information contained within that reference, and this prompt is input into an LLM-based answer generation model. The problem to be solved

[0006] The present invention aims to support the provision of more accurate answers to users by presenting a service providing device and a method of operation thereof that generate and provide answers based on reference literature corresponding to a user's query. means of solving the problem

[0007] A service providing device according to an embodiment of the present invention comprises: a document database storing information regarding a plurality of pre-specified documents to be used as reference literature and a subject field corresponding to each of the plurality of documents—wherein the subject field corresponding to each of the plurality of documents refers to any one subject field pre-specified among t types of subject fields (t is a natural number) corresponding to each document; a main keyword generating unit that, when a first query sentence is received from a user’s electronic terminal and a request command for providing an answer corresponding to the first query sentence is received, opens each of the plurality of documents and extracts main keywords from the opened documents to generate main keywords corresponding to each of the plurality of documents; a keyword set generating unit that, for each of the plurality of documents, constructs a keyword set having main keywords corresponding to each document as elements; a union generating unit that, for each of the t types of subject fields, calculates the union of the keyword sets of documents corresponding to each subject field to generate a union corresponding to each of the t types of subject fields; and the first A word set generation unit that identifies first words appearing at least once in a query sentence and generates a word set having said first words as elements; a subject field selection unit that calculates a Jaccard Index between the word set and the union corresponding to each of said t types of subject fields, and selects a first subject field among said t types of subject fields that has a union with the maximum Jaccard Index with said word set; and checks whether the Jaccard Index calculated for said first subject field exceeds a preset threshold, and if it is confirmed to exceed, determines that said first query sentence is a query sentence usable for providing an answer, andThe system includes an event generating unit that generates an answer providing event to provide an answer corresponding to the first query sentence; a document selection unit that, when the answer providing event is generated, selects one document among the documents corresponding to the first subject field that has major keywords matching the first query sentence as a reference document for generating an answer; an answer acquisition unit that generates a prompt instructing to generate an answer sentence corresponding to the first query sentence by referring to information contained within the document selected as the reference document, and then obtains an answer sentence corresponding to the first query sentence by inputting the prompt as an input to a pre-established LLM (Large Language Model)-based answer generation model; and an answer providing unit that transmits the answer sentence corresponding to the first query sentence to the electronic terminal.

[0008] In addition, a method of operation of a service providing device according to an embodiment of the present invention comprises: maintaining a document database in which information regarding a plurality of pre-specified documents to be used as reference literature and a subject field corresponding to each of the plurality of documents is stored—wherein the subject field corresponding to each of the plurality of documents means one of the pre-specified subject fields corresponding to each document among t types of subject fields (t is a natural number); when a first query sentence is received from a user’s electronic terminal and a request command for providing an answer corresponding to the first query sentence is received, the method comprises: opening each of the plurality of documents and extracting major keywords from the opened documents to generate major keywords corresponding to each of the plurality of documents; for each of the plurality of documents, constructing a keyword set having major keywords corresponding to each document as elements to generate a keyword set corresponding to each of the plurality of documents; for each of the t types of subject fields, calculating the union of keyword sets of documents corresponding to each subject field to generate a union corresponding to each of the t types of subject fields; and on the first query sentence, at least A step of identifying first words that appear at least once and generating a word set having said first words as elements; a step of calculating a Jaccard Index between the word set and the union corresponding to each of said t types of subject fields, and selecting a first subject field among said t types of subject fields that has a union with the maximum Jaccard Index with said word set; a step of checking whether the Jaccard Index calculated for said first subject field exceeds a preset threshold, and if it is confirmed to exceed, determining that said first query sentence is a query sentence usable for providing an answer, and then generating an answer providing event to provide an answer corresponding to said first query sentence.When the above-mentioned answer provision event occurs, the method includes the steps of: selecting one document among the documents corresponding to the above-mentioned first subject area that has major keywords matching the above-mentioned first query sentence as a reference document for generating an answer; generating a prompt that instructs to generate an answer sentence corresponding to the above-mentioned first query sentence by referring to information contained within the document selected as the reference document; obtaining an answer sentence corresponding to the above-mentioned first query sentence by inputting the prompt as input to a pre-established LLM (Large Language Model)-based answer generation model; and transmitting the answer sentence corresponding to the above-mentioned first query sentence to the electronic terminal. Effects of the invention

[0009] The present invention provides a service providing device and a method of operation thereof that generate and provide answers based on reference literature corresponding to a user's query, thereby supporting the provision of more accurate answers to the user. Brief explanation of the drawing

[0010] FIG. 1 is a drawing illustrating the structure of a service providing device according to an embodiment of the present invention. FIG. 2 is a flowchart illustrating the operation method of a service providing device according to an embodiment of the present invention. Specific details for implementing the invention

[0011] Embodiments according to the present invention will be described in detail below with reference to the accompanying drawings. This description is not intended to limit the present invention to specific embodiments and should be understood to include all modifications, equivalents, and substitutions that fall within the spirit and scope of the present invention. Similar reference numerals have been used for similar components in describing each drawing, and unless otherwise defined, all terms used in this specification, including technical or scientific terms, have the same meaning as generally understood by a person skilled in the art to which the present invention pertains.

[0012] In this document, when a part is described as "including" a component, it means that, unless specifically stated otherwise, it does not exclude other components but may include additional components. Furthermore, in various embodiments of the present invention, each component, functional block, or means may be composed of one or more sub-components, and the electrical, electronic, or mechanical functions performed by each component may be implemented by various known devices or mechanical elements, such as electronic circuits, integrated circuits, and ASICs (Application Specific Integrated Circuits), and may be implemented separately or two or more may be integrated into one.

[0013] Meanwhile, the blocks in the attached block diagram or the steps in the flowchart may be interpreted as computer program instructions that perform designated functions by being loaded into the processor or memory of data-processing equipment, such as general-purpose computers, specialized computers, portable notebook computers, and network computers. Since these computer program instructions may be stored in memory provided in a computer device or in memory readable by a computer, the functions described in the blocks in the block diagram or the steps in the flowchart may be produced as manufactured products containing means of instruction to perform them. Furthermore, each block or each step may represent a module, segment, or part of code containing one or more executable instructions for executing a specific logical function(s). Also, it should be noted that in some alternative embodiments, the functions mentioned in the blocks or steps may be executed in a different order than the prescribed order. For example, two blocks or steps shown in succession may be performed substantially simultaneously or in reverse order, and in some cases, some blocks or steps may be omitted.

[0014] FIG. 1 is a drawing illustrating the structure of a service providing device according to an embodiment of the present invention.

[0015] Referring to FIG. 1, a service providing device (110) according to an embodiment of the present invention includes a document database (111), a main keyword generating unit (112), a keyword set generating unit (113), a union generating unit (114), a word set generating unit (115), a subject field selection unit (116), an event generating unit (117), a document selection unit (118), an answer acquisition unit (119), and an answer providing unit (120).

[0016] The document database (111) stores information on a number of pre-specified documents to be used as references and a subject area corresponding to each of the said documents. Here, the subject area corresponding to each of the said documents refers to one of the pre-specified subject areas corresponding to each document among the t types of subject areas (t is a natural number).

[0017] For example, the above t types of subject fields refer to fields classified by subject, such as law, literature, engineering, society, science, and politics.

[0018] In this case, when multiple documents stored in the document database (111) are referred to as ‘document 1, document 2, document 3, document 4, document 5, ...’ and the subject fields of the above t types are referred to as ‘subject field 1, subject field 2, subject field 3, ...’, the document database (111) may store information regarding the subject fields corresponding to each of the multiple documents, such as ‘document 1, document 2, document 3, document 4, document 5, ...’, as shown in Table 1 below.

[0020] Multiple documents Subject Areas Document 1 Subject Area 3 Document 2 Subject Area 1 Document 3 Subject Area 2 Document 4 Subject Area 2 Document 5 Subject Area 1 ... ...

[0022] The main keyword generation unit (112) generates main keywords corresponding to each of the plurality of documents by opening each of the plurality of documents and extracting main keywords from the opened documents when a first query sentence is received from the user's electronic terminal (10) and a request command for providing an answer corresponding to the first query sentence is received.

[0023] In this case, according to one embodiment of the present invention, the service providing device (110) may include a word vector database (121).

[0024] The word vector database (121) stores word vectors corresponding to each of the multiple words.

[0025] Here, the word vector corresponding to each of the aforementioned multiple words is a pre-embedded vector based on the semantic similarity between words, and is configured such that a higher vector similarity is calculated as the semantic similarity between words increases. For example, if 'Word A' and 'Word B' have similar meanings, word vectors corresponding to 'Word A' and 'Word B' may be assigned so that a high vector similarity is calculated between the two words. Such word vectors may be assigned to each word according to the Word2Vec embedding technique.

[0026] At this time, according to one embodiment of the present invention, the main keyword generation unit (112) performs the process of generating main keywords corresponding to each of the plurality of documents by randomly assigning a sequence number to each of the plurality of documents, then sequentially opening each of the plurality of documents one by one according to the assigned sequence number, and extracting main keywords from the opened documents. When the first document, which is one of the plurality of documents, is opened as the sequence number to be opened, up to n words (n is a natural number greater than or equal to 2) are selected in order of frequency of appearance on the first document, and after confirming the word vector of each of the n words by referring to the word vector database (121), the importance score of each of the n words is calculated according to the operation of the following mathematical formula 1, and then up to k words (k is a natural number greater than or equal to 2) among the n words are extracted as main keywords corresponding to the first document.

[0028]

[0030] Here is the i-th word among the above n words ( With the importance score of ), is the i-th word among the above n words ( ) and the j-th word( It refers to the Euclidean distance between. This importance score can be used as a measure representing how many other words with high vector similarity are clustered around a specific word.

[0031] In this regard, as in the example above, when multiple documents stored in the document database (111) are referred to as ‘Document 1, Document 2, Document 3, Document 4, Document 5, ...’, the main keyword generation unit (112) can randomly assign a sequence number to each of the multiple documents as shown in Table 2 below.

[0033] Multiple documents turn Document 1 1 Document 2 3 Document 3 5 Document 4 2 Document 5 4 ... ...

[0035] Then, the main keyword generation unit (112) can generate main keywords corresponding to each of the plurality of documents by sequentially opening each of the plurality of documents one by one according to the assigned order and extracting main keywords from the opened documents. In this regard, the main keyword generation unit (112) can perform main keyword extraction by opening each document one by one in the order of 'document 1, document 4, document 2, document 5, document 3, ...'.

[0036] Let n be '15', and assume that it is time for 'Document 2', one of the multiple documents mentioned above, to be opened. Then, the main keyword generation unit (112) can open 'Document 2' and select up to 15 words in order of highest frequency of appearance in 'Document 2'. Here, since the main keyword generation unit (112) selects up to 15 words from 'Document 2' in order of highest frequency of appearance, if there are fewer than 15 words appearing in 'Document 2', the main keyword generation unit (112) can select all words appearing in 'Document 2'. That is, if there are only 9 words appearing in 'Document 2', the main keyword generation unit (112) selects all 9 words. In addition, if there are two or more words corresponding to the same rank during the process of selecting words, the main keyword generation unit (112) can randomly select one of the two or more words.

[0037] Let us assume that in this embodiment, the main keyword generation unit (112) selected up to 15 words from 'document 2' in order of greatest frequency of appearance, and selected 'word 1, word 2, word 3, word 4, word 5, word 6, word 7, word 8, word 9, word 10, word 11, word 12, word 13, word 14, word 15'.

[0038] Then, the main keyword generation unit (112) can check each word vector of ‘word 1, word 2, word 3, word 4, word 5, word 6, word 7, word 8, word 9, word 10, word 11, word 12, word 13, word 14, word 15’ by referring to the word vector database (121), and then calculate the importance score of each of the 15 words according to the operation of the above mathematical formula 1.

[0039] In this regard, to explain the process of calculating the importance score of 'Word 1', which is one of the 15 words mentioned above, the main keyword generation unit (112) can calculate the importance score of 'Word 1' according to the calculation process of the following mathematical formula 2 based on the formula of the above mathematical formula 1.

[0041]

[0043] In this way, the main keyword generation unit (112) can calculate the importance score of each of the 15 words.

[0044] Then, the main keyword generation unit (112) can extract up to k words among 'word 1, word 2, word 3, word 4, word 5, word 6, word 7, word 8, word 9, word 10, word 11, word 12, word 13, word 14, word 15' in order of importance score as main keywords corresponding to 'document 2'. For example, if k is '5', the main keyword generation unit (112) can extract up to 5 words among 'word 1, word 2, word 3, word 4, word 5, word 6, word 7, word 8, word 9, word 10, word 11, word 12, word 13, word 14, word 15' in order of importance score as main keywords corresponding to 'document 2'.

[0045] In this regard, when the five words in order of importance score among ‘word 1, word 2, word 3, word 4, word 5, word 6, word 7, word 8, word 9, word 10, word 11, word 12, word 13, word 14, word 15’ are ‘word 10, word 3, word 4, word 5, word 1’, the main keyword generation unit (112) can extract ‘word 10, word 3, word 4, word 5, word 1’ as main keywords corresponding to ‘document 2’.

[0046] In this way, the main keyword generation unit (112) can generate main keywords corresponding to each of the plurality of documents as shown in Table 3 below by sequentially opening each of the plurality of documents one by one according to the order assigned to each document and performing main keyword extraction.

[0048] Multiple documents Key Keywords Document 1 Word A, Word B, Word C, Word D, Word E Document 2 Word 10, Word 3, Word 4, Word 5, Word 1 Document 3 Word a, word b, word c, word d, word e Document 4 Word a, Word b, Word Ga, Word Na, Word Da Document 5 Word I, Word II, Word III, Word IV, Word V ... ...

[0050] The keyword set generation unit (113) generates a keyword set corresponding to each of the plurality of documents by configuring a keyword set having major keywords corresponding to each document as elements for each of the plurality of documents.

[0051] Let us assume that, as in the example described above, major keywords corresponding to each of the plurality of documents are generated by the major keyword generation unit (112) as shown in Table 3.

[0052] Then, the keyword set generation unit (113) can generate a keyword set having elements of the main keywords corresponding to 'document 1', namely 'word A, word B, word C, word D, word E', as {word A, word B, word C, word D, word E}, and can generate a keyword set having elements of the main keywords corresponding to 'document 2', namely 'word 10, word 3, word 4, word 5, word 1', as {word 10, word 3, word 4, word 5, word 1}, and can generate a keyword set having elements of the main keywords corresponding to 'document 3', namely 'word a, word b, word c, word d, word e', as {word a, word b, word c, word d, word e}.

[0053] In this way, the keyword set generation unit (113) can generate a keyword set corresponding to each of the plurality of documents as shown in Table 4 below.

[0055] Multiple documents Keyword set Document 1 {Word A, Word B, Word C, Word D, Word E} Document 2 {Word 10, Word 3, Word 4, Word 5, Word 1} Document 3 {word a, word b, word c, word d, word e} Document 4 {Word a, Word b, Word Ga, Word Na, Word Da} Document 5 {Word I, Word II, Word III, Word IV, Word V} ... ...

[0057] The union generation unit (114) generates a union corresponding to each of the t types of subject fields by calculating the union of the sets of keywords of the documents corresponding to each subject field for each of the t types of subject fields.

[0058] In this regard, let us assume that the above t types of subject fields are referred to as ‘Subject Field 1, Subject Field 2, Subject Field 3, ...’ as in the example above, and that information regarding documents corresponding to each subject field is stored as shown in Table 1 above.

[0059] Then, the union generation unit (114) can generate a union corresponding to 'subject field 1' by calculating the union of keyword sets of documents corresponding to 'subject field 1', such as 'document 2, document 5, ...'. For example, if the keyword set generated for each document is as shown in Table 4 above, the union generation unit (114) can generate a union corresponding to 'subject field 1' as {word 10, word 3, word 4, word 5, word 1, word I, word II, word III, word IV, word V, ...}.

[0060] In this way, the union generation unit (114) can generate unions corresponding to each of ‘subject area 1, subject area 2, subject area 3, ...’ as shown in Table 5 below.

[0062] Subject Areas union Subject Area 1 {Word 10, Word 3, Word 4, Word 5, Word 1, Word I, Word II, Word III, Word IV, Word V, ...} Subject Area 2 {word a, word b, word c, word d, word e, word Ga, word Na, word Da, ...} Subject Area 3 {Word A, Word B, Word C, Word D, Word E, ...} ... ...

[0064] The word set generation unit (115) identifies first words that appear at least once in the first query sentence and generates a word set having the first words as elements.

[0065] In this regard, when the first words are referred to as 'word X, word Y, word Z, ...', the word set generating unit (115) can identify the first words 'word X, word Y, word Z, ...' that appear at least once in the first query sentence and generate a word set having 'word X, word Y, word Z, ...' as elements as {word X, word Y, word Z, ...}.

[0066] The subject field selection unit (116) calculates the Jaccard Index between the union corresponding to each of the t types of subject fields and the word set, and selects a first subject field which is one of the t types of subject fields that has the maximum Jaccard Index with the word set.

[0067] Here, the Jaccard index refers to the similarity between two sets when two sets exist, and can be calculated according to the following mathematical formula 3.

[0069]

[0071] In the above mathematical formula 3, JI is the Jaccard index, is the size of the intersection of two sets A and B, Jaccard represents the size of the union of two sets A and B. Generally, the smaller the Jaccard index between two sets, the more dissimilar the two sets are considered to be, and the larger the Jaccard index between two sets, the more similar the two sets are considered to be.

[0072] In this regard, let us assume that, as in the example above, the union corresponding to each of the t types of subject fields, 'subject field 1, subject field 2, subject field 3, ...', was generated as shown in Table 5 above, and that the word set was generated as {word X, word Y, word Z, ...}.

[0073] Then, the subject area selection unit (116) can calculate the Jaccard index between the union corresponding to 'subject area 1', {word 10, word 3, word 4, word 5, word 1, word I, word II, word III, word IV, word V, ...} and the word set {word X, word Y, word Z, ...}, calculate the Jaccard index between the union corresponding to 'subject area 2', {word a, word b, word c, word d, word e, word Ga, word Na, word Da, ...} and the word set {word X, word Y, word Z, ...}, and calculate the Jaccard index between the union corresponding to 'subject area 3', {word A, word B, word C, word D, word E, ...} and the word set {word X, word Y, word Z, ...}.

[0074] Then, the subject area selection unit (116) can select a first subject area, which is one of the subject areas that has the maximum Jaccard index with the word set among the above t types of subject areas.

[0075] In this regard, if the subject field that has the maximum Jaccard index with the above set of words among ‘subject field 1, subject field 2, subject field 3, ...’ is called ‘subject field 1’, the subject field selection unit (116) can select ‘subject field 1’.

[0076] The event generating unit (117) checks whether the Jaccard index calculated for the first subject field exceeds a preset threshold, and if it is confirmed that it exceeds, determines that the first query sentence is a query sentence that can be used to provide an answer, and then generates an answer providing event to provide an answer corresponding to the first query sentence.

[0077] At this time, according to one embodiment of the present invention, the event generating unit (117) checks whether the Jaccard index calculated for the first subject field exceeds the reference value, and if it is confirmed that it does not exceed the reference value, it determines that the first query sentence is a query sentence that cannot be used for providing an answer, and instead of generating the answer providing event, it transmits a guidance message to the electronic terminal (10) instructing it to reinforce the first query sentence and query again, and at the same time, it can transmit keywords included as elements of the union corresponding to the first subject field as reference information for reinforcing the first query sentence.

[0078] Let’s assume that ‘topic field 1’ was selected as the first topic field by the topic field selection department (116) in relation to this.

[0079] Then, the event generating unit (117) checks whether the Jaccard index calculated for 'subject field 1' exceeds a preset threshold, and if it is confirmed that it exceeds, determines that the first query sentence is a query sentence that can be used to provide an answer, and then generates an answer providing event to provide an answer corresponding to the first query sentence.

[0080] Meanwhile, unlike the example above, if it is confirmed that the Jaccard index calculated for 'subject field 1' does not exceed the above threshold value, it cannot be said that there are many words matching the main keywords extracted from 'subject field 1' within the first query sentence entered by the user, and thus the event generating unit (117) can determine that the first query sentence is a query sentence that cannot be used for providing answers, in that it can be seen that there is a low degree of correlation between the first query sentence and 'subject field 1'.

[0081] Then, instead of generating the above-mentioned answer-providing event, the event generating unit (117) can transmit a guidance message instructing the electronic terminal (10) to reinforce the first query sentence and query again, and at the same time, transmit the keywords 'word 10, word 3, word 4, word 5, word 1, word I, word II, word III, word IV, word V, ...' which are included as elements of the union {word 10, word 3, word 4, word 5, word 1, word I, word II, word III, word IV, word V, ...} corresponding to 'subject field 1' as reference information for reinforcing the first query sentence.

[0082] Then, the user can view the guidance message and reference information, and reinforce the query sentence to formulate a query.

[0083] Thus, when the event generation unit (117) determines whether the first query sentence is a query sentence that can be used to provide an answer, and the first query sentence is determined to be a sentence that can be used to provide an answer, and the answer provision event occurs through the event generation unit (117), the document selection unit (118) selects one of the documents corresponding to the first subject field that has major keywords matching the first query sentence as a reference document for generating an answer.

[0084] In this case, according to one embodiment of the present invention, the document selection unit (118) may include a scoring unit (122) and a document selection processing unit (123).

[0085] The scoring unit (122) calculates a matching score for each document corresponding to the first subject area by performing a process in which, for each document corresponding to the first subject area, the main keywords corresponding to each document and the word vectors corresponding to the first words are identified by referring to the word vector database (121), the vector similarities between each main keyword corresponding to each document and each of the first words are all calculated, and when the calculated vector similarities are sorted in ascending order to calculate the quartiles, the overall average of the remaining vector similarities, excluding the vector similarities corresponding to the first quartile or lower, is calculated as the matching score corresponding to each document.

[0086] In this regard, as in the example described above, the first subject field is referred to as 'subject field 1', and the main keywords corresponding to 'document 2', which is one of the documents corresponding to 'subject field 1', are referred to as 'word 10, word 3, word 4, word 5, word 1', and assuming the first words are 'word X, word Y, word Z, ...', the process of the scoring unit (122) calculating the matching score corresponding to 'document 2' is explained as follows.

[0087] First, the scoring unit (122) can check the word vectors corresponding to ‘word 10, word 3, word 4, word 5, word 1’ and ‘word X, word Y, word Z, ...’ by referring to the word vector database (121).

[0088] Then the scoring unit (122) vector similarity between 'word 10' and 'word X', vector similarity between 'word 10' and 'word Y', vector similarity between 'word 10' and 'word Z', vector similarity between 'word 3' and 'word X', vector similarity between 'word 3' and 'word Y', vector similarity between 'word 3' and 'word Z', vector similarity between 'word 4' and 'word X', vector similarity between 'word 4' and 'word Y', vector similarity between 'word 4' and 'word Z', vector similarity between 'word 5' and 'word X', vector similarity between 'word 5' and 'word Y', vector similarity between 'word 5' and 'word Z', vector similarity between 'word 1' and 'word X', 'word 1' and 'word Vector similarity between Y', vector similarity between 'word 1' and 'word Z', ..., and vector similarity between 'word 10, word 3, word 4, word 5, word 1' and 'word X, word Y, word Z, ...' can all be calculated.

[0089] After that, the scoring unit (122) can calculate the quartiles by sorting the calculated vector similarities in ascending order.

[0090] Here, quartiles refer to data sorted in ascending order divided into four equal parts, where the first quartile is the variable value corresponding to the 1 / 4 place and the third quartile is the variable value corresponding to the 3 / 4 place.

[0091] At this time, if the vector similarities are referred to as 'V1, V2, V3, V4, V5, V6, V7, V8, V9, V10, ...' and the vector similarities corresponding to the first quartile or lower are referred to as 'V4, V7', the scoring unit (122) can calculate the total average of the remaining vector similarities, 'V1, V2, V3, V5, V6, V8, V9, V10, ...', excluding the vector similarities corresponding to the first quartile or lower among the vector similarities, as a matching score corresponding to 'document 2'.

[0092] In this way, the scoring unit (122) can calculate a matching score for each document corresponding to 'subject field 1' by calculating a matching score corresponding to each document.

[0093] The document selection processing unit (123) selects one document having the maximum matching score among the documents corresponding to the first subject area as the reference document having major keywords that match the first query sentence.

[0094] In this regard, if the first subject area is referred to as 'subject area 1' and the documents corresponding to 'subject area 1' are referred to as 'document 2, document 5, ...', the document selection processing unit (123) may select any one document among the documents corresponding to 'subject area 1', such as 'document 2, document 5, ...', that has the maximum matching score as the reference document having the main keywords that match the first query sentence. For example, if the document with the highest matching score among the documents corresponding to 'subject area 1' is referred to as 'document 2', the document selection processing unit (123) may select 'document 2' as the reference document having the main keywords that match the first query sentence.

[0095] The answer acquisition unit (119) generates a prompt that instructs the generation of an answer sentence corresponding to the first query sentence by referring to information contained in the document selected as the reference document, and then obtains an answer sentence corresponding to the first query sentence by inputting the prompt as input to a pre-established LLM (Large Language Model)-based answer generation model.

[0096] Here, an LLM (Large Language Model)-based answer generation model refers to a generative artificial intelligence model that receives a specific prompt and automatically generates an answer corresponding to the input prompt based on a pre-built LLM, and a Transformer-based language model can be utilized.

[0097] In this regard, as in the example above, when the document selected as the reference document is called ‘Document 2’, the answer acquisition unit (119) can generate a prompt that instructs to generate an answer sentence corresponding to the first query sentence by referring to the information contained in ‘Document 2’, and then obtain an answer sentence corresponding to the first query sentence by inputting the prompt as input to a pre-established LLM-based answer generation model.

[0098] When an answer sentence corresponding to the first query sentence is obtained in this way, the answer providing unit (120) transmits the answer sentence corresponding to the first query sentence to the electronic terminal (10).

[0099] Through this, the user of the electronic terminal (10) can obtain an appropriate answer sentence corresponding to the first query sentence entered by the user.

[0100] FIG. 2 is a flowchart illustrating the operation method of a service providing device according to an embodiment of the present invention.

[0101] In step (S211), a document database is maintained that stores information about a plurality of pre-specified documents to be used as references and a subject field corresponding to each of the plurality of documents (the subject field corresponding to each of the plurality of documents means one of the pre-specified subject fields corresponding to each document among the pre-specified t types of subject fields (t is a natural number)).

[0102] In step (S212), when a first query sentence is received from a user's electronic terminal and a request command for providing an answer corresponding to the first query sentence is received, each of the plurality of documents is opened and major keywords are extracted from the opened documents, thereby generating major keywords corresponding to each of the plurality of documents.

[0103] In step (S213), for each of the plurality of documents, a keyword set corresponding to each of the plurality of documents is generated by configuring a keyword set having major keywords corresponding to each document as elements.

[0104] In step (S214), for each of the t types of subject fields, a union of keyword sets of documents corresponding to each subject field is calculated, thereby generating a union corresponding to each of the t types of subject fields.

[0105] In step (S215), first words that appear at least once in the first query sentence are identified, and a word set having the first words as elements is generated.

[0106] In step (S216), the Jaccard index between the union corresponding to each of the t types of subject fields and the word set is calculated, and among the t types of subject fields, a first subject field is selected which is one of the subject fields having the union with the maximum Jaccard index with the word set.

[0107] In step (S217), it is checked whether the Jaccard index calculated for the first subject area exceeds a preset threshold. If it is confirmed that it exceeds the threshold, the first query statement is determined to be a query statement that can be used to provide an answer, and then an answer provision event is generated to provide an answer corresponding to the first query statement.

[0108] In step (S218), when the above-mentioned answer provision event occurs, one document among the documents corresponding to the above-mentioned first subject area that has major keywords matching the above-mentioned first query sentence is selected as a reference document for generating an answer.

[0109] In step (S219), a prompt is generated to instruct the generation of an answer sentence corresponding to the first query sentence by referring to the information contained in the document selected as the reference document, and then the answer sentence corresponding to the first query sentence is obtained by inputting the prompt as input to a pre-built LLM-based answer generation model.

[0110] In step (S220), an answer sentence corresponding to the first query sentence is transmitted to the electronic terminal.

[0111] At this time, according to one embodiment of the present invention, the method of operation of the service providing device further includes the step of maintaining a word vector database in which word vectors corresponding to each of a plurality of words are stored (the word vectors corresponding to each of the plurality of words are pre-embedded vectors based on semantic similarity between words, and are designated so that a higher vector similarity is calculated as the semantic similarity between words is higher), and in step (S212), a sequence number is randomly assigned to each of the plurality of documents, and each of the plurality of documents is opened one by one sequentially according to the assigned sequence number, and major keywords are extracted from the opened documents to generate major keywords corresponding to each of the plurality of documents, wherein when the sequence number for opening a first document, which is one of the plurality of documents, arrives, up to n words (n is a natural number greater than or equal to 2) are selected in order of decreasing frequency of appearance on the first document, and after confirming the word vectors of each of the n words by referring to the word vector database, the importance scores of each of the n words are calculated according to the operation of Equation 1. After calculation, up to k words (where k is a natural number greater than or equal to 2) among the n words can be extracted as major keywords corresponding to the first document in order of importance score.

[0112] In addition, according to one embodiment of the present invention, in step (S217), if it is determined that the Jaccard index calculated for the first subject field exceeds the reference value and does not exceed it, the first query sentence is determined to be a query sentence that cannot be used for providing an answer, and instead of generating the answer provision event, a guidance message is transmitted to the electronic terminal instructing it to reinforce the first query sentence and query again, and at the same time, keywords included as elements of the union corresponding to the first subject field can be transmitted together as reference information for reinforcing the first query sentence.

[0113] In addition, according to one embodiment of the present invention, step (S218) may include the step of calculating a matching score for each of the documents corresponding to the first subject field by performing a process of, for each of the documents corresponding to the first subject field, identifying the main keywords corresponding to each document and the word vectors corresponding to the first words by referring to the word vector database, calculating all vector similarities between each of the main keywords corresponding to each document and each of the first words, and when the calculated vector similarities are sorted in ascending order to calculate the quartiles, calculating the overall average of the remaining vector similarities, excluding the vector similarities corresponding to the first quartile or lower, as the matching score corresponding to each document, and the step of selecting one document having the maximum matching score among the documents corresponding to the first subject field as the reference document having the main keywords that match the first query sentence.

[0114] Hereinafter, a method of operation of a service providing device according to an embodiment of the present invention has been described with reference to FIG. 2. Since the method of operation of a service providing device according to an embodiment of the present invention may correspond to the configuration of the operation of the service providing device (110) described using FIG. 1, a more detailed description thereof will be omitted.

[0115] The method of operation of a service providing device according to one embodiment of the present invention may be implemented as a computer program stored in a storage medium for execution through combination with a computer.

[0116] In addition, the method of operation of a service providing device according to one embodiment of the present invention may be implemented in the form of computer program instructions for execution through combination with a computer and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either individually or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the present invention, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROMs and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

[0117] As described above, the present invention has been explained by specific details such as specific components, limited embodiments, and drawings; however, this is provided merely to aid in a more comprehensive understanding of the invention, and the invention is not limited to the above embodiments. A person skilled in the art can make various modifications and variations from this description.

[0118] Therefore, the scope of the present invention is not limited to the described embodiments, and all things equivalent to or having equivalent variations to the claims set forth below, as well as the claims set forth below, shall be considered to fall within the scope of the concept of the present invention. Explanation of the symbols

[0119] 110: Service providing device 111: Document Database 112: Major Keyword Generation Section 113: Keyword set generator 114: Union generator 115: Word Set Generation Unit 116: Subject Area Selection Unit 117: Event Generation Section 118: Document Selection Section 119: Answer Acquisition Unit 120: Answer Provision Unit 121: Word Vector Database 122: Scoring Section 123: Document Selection Processing Unit 10: Electronic terminal

Claims

Claim 1 A document database storing information regarding a plurality of pre-designated documents to be used as references and a subject field corresponding to each of the plurality of documents—wherein the subject field corresponding to each of the plurality of documents refers to any one subject field pre-designated among t types of subject fields (where t is a natural number) corresponding to each document—; a main keyword generation unit that generates main keywords corresponding to each of the plurality of documents by opening each of the plurality of documents and extracting main keywords from the opened documents when a first query sentence is received from a user's electronic terminal and a request command for providing an answer corresponding to the first query sentence is received; a keyword set generation unit that generates a keyword set corresponding to each of the plurality of documents by constructing a keyword set having the main keywords corresponding to each document as elements for each of the plurality of documents; a union generation unit that generates a union corresponding to each of the t types of subject fields by calculating the union of the keyword sets of documents corresponding to each subject field for each of the t types of subject fields; and appearing at least once in the first query sentence A word set generation unit that identifies first words and generates a word set having said first words as elements; a subject field selection unit that calculates a Jaccard Index between the word set and the union corresponding to each of said t types of subject fields, and selects a first subject field among said t types of subject fields that has a union with the maximum Jaccard Index with said word set; and an event generation unit that checks whether the Jaccard Index calculated for said first subject field exceeds a preset threshold, and if it is confirmed to exceed, determines that said first query sentence is a query sentence usable for providing an answer, and then generates an answer provision event to provide an answer corresponding to said first query sentence.A service providing device comprising: a document selection unit that, when the above-mentioned answer providing event occurs, selects one document among the documents corresponding to the above-mentioned first subject field that has major keywords matching the above-mentioned first query sentence as a reference document for generating an answer; an answer acquisition unit that generates a prompt instructing to generate an answer sentence corresponding to the above-mentioned first query sentence by referring to information contained within the document selected as the reference document, and then obtains an answer sentence corresponding to the above-mentioned first query sentence by inputting the prompt as input to a pre-established LLM (Large Language Model)-based answer generation model; and an answer providing unit that transmits the answer sentence corresponding to the above-mentioned first query sentence to the electronic terminal. Claim 2 In claim 1, the invention further includes a word vector database in which a word vector corresponding to each of the plurality of words is stored—the word vector corresponding to each of the plurality of words is a pre-embedded vector based on semantic similarity between words, specified so that a higher vector similarity is produced as the semantic similarity between words increases—and the main keyword generation unit performs a process of generating main keywords corresponding to each of the plurality of documents by randomly assigning a sequence number to each of the plurality of documents, then sequentially opening each of the plurality of documents one by one according to the assigned sequence number, and extracting main keywords from the opened documents; wherein, when the sequence number for opening a first document, which is one of the plurality of documents, arrives, up to n words (n is a natural number greater than or equal to 2) are selected in order of decreasing frequency of appearance on the first document, and after confirming the word vector of each of the n words by referring to the word vector database, the importance score of each of the n words is calculated according to the operation of the following mathematical formula 1, and then among the n words, the importance A service providing device characterized by extracting up to k words (where k is a natural number greater than or equal to 2) in descending order of score as major keywords corresponding to the first document.[Mathematical Formula 1] Here is the i-th word among the above n words ( With the importance score of ), is the i-th word among the above n words ( ) and the j-th word( It refers to the Euclidean distance between ). Claim 3 A service providing device according to claim 1, wherein the event generating unit checks whether the Jaccard index calculated for the first subject field exceeds the reference value, and if it is confirmed that it does not exceed the reference value, determines that the first query sentence is a query sentence that cannot be used for providing an answer, and instead of generating the answer providing event, transmits a guidance message instructing the electronic terminal to reinforce the first query sentence and query again, and at the same time transmits keywords included as elements of the union corresponding to the first subject field as reference information for reinforcing the first query sentence. Claim 4 A service providing device comprising: a document selection unit, for each of the documents corresponding to the first subject field, a scoring unit that calculates a matching score for each of the documents corresponding to the first subject field by performing a process of: identifying the main keywords corresponding to each document and the word vectors corresponding to the first words by referring to the word vector database for each of the documents corresponding to the first subject field; calculating all vector similarities between each of the main keywords corresponding to each document and each of the first words; and, when the calculated vector similarities are sorted in ascending order and quartiles are calculated, calculating the overall average of the remaining vector similarities, excluding the vector similarities corresponding to the first quartile or lower, as the matching score corresponding to each document; and a document selection processing unit that selects one of the documents corresponding to the first subject field that has the maximum matching score as the reference document having the main keywords that match the first query sentence. Claim 5 A step of maintaining a document database in which information regarding a plurality of pre-designated documents to be used as references and a subject field corresponding to each of the plurality of documents—wherein the subject field corresponding to each of the plurality of documents refers to any one subject field pre-designated among t types of subject fields (where t is a natural number) corresponding to each document—is stored; a step of generating major keywords corresponding to each of the plurality of documents by opening each of the plurality of documents and extracting major keywords from the opened documents when a first query sentence is received from a user's electronic terminal and a command requesting the provision of an answer corresponding to the first query sentence is received; a step of generating a keyword set corresponding to each of the plurality of documents by constructing a keyword set having the major keywords corresponding to each document as elements for each of the plurality of documents; a step of generating a union corresponding to each of the t types of subject fields by calculating the union of the keyword sets of documents corresponding to each subject field for each of the t types of subject fields; and a step of identifying first words that appear at least once in the first query sentence, and the A step of generating a word set having first words as elements; a step of calculating a Jaccard Index between the word set and the union corresponding to each of the t types of subject fields, and selecting a first subject field among the t types of subject fields that has a union with the maximum Jaccard Index with the word set; a step of checking whether the Jaccard Index calculated for the first subject field exceeds a preset threshold, and if it is confirmed to exceed, determining that the first query sentence is a query sentence usable for providing an answer, and then generating an answer providing event to provide an answer corresponding to the first query sentence;A method of operation of a service providing device comprising: a step of selecting, when the above-mentioned answer providing event occurs, one document having major keywords matching the first query sentence among the documents corresponding to the first subject field as a reference document for generating an answer; a step of generating a prompt instructing to generate an answer sentence corresponding to the first query sentence by referring to information contained within the document selected as the reference document, and then obtaining an answer sentence corresponding to the first query sentence by inputting the prompt as input to a pre-established LLM (Large Language Model)-based answer generation model; and a step of transmitting the answer sentence corresponding to the first query sentence to the electronic terminal. Claim 6 In claim 5, the method further includes the step of maintaining a word vector database in which word vectors corresponding to each of the plurality of words are stored—the word vectors corresponding to each of the plurality of words are pre-embedded vectors based on semantic similarity between words, and are specified such that a higher vector similarity is calculated as the semantic similarity between words increases—and the step of generating the main keywords comprises randomly assigning a sequence number to each of the plurality of documents, then sequentially opening each of the plurality of documents one by one according to the assigned sequence number, and extracting main keywords from the opened documents to generate main keywords corresponding to each of the plurality of documents, wherein when the sequence number for opening the first document, which is one of the plurality of documents, arrives, up to n words (where n is a natural number greater than or equal to 2) are selected in order of decreasing frequency of appearance on the first document, and after confirming the word vectors of each of the n words by referring to the word vector database, the importance score of each of the n words is calculated according to the operation of the following mathematical formula 1, and then among the n words A method of operation of a service providing device characterized by extracting up to k words (where k is a natural number greater than or equal to 2) in descending order of importance score as major keywords corresponding to the first document. [Mathematical Formula 1] Here is the i-th word among the above n words ( With the importance score of ), is the i-th word among the above n words ( ) and the j-th word( It refers to the Euclidean distance between ). Claim 7 A method of operation of a service providing device, wherein, in claim 5, the step of generating the answer providing event is characterized by checking whether the Jaccard index calculated for the first subject field exceeds the reference value, and if it is confirmed that it does not exceed the reference value, determining that the first query sentence is a query sentence that cannot be used for providing an answer, and instead of generating the answer providing event, sending a guidance message instructing the electronic terminal to reinforce the first query sentence and query again, and simultaneously transmitting keywords included as elements of the union corresponding to the first subject field as reference information for reinforcing the first query sentence. Claim 8 In claim 6, the step of selecting the reference document comprises: a step of calculating a matching score for each of the documents corresponding to the first subject field by performing a process of, for each of the documents corresponding to the first subject field, confirming the main keywords corresponding to each document and the word vectors corresponding to the first words by referring to the word vector database, calculating all vector similarities between each of the main keywords corresponding to each document and each of the first words, and, when the calculated vector similarities are sorted in ascending order and quartiles are calculated, calculating the overall average of the remaining vector similarities, excluding the vector similarities corresponding to the first quartile or lower, as the matching score corresponding to each document; and a step of selecting one document having the maximum matching score among the documents corresponding to the first subject field as the reference document having the main keywords that match the first query sentence. Claim 9 A computer-readable recording medium having a computer program for executing the method of any one of paragraphs 5 through 8 through combination with a computer. Claim 10 A computer program stored on a storage medium for executing the method of any one of paragraphs 5 through 8 through combination with a computer.

Citation Information

Patent Citations

  • Witness contradiction extraction method and system based on key comparison

    CN112434138A