Apparatus and method for processing civil service of public organization based on large language model

Abstract

A civil complaint processing device for a public institution based on a large language model according to one embodiment disclosed in this document includes a communication circuit configured to communicate with the outside, a memory, and a processor electrically connected to the communication circuit and the memory. The processor receives a query including a query from a civil complainant using the communication circuit, generates a dense vector and a sparse vector corresponding to the query using a first embedder and a second embedder stored in the memory, calculates a first relevance vector between the dense vectors corresponding to each of the multiple units included in each of the multiple reference documents and the dense vector corresponding to the query, calculates a second relevance vector between the sparse vectors corresponding to each of the multiple units included in each of the multiple reference documents and the sparse vector corresponding to the query, searches for the reference documents corresponding to the query and the units included in the reference documents based on the first relevance vector and the second relevance vector, and can provide an answer corresponding to the query based on the reference documents and the units included in the reference documents using the communication circuit.

Description

Technology Field

[0001] The embodiments disclosed in this document relate to an apparatus and method for processing civil complaint tasks of a public institution using a large language model. Background Technology

[0002] Currently, most public institutions operate online civil complaint reception systems. Online complaint submission can provide convenience to petitioners and improve administrative efficiency. Due to the high accessibility of online systems, petitioners can communicate with public institutions conveniently and quickly.

[0003] However, if the number of civil complaints continues to increase following the introduction of online systems, the workload of public officials handling the complaints may increase, which can cause difficulties in personnel management for public institutions. For example, requests for information disclosure targeting the government and local governments have increased every year, more than doubling over five years from approximately 860,000 cases in 2017 to 1.82 million cases in 2022. Therefore, the development of an artificial intelligence system capable of responding to online civil complaints may be required, and such an AI system can be implemented by a language model. The problem to be solved

[0004] To process civil complaints using language models, functionalities are required to provide a summary, search for relevant reference documents, and formulate a final response. Building a comprehensive system to answer queries using conventional large language models typically necessitates access to the open web or the possession of high-level resources. However, due to the specific characteristics of public institutions handling civil complaints, access to the open web often needs to be restricted, and possessing the high-level local resources required to build a system utilizing large language models may be practically difficult. Therefore, there is a need to develop a model specialized for deriving responses to civil complaints, taking into account the specific characteristics of public institutions.

[0005] Embodiments of the present invention are intended to provide an on-premise solution capable of operating on a system with limited resources to process civil complaint tasks using a language model. means of solving the problem

[0006] A civil complaint processing device for a public institution based on a large language model according to one embodiment disclosed in this document includes a communication circuit configured to communicate with the outside, a memory, and a processor electrically connected to the communication circuit and the memory. The processor receives a query including a query from a civil complainant using the communication circuit, generates a dense vector and a sparse vector corresponding to the query using a first embedder and a second embedder stored in the memory, calculates a first relevance vector between the dense vectors corresponding to each of the multiple units included in each of the multiple reference documents and the dense vector corresponding to the query, calculates a second relevance vector between the sparse vectors corresponding to each of the multiple units included in each of the multiple reference documents and the sparse vector corresponding to the query, searches for the reference documents corresponding to the query and the units included in the reference documents based on the first relevance vector and the second relevance vector, and can provide an answer corresponding to the query based on the reference documents and the units included in the reference documents using the communication circuit.

[0007] According to one embodiment, the processor scales a first relevance vector and a second relevance vector, and the first relevance vector is scaled based on a component to be scaled, a minimum value among the components of the first relevance vector, a deviation between the components and the minimum value, and a first scaling parameter, and the second relevance vector can be scaled based on a component to be scaled, a minimum value among the components of the second relevance vector, a deviation between the components and the minimum value, and a second scaling parameter.

[0008] According to one embodiment, the processor can automatically generate keywords and questions corresponding to the keywords for each of a plurality of units of each of a plurality of reference documents, calculate a first scaling parameter for a first embedder based on first relevance vectors between the automatically generated questions and the plurality of units, and calculate a second scaling parameter for a second embedder based on second relevance vectors between the automatically generated questions and the plurality of units.

[0009] According to one embodiment, the processor can generate a summary corresponding to a query using a machine learning model, search for a reference document corresponding to the query and a unit included in the reference document based on a first relevance vector and a second relevance vector between a plurality of units and the summary, and generate content included in the answer using causal language models.

[0010] A method for processing civil complaint tasks of a public institution based on a large language model according to one embodiment disclosed in this document may include: receiving a query including a query from a complainant; generating a dense vector and a sparse vector corresponding to the query using a first embedder and a second embedder; calculating a first relevance vector between the dense vectors corresponding to each of the multiple units included in each of the multiple reference documents and the dense vector corresponding to the query; calculating a second relevance vector between the sparse vectors corresponding to each of the multiple units included in each of the multiple reference documents and the sparse vector corresponding to the query; searching for the reference documents corresponding to the query and the units included in the reference documents based on the first relevance vector and the second relevance vector; and providing an answer corresponding to the query based on the reference documents and the units included in the reference documents. Effects of the invention

[0011] According to the embodiments disclosed in this document, by constructing a causal language model for generating answers that can be executed on a local device, a civil complaint processing system usable by public institutions requiring internal information security can be provided.

[0012] In addition, by searching for reference documents corresponding to a complaint based on the correlation vector between the dense and sparse vectors for the reference documents and the dense and sparse vectors for the query, the accuracy and efficiency of the search for resolving the complaint can be improved.

[0013] In addition, by automatically calculating parameters for scaling two relevance vectors, reference documents can be searched by considering the two relevance vectors in a balanced manner.

[0014] In addition, the accuracy of the generated answer can be improved by searching for reference documents that consider not only the petitioner's inquiry but also the summary of the inquiry.

[0015] In addition, various effects that can be identified directly or indirectly through this document may be provided. Brief explanation of the drawing

[0016] FIG. 1 is a block diagram illustrating the operation of a civil complaint processing device of a public institution based on a large language model according to one embodiment. FIG. 2 is a block diagram illustrating the configuration of a civil complaint processing device of a public institution based on a large language model according to one embodiment. FIG. 3 is a block diagram illustrating the operation of a civil complaint processing device of a public institution based on a large language model according to one embodiment. FIG. 4 is a diagram illustrating an exemplary reference document search operation of a civil complaint processing device of a public institution based on a large language model according to one embodiment. FIG. 5 is a diagram illustrating the embedding operation of an exemplary reference document of a civil complaint processing device of a public institution based on a large language model according to one embodiment. FIG. 6 is a diagram illustrating an exemplary metadata generation operation of a civil complaint processing device of a public institution based on a large language model according to one embodiment. FIG. 7 is a diagram illustrating an exemplary correlation vector calculation operation of a civil complaint processing device of a public institution based on a large language model according to one embodiment. FIG. 8 is a graph illustrating the characteristics of scaling parameters calculated in a public institution's civil complaint processing device based on a large language model according to one embodiment. FIG. 9 is a diagram illustrating an exemplary scaling parameter calculation operation of a public institution's civil complaint processing device based on a large language model according to one embodiment. FIG. 10 is a flowchart illustrating a method for processing civil complaints of a public institution based on a large language model according to one embodiment. In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Specific details for implementing the invention

[0017] Hereinafter, some embodiments of the present invention will be described in detail with reference to the exemplary drawings. However, this is not intended to limit the present invention to specific embodiments, and it should be understood that the invention includes various modifications, equivalents, or substitutions of the embodiments. It should be noted that in assigning reference numerals to the components of each drawing, the same components are given the same reference numeral whenever possible, even if they are shown in different drawings. Furthermore, in describing the embodiments of the present invention, if it is determined that a detailed description of related known components or functions would hinder understanding of the embodiments of the present invention, such detailed description is omitted.

[0019] FIG. 1 is a block diagram illustrating the operation of a civil complaint processing device of a public institution based on a large language model according to one embodiment.

[0020] Referring to FIG. 1, a civil complaint processing device according to one embodiment may include a model for generating an answer corresponding to a query entered by a complainant. The civil complaint processing device may be implemented as a model driving server on an institutional intranet.

[0021] The civil complaint processing device can receive a new civil complaint entered by a complainant through an application. The civil complaint processing device can generate a response to the new complaint, and the response may include a summary, response content, and reference document information. The civil complaint processing device can, for example, generate a summary of the query included in the new complaint using an end-to-end (E2E) model. The civil complaint processing device can, for example, generate response content using a causal language model. The civil complaint processing device can, for example, obtain a reference document based on the similarity between a vector embedding the query and a vector embedding the reference document. The obtained reference document may be utilized to generate the response content. The civil complaint processing device can generate a response including a summary, response content, and reference document information, and the structure of the response may be in a pre-specified format. The civil complaint processing device can output the generated response to the civil complaint manager.

[0022] The civil complaint processing device can receive CRUD (create, read, update, delete) requests for reference documents. The civil complaint processing device determines the validity of the processing, performs CRUD operations if valid, and rejects CRUD operations if invalid. The civil complaint processing device can output results. After performing CRUD operations, the civil complaint processing device can update the reference documents and the databases related to the reference documents. The database for the reference documents can, for example, store vectors that embed the reference documents.

[0024] FIG. 2 is a block diagram illustrating the configuration of a civil complaint processing device of a public institution based on a large language model according to one embodiment.

[0025] Referring to FIG. 2, a civil complaint processing device (200) of a public institution based on a large language model according to one embodiment may be implemented as one of various types of computing devices and may be physically implemented on-premises. The civil complaint processing device (200) according to one embodiment may include a communication circuit (210), a memory (220), and a processor (230).

[0026] The communication circuit (210) may be an interface that communicates wirelessly or via a wire with an external device (e.g., an external storage device or an external computing device). The communication circuit (210) may transmit and receive data with the external device.

[0027] The memory (220) may include volatile memory and / or non-volatile memory. The memory (220) may store various data handled by the civil complaint processing device (200), and may store language models and embedders, etc.

[0028] The processor (230) may be electrically connected to the communication circuit (210) and the memory (220). The processor (230) may control the communication circuit (210) and the memory (220) and perform various data processing and operations. The processor (230) may perform the following operations by executing software or instructions stored in the memory (220).

[0029] According to one embodiment, the processor (230) can receive a query including a petitioner's inquiry using a communication circuit (210). The petitioner's inquiry includes text and can be entered through a web page that receives online petitions. The processor (230) can receive a query including a inquiry through a web page.

[0030] According to one embodiment, the processor (230) can generate a dense vector and a sparse vector corresponding to a query using a first embedder and a second embedder stored in memory (220). The processor (230) can generate a vector corresponding to a query for data processing. For example, the first embedder may be a pre-trained dense passage retrieval (DPR). The second embedder may be BM25. The processor (230) can generate a dense vector corresponding to a query using the first embedder and generate a sparse vector corresponding to a query using the second embedder. The generated dense vector and sparse vector may be stored in memory (220).

[0031] According to one embodiment, the processor (230) can calculate a first relevance vector between dense vectors corresponding to each of the multiple units included in each of the multiple reference documents and dense vectors corresponding to the query. The processor (230) can divide the reference documents into multiple units. The processor (230) can divide the reference documents into sentence units or semantic units. The processor (230) can generate dense vectors corresponding to each of the units included in each of the previously stored reference documents using a first embedder, and the dense vectors corresponding to the units can be stored in advance. DPR is one of the deep learning algorithms used for phrase search and may include two independent encoders for processing phrases and queries, respectively. The two sub-modules can learn expressions of phrases and queries bidirectionally. The expressions learned or utilized may be modeled as embeddings of CLS (special classification token) tokens. The first relevance vector may be a vector representing the relevance between the dense vector corresponding to the query and each of the dense vectors corresponding to each of the multiple units. An exemplary mathematical formula for calculating the first correlation vector is as follows.

[0032] [Mathematical Formula 1]

[0033]

[0034] Here, q is the query, and P i is the i-th unit, and h CLS is the embedding of the CLS token, and s DPR (q; P) i represents the i-th component of the first relevance vector. That is, the i-th component of the first relevance vector can be defined as the cosine similarity between the query vector and the i-th unit vector. The range of each component of the first relevance vector can be from -1 to +1.

[0035] According to one embodiment, the processor (230) can calculate a second relevance vector between sparse vectors corresponding to each of a plurality of units included in each of a plurality of reference documents and a sparse vector corresponding to a query. The processor (230) can generate sparse vectors corresponding to each of all units included in each of all previously stored reference documents using a second embedder, and the sparse vectors corresponding to the units can be stored in advance. BM25 is an algorithm used as a ranking function to evaluate the relevance to documents for a given query, and is an algorithm that improves the performance of TF-IDF. The second relevance vector may be a vector representing the relevance between the sparse vector corresponding to the query and each of the sparse vectors corresponding to each of the plurality of units. An exemplary mathematical formula for calculating the second relevance vector is as follows.

[0036] [Mathematical Formula 2]

[0037]

[0038]

[0039] Here, f(q i , p) is the query token q i g is the number of times it appears in unit p, |p| is the unit length in token units, avgl is the average length of all units, and n(q i ) is token q i is the number of documents containing, N is the assumed number of documents, k1 and b are hyperparameters, and sBM25(q, P) j represents the j-th component of the second relevance vector.

[0040] According to one embodiment, the processor (230) can scale the first relevance vector and the second relevance vector. The processor (230) can search for reference documents to resolve complaints by the ensemble of the first relevance vector and the second relevance vector. However, since the ranges of each component of the first relevance vector and the second relevance vector are different, when the first relevance vector and the second relevance vector are summed (even if weights are applied), one of the two vectors may have a dominant influence on the result, and the other may be ignored. Therefore, rescaling of the first relevance vector and the second relevance vector may be required. The first relevance vector may be scaled based on the component to be scaled, the minimum value among the components of the first relevance vector, the deviation between the components and the minimum value, and the first scaling parameter, and the second relevance vector may be scaled based on the component to be scaled, the minimum value among the components of the second relevance vector, the deviation between the components and the minimum value, and the second scaling parameter. An exemplary mathematical formula for scaling the relevance vector is as follows.

[0041] [Mathematical Formula 3]

[0042]

[0043] Here, model refers to the embedding model, τ is a scaling parameter (which can be set to 1 by default), and can be, s' model (q; P) j can be the j-th component of the scaled relevance vector. Scaled relevance vector s' model The sum of each component of (q; P) can be 1. Thus, both relevance vectors can have the same effect on the search results.

[0044] τ can be a hyperparameter as a scaling parameter. By manipulating τ, the shape of the distribution of each component of the relevance vector can be adjusted. If τ is increased, the difference between large values ​​can be emphasized, and if τ is decreased, the difference between small values ​​can be emphasized. However, since the time required for sweeping the hyperparameter is very long, for efficiency, the processor (230) may automatically adjust τ by considering the distribution of each component of the relevance vector.

[0045] For example, the processor (230) can automatically generate keywords and questions corresponding to the keywords for each of the multiple units of each of the multiple reference documents, calculate a first scaling parameter for a first embedder based on first correlation vectors between the automatically generated questions and the multiple units, and calculate a second scaling parameter for a second embedder based on second correlation vectors between the automatically generated questions and the multiple units. The calculation of the scaling parameter is described in detail below with reference to FIG. 9. An exemplary mathematical formula for calculating the scaling parameter is as follows.

[0046] [Mathematical Formula 4]

[0047]

[0048] According to one embodiment, the processor (230) can search for reference documents and units contained in reference documents corresponding to a query based on a first relevance vector and a second relevance vector. For example, the processor (230) can search for highly relevance reference documents and units based on a weighted sum of scaled first relevance vectors and second relevance vectors. The processor (230) may also search for a predetermined number of units in order of high relevance using the relevance vectors. An exemplary mathematical formula for calculating the weighted sum of the relevance vectors is as follows.

[0049] [Mathematical Formula 5]

[0050]

[0051] Here, α is a weight for the first correlation vector and can be a hyperparameter.

[0052] According to one embodiment, the processor (230) can provide an answer corresponding to a query based on a reference document and a unit included in the reference document using a communication circuit (210). The answer may include a greeting, a summary, the content of the answer, and information about the reference document. The processor (230) can generate an answer according to a preset structure and provide the generated answer to a terminal of a civil complaint officer or an online civil complaint system. The reference document can be searched by the relevance vector described above, and the summary and the content of the answer can be generated in the following manner.

[0053] According to one embodiment, the processor (230) can generate a summary corresponding to a query using a machine learning model. The machine learning model for generating the summary may be an E2E model, for example, BART or T5. The generation of the summary is not only to shorten the query, but also to mimic the linguistic behavior of a public official responding to a civil complaint and to extract information for providing an appropriate response. The machine learning model may be a language model pre-trained using a Korean dataset (e.g., KoBART), and the language model may be fine-tuned using a dataset obtained from civil complaint log data.

[0054] According to one embodiment, the processor (230) can generate content (answer content) included in the answer using causal language models. Causal language models are relatively small models compared to conventional large language models such as ChatGPT, and the models can be pre-trained using documents containing complaints from which personal information has been removed. The pre-trained model may be selected as a language model with a limited number of parameters to ensure low system requirements. Causal language models may be, for example, Polyglot-ko-3.8b or llama-3-8b.

[0055] When a summary is generated, reference documents may be searched using the summary. Since the summary extracts information for providing an appropriate answer and removes unnecessary information, the accuracy of the search may be improved when the summary is utilized. For example, the processor (230) may search for reference documents corresponding to a query and units contained in the reference documents based on a first relevance vector and a second relevance vector between multiple units and the summary. That is, the processor (230) may generate dense vectors and sparse vectors corresponding to the summary, and calculate a first relevance vector and a second relevance vector between the dense vectors and sparse vectors corresponding to the summary and the dense vectors and sparse vectors corresponding to the units of the reference documents. The calculated first relevance vector and second relevance vector may be used to search for reference documents and units in a manner similar to that for the query. Additionally, the summary may be utilized to generate answer content.

[0057] FIG. 3 is a block diagram illustrating the operation of a civil complaint processing device of a public institution based on a large language model according to one embodiment.

[0058] Referring to FIG. 3, a civil complaint processing device according to one embodiment can receive a new civil complaint entered by a complainant through an application. The civil complaint processing device can generate a response to the new civil complaint, and the response may include a summary, response content, and reference document information. For convenience of explanation, a description of an operation similar to the one described with reference to FIG. 1 is omitted.

[0059] The civil complaint processing device may also utilize the generated summary for searching for reference documents. The civil complaint processing device may obtain reference documents by using a vector embedding the summary (in place of, or together with, a vector embedding the query). The civil complaint processing device may obtain reference documents based on the similarity between the vector embedding the summary and the vector embedding the reference document.

[0060] The civil complaint processing device may also utilize acquired reference documents to generate response content. The civil complaint processing device can generate response content by inputting reference documents along with a query into a causal language model.

[0062] FIG. 4 is a diagram illustrating an exemplary reference document search operation of a civil complaint processing device of a public institution based on a large language model according to one embodiment.

[0063] Referring to FIG. 4, a civil complaint processing device according to one embodiment can secure a plurality of reference documents for civil complaint processing. The civil complaint processing device can generate a vector database for each unit constituting the reference documents. The vector database may include dense vectors, sparse vectors, and metadata for each unit.

[0064] The civil complaint processing device can map a petitioner's query into a pair of vectors including a dense vector and a sparse vector. By executing a search routine on the vector database and the vector mapped to the query, the civil complaint processing device can retrieve a predetermined number (e.g., k) of highly relevant reference documents (or units).

[0066] FIG. 5 is a diagram illustrating the embedding operation of an exemplary reference document of a civil complaint processing device of a public institution based on a large language model according to one embodiment.

[0067] Referring to FIG. 5, a civil complaint processing device according to one embodiment can generate dense vectors, sparse vectors, and metadata for units included in a reference document. The civil complaint processing device can split the reference document into units using a tokenizer of a Korean sentence splitter (KSS) and / or a DPR. The civil complaint processing device can fit the units to a first embedder DPR and a second embedder BM25.

[0068] The civil complaint processing device can embed each unit into a dense vector using DPR and embed each unit into a sparse vector using BM25. Civil complaint data can generate metadata for each unit. The civil complaint processing device can store the generated dense vector, sparse vector, and metadata in a database.

[0070] FIG. 6 is a diagram illustrating an exemplary metadata generation operation of a civil complaint processing device of a public institution based on a large language model according to one embodiment.

[0071] Referring to FIG. 6, metadata for each unit of a reference document may include the path of the reference document, the ID of the reference document, the ID of the unit, and the content of the unit. The ID of the reference document is information assigned to identify each reference document, and the ID of the unit may be information assigned to identify each unit included in a specific reference document. The content of the unit may be a string consisting of text included in the unit. The ID of the metadata may be generated by combining the reference document ID and the unit ID. For example, if the reference document ID is a and the unit ID is b, the metadata ID for the corresponding unit may be ab. The metadata for the unit supports block-wise reading, and vectors may be grouped into blocks.

[0073] FIG. 7 is a diagram illustrating an exemplary correlation vector calculation operation of a civil complaint processing device of a public institution based on a large language model according to one embodiment.

[0074] Referring to FIG. 7, a civil complaint processing device according to one embodiment can generate a dense vector and a sparse vector for a query. For convenience of explanation, the dense vector and the sparse vector are assumed to be 8-dimensional.

[0075] A vector database may pre-store dense vectors for each unit of a reference document. For the convenience of explanation, it is assumed that the total number of units is 6. In this case, 6 8-dimensional dense vectors may be stored in the vector database. The civil complaint processing device may calculate a first relevance vector representing the similarity between the 6 dense vectors for the units and the dense vector for the query. When the total number of units is 6, the first relevance vector representing the similarity between the 6 unit dense vectors and the query dense vector may be 6-dimensional.

[0076] Sparse vectors for each unit of a reference document may be stored in advance in the vector database. In this case, six 8-dimensional sparse vectors may be stored in the vector database. The civil complaint processing device may calculate a second relevance vector representing the similarity between the six sparse vectors for the units and the sparse vector for the query. If the total number of units is six, the second relevance vector representing the similarity between the six unit sparse vectors and the query sparse vector may be six-dimensional.

[0077] The civil complaint processing device can search for two documents with high relevance (the number of documents searched can be set in various ways) by the ensemble of the first relevance vector and the second relevance vector.

[0079] FIG. 8 is a graph illustrating the characteristics of scaling parameters calculated in a public institution's civil complaint processing device based on a large language model according to one embodiment.

[0080] Referring to Fig. 8, the horizontal axis of the graph represents the normalized value of each component of the relevance vector, and the vertical axis represents the scaled probability distribution. τ may be a hyperparameter or may be optimized by a civil complaint processing device, as will be explained with reference to Fig. 9. If τ is set to a small value, the difference between small component values ​​can be emphasized, as shown in the top graph (τ=0.125). If τ is set to a large value, the difference between large component values ​​can be emphasized, as shown in the bottom graph (τ=8). τ needs to be set appropriately according to the distribution of each component of the relevance vector.

[0082] FIG. 9 is a diagram illustrating an exemplary scaling parameter calculation operation of a public institution's civil complaint processing device based on a large language model according to one embodiment.

[0083] Referring to FIG. 9, a civil complaint processing device according to one embodiment can calculate parameters used for scaling a relevance vector. The civil complaint processing device can extract keywords from each unit of a reference document. Keywords can be extracted using TF-IDF. Here, keywords may include multiple words, and in this case, may be referred to as a key phrase.

[0084] The civil complaint processing device can automatically generate questions based on templates. The format of the question can be predefined, and a question can be generated by combining that format with keywords.

[0085] The civil complaint processing device can calculate the entropy for a correlation vector produced by two models (e.g., DPR and BM25) (defined in the present invention as a value representing the degree of uncertainty of the vector's components). Typically, the uncertainty may be higher for DPR and lower for BM25. To predict the uncertainty of the model itself, τ is set, and the entropy for each of the two models can be calculated according to Equation 4.

[0086] The civil complaint processing device has the calculated entropy ( τ can be calculated by taking the square root of ) and the calculated τ can be applied to Equation 3. Taking the square root of entropy was determined empirically, and various operations can be applied to entropy as needed.

[0088] FIG. 10 is a flowchart illustrating a method for processing civil complaints of a public institution based on a large language model according to one embodiment.

[0089] In the following, it is assumed that the civil complaint processing device of FIG. 2 performs the process of FIG. 10. Also, in the description of FIG. 10, the operation described as being performed by the civil complaint processing device can be understood as being controlled by the processor (230).

[0090] Referring to FIG. 10, in step 1010, the civil complaint processing device can receive a query including a query from a civil complainter.

[0091] In step 1020, the civil complaint processing device can generate dense vectors and sparse vectors corresponding to the query using the first embedder and the second embedder.

[0092] In step 1030, the civil complaint processing device can calculate a first correlation vector between the dense vectors corresponding to each of the multiple units included in each of the multiple reference documents and the dense vector corresponding to the query.

[0093] In step 1040, the civil complaint processing device can calculate a second correlation vector between sparse vectors corresponding to each of the multiple units included in each of the multiple reference documents and sparse vectors corresponding to the query.

[0094] In step 1050, the civil complaint processing device can search for reference documents corresponding to the query and units included in the reference documents based on the first relevance vector and the second relevance vector.

[0095] In step 1060, the civil complaint processing device can provide an answer corresponding to the query based on a reference document and a unit included in the reference document.

[0097] The embodiments of this document and the terms used therein are not intended to limit the technology described in this document to specific embodiments and should be understood to include various modifications, equivalents, and / or substitutions of said embodiments. In relation to the description of the drawings, similar reference numerals may be used for similar components. A singular expression may include a plural expression unless the context clearly indicates otherwise. In this document, expressions such as "A or B," "at least one of A and / or B," "A, B or C," or "at least one of A, B and / or C" may include all possible combinations of items listed together. Expressions such as "first," "second," "first," or "second" may modify said components regardless of order or importance and are used only to distinguish one component from another and do not limit said components. When it is mentioned that a component is "(functionally or telecommunicationally) connected" or "connected" to another component, said component may be directly connected to said other component or connected through said other component.

[0098] In this document, "adapted to or configured to" may be used interchangeably with, depending on the context, for example, hardware- or software-wise, "suitable for," "capable of," "modified to," "made to," "capable of," or "designed to." In some cases, the expression "device configured to" may mean that the device is "capable of" in conjunction with other devices or components. For example, the phrase "processor configured to perform A, B, and C" may mean a dedicated processor for performing those operations (e.g., an embedded processor) or a general-purpose processor (e.g., a CPU) capable of performing those operations by executing one or more programs stored in a memory device.

[0099] As used in this document, the term “module” includes a unit composed of hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A “module” may be a component formed as a whole or a minimum unit or part thereof that performs one or more functions. A “module” may be implemented mechanically or electronically and may include, for example, an application-specific integrated circuit (ASIC) chip, field-programmable gate arrays (FPGAs), or programmable logic device, known or under development, that performs certain operations.

[0100] At least a portion of a device (e.g., modules or functions thereof) or a method (e.g., operations) according to one embodiment may be implemented as instructions stored in a computer-readable storage medium in the form of program modules. When said instructions are executed by a processor, the processor may perform a function corresponding to said instructions.

[0101] Each component (e.g., module or program module) according to one embodiment may be composed of a single or multiple entities, and some of the aforementioned sub-components may be omitted or additional sub-components may be included. Generally or additionally, some components (e.g., module or program module) may be integrated into a single entity to perform the functions performed by each of the respective components prior to integration in the same or similar manner. The operations performed by the module, program module, or other components according to one embodiment may be executed sequentially, in parallel, iteratively, or heuristically, or at least some operations may be executed in a different order, omitted, or other operations added.

Claims

Claim 1 In a civil complaint processing device for a public institution based on a large language model, a communication circuit configured to communicate with the outside; memory; and includes a processor electrically connected to the communication circuit and the memory, wherein the processor receives a query including a petitioner's inquiry using the communication circuit, generates a dense vector and a sparse vector corresponding to the query using a first embedder and a second embedder stored in the memory, calculates a first relevance vector between the dense vectors corresponding to each of the multiple units included in each of the multiple reference documents and the dense vector corresponding to the query, calculates a second relevance vector between the sparse vectors corresponding to each of the multiple units included in each of the multiple reference documents and the sparse vector corresponding to the query, searches for the reference document corresponding to the query and the unit included in the reference document based on the first relevance vector and the second relevance vector, and provides an answer corresponding to the query based on the reference document and the unit included in the reference document using the communication circuit, wherein the processor scales the first relevance vector and the second relevance vector, and the first relevance vector comprises a component to be scaled and components of the first relevance vector The second relevance vector is scaled based on a minimum value, a deviation between the components and the minimum value, and a first scaling parameter, and the second relevance vector is scaled based on a component to be scaled, a minimum value among the components of the second relevance vector, a deviation between the components and the minimum value, and a second scaling parameter, and the processor automatically generates a keyword for each of the plurality of units of each of the plurality of reference documents and a question corresponding to the keyword, and calculates the first scaling parameter for the first embedder based on the first relevance vectors between the automatically generated question and the plurality of units, andA device characterized by calculating the second scaling parameter for the second embedder based on the second correlation vectors between the automatically generated question and the plurality of units. Claim 2 delete Claim 3 delete Claim 4 An apparatus according to claim 1, wherein the processor generates a summary corresponding to the query using a machine learning model, searches for the reference document corresponding to the query and the unit included in the reference document based on a first relevance vector and a second relevance vector between the plurality of units and the summary, and generates content included in the answer using causal language models. Claim 5 A method for processing civil complaint tasks of a public institution based on a large language model executed by a processor included in a computing device, comprising: receiving a query including a query from a complainant; generating a dense vector and a sparse vector corresponding to the query using a first embedder and a second embedder; calculating a first relevance vector between the dense vectors corresponding to each of the multiple units included in each of the multiple reference documents and the dense vector corresponding to the query; calculating a second relevance vector between the sparse vectors corresponding to each of the multiple units included in each of the multiple reference documents and the sparse vector corresponding to the query; scaling the first relevance vector and the second relevance vector; and searching for the reference document corresponding to the query and the unit included in the reference document based on the first relevance vector and the second relevance vector.A method comprising the step of providing an answer corresponding to the query based on the reference document and the unit included in the reference document, wherein the first relevance vector is scaled based on a component to be scaled, a minimum value among the components of the first relevance vector, a deviation between the components and the minimum value, and a first scaling parameter, and the second relevance vector is scaled based on a component to be scaled, a minimum value among the components of the second relevance vector, a deviation between the components and the minimum value, and a second scaling parameter, wherein the scaling step comprises the step of automatically generating a keyword for each of the plurality of units of each of the plurality of reference documents and a question corresponding to the keyword, the step of calculating the first scaling parameter for the first embedder based on the first relevance vectors between the automatically generated question and the plurality of units, and the step of calculating the second scaling parameter for the second embedder based on the second relevance vectors between the automatically generated question and the plurality of units.

Images