Method and system for retrieving hybrid ranking data based on retrieval augmented generation

Abstract

A method for retrieving data related to a query is provided. A data retrieval method performed by a computing system according to some embodiments may include the steps of receiving a query from a user terminal, searching for a plurality of researcher data related to the characteristics of the query in a first database, searching for a plurality of document data related to the plurality of researcher data in a second database, calculating a score for each of the retrieved plurality of document data, and determining one or more document data to be used to generate a response to the query based on the calculated score of the plurality of document data.

Description

Technology Field

[0001] The present disclosure relates to a method and system for retrieving data related to a query. More specifically, it relates to a method and system for retrieving a plurality of data similar to a feature of a query from a database and determining data necessary for generating a response based on the score of the retrieved data. Background Technology

[0002] Keyword-based search methods are utilized for information retrieval. However, keyword-based search methods may yield information that does not match what the user intends. For instance, without considering synonyms and related terms for a keyword, only information matching the keyword may be provided to the user. In such cases, the information the user wants may not be found, requiring the user to search again by changing the keywords.

[0003] Meanwhile, when a user searches for desired information in a database storing research papers, etc., if a keyword-based search method is used, the retrieved data may vary depending on the combination of keywords.

[0004] Accordingly, there is a growing need for technology that can conveniently and accurately search for the information users require. The problem to be solved

[0005] The technical problem to be solved in some embodiments of the present disclosure is to provide a data retrieval method and system capable of accurately retrieving and providing required information.

[0006] The technical problem to be solved in some embodiments of the present disclosure is to provide a method and system for accurately retrieving information related to a researcher.

[0007] The technical problem to be solved in some embodiments of the present disclosure is to provide a data retrieval method and system that improves the accuracy of search results using activity scores.

[0008] The technical problems of the present disclosure are not limited to those mentioned above, and other unmentioned technical problems will be clearly understood by a person skilled in the art from the description below. means of solving the problem

[0009] A data retrieval method performed by a computing system according to an embodiment of the present disclosure for solving the above technical problem may include: receiving a query from a user terminal; searching for a plurality of researcher data related to the characteristics of the query in a first database; searching for a plurality of document data related to the plurality of researcher data in a second database; calculating a score for each of the retrieved plurality of document data; and determining one or more document data used to generate a response to the query based on the calculated score of the plurality of document data.

[0010] According to some embodiments, the step of calculating a score for each of the retrieved multiple document data may include the step of calculating an activity score for each of the multiple document data and the step of calculating a score for each of the multiple documents based on the similarity between the features of the query and the features of the document data and the activity score.

[0011] According to some embodiments, the step of calculating an activity score may include the step of calculating at least one of a research productivity indicator, a collaboration network indicator, or an expertise indicator for specific document data using at least one of a plurality of researcher data or a plurality of document data, and the step of calculating an activity score for specific document based on at least one of the calculated research productivity indicator, collaboration network indicator, or expertise indicator.

[0012] According to some embodiments, the step of calculating at least one of a research productivity indicator, a collaboration network indicator, or an expertise indicator for specific document data may include the step of selecting document data having a publication time included in a predetermined period among a plurality of retrieved document data, the step of identifying researcher identifiers included in the specific document data and counting the number of researcher identifiers in the selected document data, and the step of determining the counted number of researcher identifiers as a research productivity indicator for the specific document data.

[0013] According to some embodiments, the step of calculating at least one of a research productivity indicator, a collaboration network indicator, or an expertise indicator for specific document data may include the step of identifying a researcher identifier included in the specific document data, the step of counting the number of other researcher identifiers listed as co-authors with the researcher identifier in a plurality of retrieved document data, and the step of determining the value obtained by dividing the counted number of other researcher identifiers by the total number of researchers as the collaboration network indicator.

[0014] According to some embodiments, the step of calculating at least one of a research productivity indicator, a cooperation network indicator, or an expertise indicator for specific document data may include the step of identifying keywords included in the specific document data, the step of calculating entropy indicating how much the keywords are distributed in document data other than the specific document data, and the step of determining the calculated entropy as an expertise indicator for the specific document data.

[0015] According to some embodiments, the step of searching for a plurality of document data in a second database may include the step of obtaining a plurality of researcher identifiers included in the plurality of researcher data and the step of searching for a plurality of document data including at least one of the plurality of researcher identifiers in the second database.

[0016] According to some embodiments, the data retrieval method may further include, after the step of determining one or more document data used to generate a response to a query, the step of transmitting a response generated using the determined one or more document data to a user terminal.

[0017] According to some embodiments, the step of transmitting the generated response to a user terminal may include the step of inputting a query and one or more determined document data into a large-scale language model to obtain a response.

[0018] According to some embodiments, the data retrieval method may further include, prior to the step of retrieving a plurality of researcher data in a first database, the step of applying document or researcher information to an artificial intelligence model to obtain researcher data including an institution name and a researcher identifier, and the step of storing the obtained researcher data in the first database.

[0019] According to some embodiments, the data retrieval method may further include, prior to the step of retrieving a plurality of researcher data in a first database, the step of applying a plurality of documents to an artificial intelligence model to obtain a plurality of document data including a title, abstract, and researcher identifier, and the step of storing the obtained plurality of document data in a second database.

[0020] According to some embodiments, the step of storing a plurality of document data in a second database may include the step of dividing the plurality of document data into a plurality of items and the step of storing the divided plurality of items in the second database in units of a predetermined batch size.

[0021] According to some embodiments, the step of dividing each of the plurality of document data into a plurality of items includes the step of determining a weight for each of the divided plurality of items, and the weight may be reflected when calculating the similarity between the query and the document data.

[0022] A data retrieval method by a computing system according to another embodiment of the present disclosure for solving the above technical problem may include: receiving a query from a user terminal; searching a plurality of document data related to the features of the query in a database; calculating an activity score for each of the plurality of document data; calculating a score for each of the plurality of documents based on the similarity between the features of the query and the features of the document data and the activity score; and determining one or more document data used to generate a response to the query based on the calculated scores of the plurality of document data.

[0023] A computing system according to one embodiment of the present disclosure for solving the above technical problem comprises one or more processors and a memory for storing a computer program executed by one or more processors, and the computer program may include instructions for performing operations such as receiving a query from a user terminal, searching for a plurality of researcher data related to the characteristics of the query in a first database, searching for a plurality of document data related to the plurality of researcher data in a second database, calculating a score for each of the searched plurality of document data, and determining one or more document data used to generate a response to the query based on the calculated score of the plurality of document data.

[0024] A computing system according to another embodiment of the present disclosure for solving the above technical problem comprises one or more processors and a memory for storing a computer program executed by one or more processors, and the computer program may include instructions for performing operations such as receiving a query from a user terminal, searching a plurality of document data related to the features of the query in a database, calculating an activity score for each of the plurality of document data, calculating a score for each of the plurality of documents based on the similarity between the features of the query and the features of the document data and the activity score, and determining one or more document data used to generate a response to the query based on the calculated score of the plurality of document data.

[0025] A computer program according to one embodiment of the present disclosure for solving the above technical problem may be stored on a computer-readable, non-transient recording medium to execute the steps of: receiving a query from a user terminal; searching for a plurality of researcher data related to the characteristics of the query in a first database; searching for a plurality of document data related to the plurality of researcher data in a second database; calculating a score for each of the retrieved plurality of document data; and determining one or more document data used to generate a response to the query based on the calculated score of the plurality of document data. Brief explanation of the drawing

[0026] FIG. 1 is a diagram illustrating a system environment for providing a search service according to one embodiment of the present disclosure. FIG. 2 is a diagram illustrating researcher data stored in a first database and document data stored in a second database, according to one embodiment of the present disclosure. FIG. 3 is a diagram illustrating researcher data and document data being obtained through an artificial intelligence model according to one embodiment of the present disclosure. FIG. 4 is a flowchart illustrating a method for storing researcher data and document data according to one embodiment of the present disclosure. FIG. 5 is a flowchart illustrating a method of storing a plurality of data in batch sizes according to one embodiment of the present disclosure. Figure 6 is a diagram visualizing document data stored in batch size. FIG. 7 is a flowchart illustrating a method for generating a response to a query according to one embodiment of the present disclosure. Figure 8 is a diagram to explain step S740 of Figure 7 in more detail. FIG. 9 is a flowchart illustrating a method for generating a response to a query according to another embodiment of the present disclosure. FIG. 10 is an exemplary hardware configuration diagram that can implement a computing system in various embodiments. Specific details for implementing the invention

[0027] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the attached drawings. The advantages and features of the present disclosure and the methods for achieving them will become clear by referring to the embodiments described below in detail together with the attached drawings. However, the technical concept of the present disclosure is not limited to the following embodiments but can be implemented in various different forms. The following embodiments are provided merely to complete the technical concept of the present disclosure and to fully inform those skilled in the art of the scope of the present disclosure, and the technical concept of the present disclosure is defined only by the scope of the claims.

[0028] It should be noted that when assigning reference numerals to the components of each drawing, the same components are given the same reference numeral whenever possible, even if they are shown in different drawings. Furthermore, in describing the present disclosure, if it is determined that a detailed description of related known components or functions could obscure the essence of the present disclosure, such detailed description is omitted.

[0029] Unless otherwise defined, all terms used herein (including technical and scientific terms) may be used in a meaning commonly understood by those skilled in the art to which this disclosure pertains. Additionally, terms defined in commonly used dictionaries are not to be interpreted ideally or excessively unless explicitly and specifically defined otherwise. The terms used herein are for describing the embodiments and are not intended to limit this disclosure. In this specification, the singular form includes the plural form unless specifically stated otherwise in the text.

[0030] Additionally, terms such as first, second, A, B, (a), (b), etc., may be used to describe the components of the present disclosure. These terms are intended only to distinguish the components from other components and do not limit the nature, order, or sequence of the components. Where it is stated that a component is "connected," "coupled," or "joined" to another component, it should be understood that the component may be directly connected or joined to the other component, but that another component may also be "connected," "coupled," or "joined" between each component.

[0031] The components, steps, actions, and / or elements mentioned in "comprises" and / or "comprising" as used in the specification do not exclude the presence or addition of one or more other components, steps, actions, and / or elements.

[0032] Hereinafter, several embodiments of the present disclosure will be described in detail with reference to the attached drawings.

[0033] FIG. 1 is a diagram illustrating a system environment for providing a search service according to one embodiment of the present disclosure.

[0034] As illustrated in FIG. 1, the information processing system (110) may be a system for providing search services through question and answer. The information processing system (110) may communicate with the user terminal (140) and the document management system (130), respectively, through the network (150). Here, the network (150) includes a wired communication network and a wireless communication network, and since this corresponds to a well-known conventional technology in the present disclosure, a detailed description is omitted.

[0035] The document management system (130) can store various documents such as papers, reports, books, and academic materials. Each document may include a researcher, an institution name, keywords, an abstract, etc. In this embodiment, the researcher is a person who has researched the content of the document and may be referred to by other expressions such as author or writer. In addition, there may be multiple researchers. In such cases, additional co-researchers may be included in the document.

[0036] The user terminal (140) is a personal computer, a smartphone, etc., and can transmit a query related to a search to an information processing system (110) and then receive a response related to the search results from the information processing system (110). In some embodiments, the query may be composed of natural language.

[0037] The database (120) is a large-capacity storage means and may include a first database (210) containing researcher data and a second database (220) containing document data. According to some embodiments, features of the researcher data may be associated with the researcher data and stored in the first database (210), and features of the document data may be associated with the document data and stored in the second database (220). Here, the features may be vectors. The vectors may be multidimensional vectors generated through an embedding model. The first database (210) and the second database (220) included in the database (120) will be described later with reference to FIG. 2.

[0038] The information processing system (110) is a computing system including one or more processors and memory, and can construct a first database (210) and a second database (220). Additionally, the information processing system (110) can perform the function of providing a search service for documents.

[0039] According to one embodiment, when an information processing system (110) receives a query related to document search from a user terminal (140), it may obtain a response to the query and transmit it to the user terminal (140). The information processing system (110) may search for multiple researcher data related to the characteristics of the query in a first database (210) and search for multiple document data related to the multiple researcher data in a second database (220). Additionally, the information processing system (110) may calculate a score for each of the multiple document data retrieved, obtain a response to the query based on the multiple document data retrieved and the score, and then transmit the obtained response to the user terminal (140).

[0040] The method of generating a response through the information processing system (110) will be described later with reference to FIGS. 7 to 9.

[0041] FIG. 2 is a diagram illustrating researcher data stored in a first database (210) and document data stored in a second database (220) according to one embodiment of the present disclosure.

[0042] As illustrated in FIG. 2, the researcher data stored in the first database (210) may include a researcher identifier (AID), an organization name, terms, etc. Here, terms may be technical terms and may be extracted from a document based on a list of technical terms. For example, terms may be words that can be extracted from abstracts, titles, keywords, etc., in a document. As another example, terms may be extracted from words included in the original document.

[0043] According to some embodiments, a first feature of the researcher data is obtained through an embedding model, etc., and the first feature and the researcher data can be stored together in a first database (210).

[0044] Document data stored in the second database (220) may include a researcher's name, title, abstract, keyword, researcher identifier, document publication date, etc. Each of the researcher's name, title, abstract, keyword, and document publication date may be extracted from the document. A second feature of the document data is obtained through an embedding model, and the second feature and the document data may be stored together in the second database (220).

[0045] According to some embodiments, researcher data and document data may include one or more of metadata and source data. The source data included in the document data may be the document itself.

[0046] Researcher data can be obtained through a pre-trained artificial intelligence model (300). According to some embodiments, the artificial intelligence model may include an artificial neural network and be trained to understand natural language. For example, the artificial intelligence model (300) may include a Large Language Model (LLM). The artificial intelligence model (300) may be pre-trained to interpret input data and output data required from the input data. According to some embodiments, the artificial intelligence model (300) may be trained and configured to infer at least one of an institution name or a researcher name from the input data. For example, the artificial intelligence model (300) may be a generative artificial intelligence model capable of generating a response to a query.

[0047] Hereinafter, with reference to FIGS. 3 to 6, a method for storing researcher data in the first database (210) will be explained.

[0048] FIG. 3 is a diagram illustrating researcher data and document data being obtained through an artificial intelligence model according to one embodiment of the present disclosure.

[0049] As illustrated in FIG. 3, the artificial intelligence model (300) may be configured to obtain researcher data (320) and document data (330) from a document (310) in the form of a file or web page. The researcher data may include a researcher identifier, an institution name, terms, etc., and the document data may include a researcher name, a title, an abstract, keywords, and a researcher identifier.

[0050] According to one embodiment, the artificial intelligence model (300) may be configured to identify a researcher's email from a document (310) and determine the researcher's email as a researcher identifier. Additionally, the artificial intelligence model (300) may be configured to identify an organization name, terms, etc. from a document (310). Additionally, the artificial intelligence model (300) may be configured to identify a researcher's name, title, abstract, keywords, etc. from a document (310).

[0051] According to some embodiments, the artificial intelligence model (300) may include a plurality of artificial intelligence models. In this case, the first artificial intelligence model included in the artificial intelligence model (300) may be configured to identify a researcher identifier from a document (310) or separate researcher information (e.g., a researcher file). Additionally, the second artificial intelligence model included in the artificial intelligence model (300) may be configured to identify an institution name from a document (310) or separate institution information (e.g., an institution file). Additionally, the third artificial intelligence model included in the artificial intelligence model (300) may be configured to obtain document data including a researcher name, title, abstract, keyword, researcher identifier, etc. from the document (310).

[0052] FIG. 4 is a flowchart illustrating a method for storing researcher data and document data according to one embodiment of the present disclosure.

[0053] Referring to FIG. 4, the information processing system (110) can collect documents (S410). For example, the information processing system (110) can access the document management system (130) to collect the relevant documents. As another example, the information processing system (110) can collect documents through web crawling. The documents may be in the form of files or web pages.

[0054] Afterward, the information processing system (110) can apply the document to the artificial intelligence model (300) to obtain researcher data and document data (S420). According to some embodiments, a researcher file in which researcher information is recorded can be applied to the artificial intelligence model (300) to obtain researcher data. The document data may include one or more of the original document and metadata.

[0055] Next, the information processing system (110) can obtain a first feature of the researcher data by applying the researcher data to the embedding model (S430). Additionally, the information processing system (110) can obtain a second feature of the document data by applying the document data to the embedding model (S440). Here, the first feature and the second feature may be an embedded vector.

[0056] After that, the information processing system (110) can store researcher data and first features in the first database (210) (S450). Additionally, the information processing system (110) can store document data and second features in the second database (220) (S460).

[0057] Researcher data stored in the first database (210) and document data stored in the second database (220) can be associated with each other through a researcher identifier (AID).

[0058] If a first database (210) containing researcher-centered data (i.e., researcher data) is separately established, when a query related to a researcher or institution is received, researcher data and document data that more accurately match the query can be retrieved.

[0059] Meanwhile, weights are assigned to each item included in the document data, and multiple document data can be stored in the second database (220) in units of a predetermined batch size.

[0060] FIG. 5 is a flowchart illustrating a method of storing a plurality of data in batch sizes according to one embodiment of the present disclosure.

[0061] Referring to FIG. 5, the information processing system (110) can identify a predetermined batch size (S510).

[0062] Afterward, the information processing system (110) can determine item-specific weights included in each of the multiple document data. Each item-specific weight can be determined in advance. For example, a first weight can be determined for the abstract, a second weight for the keyword, and a third weight for the title. Each item-specific weight is applied when calculating the similarity between the features of the query and the features of the document data, and if an item with a higher weight is similar to the query, the similarity can be calculated to be higher.

[0063] Next, the information processing system (110) can store a plurality of documents corresponding to a batch size in a second database (220) (S530). According to some embodiments, the information processing system (110) can divide the plurality of document data into individual items, and then group each divided item (i.e., fragments) into a predetermined batch size unit and store them in the second database (220) in the form of a batch file.

[0064] Figure 6 is a diagram visualizing document data stored in batch size.

[0065] In Fig. 6, a batch size of 10 is exemplified. In Fig. 6, a rectangle is exemplified as a batch file with a batch size of 10. Item-specific weights can be applied to each batch file.

[0066] When items of documents corresponding to the batch size are stored in the second database (220), the storage speed can also be improved. In addition, when modifications to document data occur, only the corresponding piece (item) is modified, so the speed of modifying data can also be improved.

[0067] FIG. 7 is a flowchart illustrating a method for generating a response to a query according to one embodiment of the present disclosure.

[0068] Referring to FIG. 7, the information processing system (110) can receive a query related to information retrieval from a user terminal (140) (S710). The query may include natural language.

[0069] Afterward, the information processing system (110) can apply the query to an embedding model to obtain a third feature related to the query and search for one or more researcher data related to the third feature in the first database (210). According to some embodiments, the information processing system (110) can search for one or more researcher data having a first feature having a similarity to the third feature and a first threshold value or higher. The similarity between the first feature and the third feature can be calculated through cosine similarity, Euclidean distance, etc.

[0070] Next, the information processing system (110) can search for one or more document data related to one or more retrieved researcher data in the second database (220) (S730). According to one embodiment, a researcher identifier (AID) included in the document data and a researcher identifier (AID) included in the researcher data can serve as a means of linking the two data. That is, the information processing system (110) can search for one or more document data including each researcher identifier of the one or more retrieved researcher data in the second database (220).

[0071] The information processing system (110) can calculate a score for each document data based on the similarity and activity score between the query and the document data (S740). Here, the activity score may be a numerical representation of the influence on the document. The method of calculating the activity score and the document data score will be explained in detail with reference to FIG. 8.

[0072] After that, the information processing system (110) can determine one or more document data to be used to generate a response based on the score of each document data (S750). For example, the information processing system (110) can determine one or more document data having a score greater than or equal to a predetermined score among the searched multiple document data as document data to be used to generate a response. As another example, the information processing system (110) can sort the searched multiple document data in descending order of scores and determine one or more document data included in a predetermined rank among the sorted multiple document data as document data to be used to generate a response.

[0073] Next, the information processing system (110) can generate a response message using one or more determined document data and then transmit the generated response message to a user terminal (140) (S760). According to some embodiments, the information processing system (110) can input the determined document data and a query into a Large Language Model (LLM) and receive a response to the query from the Large Language Model. At this time, the Large Language Model can generate a response to the query by referring to one or more input document data.

[0074] Figure 8 is a diagram to explain step S740 of Figure 7 in more detail.

[0075] Referring to FIG. 8, the information processing system (110) can calculate a research productivity index for each researcher based on a plurality of retrieved document data (S810). The research productivity index is a numerical representation of research productivity, and the research productivity index for a specific researcher may be the number of documents issued by the researcher within a predetermined period. According to some embodiments, the information processing system (110) may select document data having a publication time included in the predetermined period from among the retrieved plurality of document data. Additionally, the information processing system (110) may identify researcher identifiers included in specific document data, count the number of researcher identifiers in the selected document data, and determine the counted number of researcher identifiers as the research productivity index for specific document data.

[0076] After that, the information processing system (110) can calculate a cooperation network indicator based on the retrieved multiple document data (S820). Here, the cooperation network indicator may be a numerical value indicating how actively a researcher conducts cooperative research with other researchers. The cooperation network indicator can be calculated through the following mathematical formula 1.

[0077]

[0078] Here, a specific researcher's Net_score is a collaborative network metric, N_col is the number of researchers who conducted research jointly with that specific researcher, and N_total can be the total number of all researchers.

[0079] The information processing system (110) can determine the identifier of the target researcher to be used for calculating the cooperation network indicator in the retrieved researcher data, and count the number of identifiers of other researchers listed as co-authors with the target researcher in multiple document data. The information processing system (110) can calculate the total number of researchers (N_total) by counting the researcher identifiers stored in the second database (220). The information processing system (110) can determine the value obtained by dividing the total number of researchers (N_total) by the calculated number of co-researchers (N_col) as the cooperation network indicator (Net_socre) of a specific researcher.

[0080] After that, the information processing system (110) may obtain one or more keywords from the retrieved document data, calculate entropy indicating how much the obtained keywords are distributed in other document data, and determine the calculated entropy as an indicator of expertise of the document data (S840). According to some embodiments, the Shannon Entropy formula may be used to calculate the entropy associated with the keywords.

[0081] After that, the information processing system (110) can calculate an activity score for the document based on research productivity indicators, cooperation network indicators, and expertise indicators (S840). Here, weights are applied to each of the research productivity indicators, cooperation network indicators, and expertise indicators, and the weighted research productivity indicators, cooperation network indicators, and expertise indicators are summed to calculate the activity score.

[0082] Next, the information processing system (110) can calculate a document score based on an activity score and a feature similarity (S850). Here, the feature similarity may be the similarity between a third feature of the query and a second feature of the document data. A weight is applied to each of the activity score and the feature similarity, and the weighted activity score and feature similarity are summed to calculate a score for the document.

[0083] For example, let us assume a case where a query is received saying, "Tell me who the researcher is that has written the most papers at the Korea Institute of Science and Technology." In this query, the characteristic of the institution name "Korea Institute of Science and Technology" is obtained, and multiple researcher data related to the characteristic of "Korea Institute of Science and Technology" can be searched in the first database (210). Additionally, a researcher identifier included in each of the searched multiple researcher data is identified, and multiple document data containing at least one of the identified researcher identifiers can be searched in the second database (220). After the score of each of the searched multiple document data is calculated, one or more document data and queries that are included within a certain rank or have a score above a threshold are applied to a large-scale language model, and a response to the query "Tell me who the researcher is that has written the most papers at the Korea Institute of Science and Technology" can be generated.

[0084] Meanwhile, the search for researcher data in the first database can be omitted, and a response to the query can be generated through a search of the second database (220).

[0085] FIG. 9 is a flowchart illustrating a method for generating a response to a query according to another embodiment of the present disclosure.

[0086] Referring to FIG. 9, the information processing system (110) can receive a query related to information retrieval from a user terminal (140) (S910). The query may be composed of natural language.

[0087] Afterward, the information processing system (110) applies the query to an embedding model to obtain a third feature related to the query, and can search for multiple document data related to the third feature in the second database (220) (S920). According to some embodiments, the information processing system (110) can search for one or more document data having a second feature having a similarity to the third feature and a second threshold value or higher. The similarity between the second feature and the third feature can be calculated through cosine similarity, Euclidean distance, etc.

[0088] The information processing system (110) can calculate a score for each document data based on the similarity and activity scores of the query and document data (S930). Since the method of calculating the score for the document data has been explained in detail with reference to FIG. 8, a detailed explanation is omitted.

[0089] After that, the information processing system (110) can determine one or more document data to be used to generate a response based on the score of each document data (S940). For example, the information processing system (110) can determine one or more document data having a score greater than or equal to a predetermined score among the searched multiple document data as document data to be used to generate a response. As another example, the information processing system (110) can sort the searched multiple document data in descending order of scores and determine one or more document data included in a predetermined rank among the sorted multiple document data as document data to be used to generate a response.

[0090] Next, the information processing system (110) can generate a response message using one or more determined document data and then transmit the generated response message to a user terminal (140) (S950). According to some embodiments, the information processing system (110) can input the determined document data and a query into a Large Language Model (LLM) and receive a response to the query from the Large Language Model. At this time, the Large Language Model can generate a response to the query by referring to one or more input document data.

[0091] Hereinafter, the hardware configuration of an exemplary computing system according to some embodiments will be described with reference to FIG. 10. The computing system described with reference to FIG. 10 may refer to the information processing system (110) described above.

[0092] FIG. 10 is an exemplary hardware configuration diagram that can implement a computing system in various embodiments.

[0093] A computing system (1000) according to the present embodiment may include one or more processors (1100), a system bus (1600), a communication interface (1200), a memory (1400) for loading a computer program (1500) executed by the processor (1100), and a storage (1300) for storing the computer program (1500). Only components related to the embodiment are illustrated in FIG. 10. Therefore, a person skilled in the art to which the embodiments of the present specification belong will understand that other general-purpose components may be included in addition to the components illustrated in FIG. 10.

[0094] The processor (1100) can control the overall operation of each component of the computing system (1000). The processor (1100) may be configured to include at least one of a CPU (Central Processing Unit), MPU (Micro Processor Unit), MCU (Micro Controller Unit), GPU (Graphic Processing Unit), or any form of processor well known in the art of this specification. Additionally, the processor (1100) may perform operations for at least one application or program for executing a method / operation according to various embodiments. The computing system (1000) may have two or more processors.

[0095] The memory (1400) stores various data, commands and / or information. The memory (1400) may load one or more programs (1500) from the storage (1300) to execute methods / operations according to various embodiments of the present specification. An example of the memory (1400) may be RAM, but is not limited thereto. The system bus (1600) provides communication functions between components of the computing system (1000).

[0096] The bus (1600) can be implemented as various types of buses, such as an address bus, a data bus, and a control bus. The communication interface (1200) can be connected to a network. The storage (1300) can store one or more computer programs (1500) non-temporarily. The storage (1300) may be configured to include non-volatile memory such as flash memory, a hard disk, a removable disk, or any type of computer-readable recording medium well known in the art to which the embodiments of this specification belong.

[0097] A computer program (1500) may include one or more instructions in which methods / operations according to various embodiments of the present specification are implemented. When the computer program (1500) is loaded into memory (1400), a processor (1100) may perform methods / operations according to various embodiments of the present specification by executing one or more instructions. The computer program (1500) may include instructions for methods according to various embodiments described with reference to FIGS. 1 through 9.

[0098] According to some embodiments, a computer program (1500) may include instructions for performing operations such as receiving a query from a user terminal, searching for a plurality of researcher data related to the characteristics of the query in a first database, searching for a plurality of document data related to the plurality of researcher data in a second database, calculating a score for each of the searched plurality of document data, and determining one or more document data used to generate a response to the query based on the calculated score of the plurality of document data.

[0099] According to some embodiments, a computer program (1500) may include instructions for performing operations such as receiving a query from a user terminal, searching a plurality of document data related to the features of the query in a database, calculating an activity score for each of the plurality of document data, calculating a score for each of the plurality of documents based on the similarity between the features of the query and the features of the document data and the activity score, and determining one or more document data used to generate a response to the query based on the calculated score of the plurality of document data.

[0100] Additionally, the computer program (1500) may be combined with a computing system (1000) and stored in a computer-readable non-transient recording medium to execute each step described in FIGS. 4 through 9. Here, the non-transient recording medium may include RAM, SSD (Solid State Drive) related memory devices, flash memory, etc.

[0101] In some embodiments, the computing system (1000) described with reference to FIG. 10 may be configured using one or more physical servers included in a server farm based on cloud technology such as a virtual machine. In this case, at least some of the components shown in FIG. 10, such as the processor (1100), memory (1400), and storage (1300), may be virtual hardware, and the communication interface (1200) may also be configured as a virtualized networking element such as a virtual switch.

[0102] Various embodiments of the present disclosure and effects according to those embodiments have been described with reference to FIGS. 1 to 10. The effects according to the technical concept of the present disclosure are not limited to those described above, and other unmentioned effects will be clearly understood by a person skilled in the art from the description below.

[0103] The methods according to the embodiments of the present invention described so far may be performed by executing a computer program implemented in computer-readable code. The computer program may be transmitted from a first computing device to a second computing device via a network such as the Internet and installed on the second computing device, thereby being used on the second computing device. Furthermore, although operations are depicted in a specific order in the drawings, it should not be understood that the operations must be executed in the specific order depicted or in a sequential order, or that all depicted operations must be executed to obtain the desired result. In certain situations, multitasking and parallel processing may be advantageous.

[0104] Although embodiments of the present disclosure have been described above with reference to the attached drawings, those skilled in the art will understand that the present invention may be practiced in other specific forms without altering the technical concept or essential features thereof. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. The scope of protection of the present invention shall be interpreted by the claims below, and all technical concepts within the equivalent scope shall be interpreted as being included within the scope of rights of the technical concepts defined by the present disclosure.

Claims

Claim 1 A data retrieval method performed by a computing system comprises: receiving a query from a user terminal; searching for a plurality of researcher data related to the characteristics of the query in a first database; searching for a plurality of document data related to the plurality of researcher data in a second database; calculating a score for each of the searched plurality of document data; and determining one or more document data used to generate a response to the query based on the calculated score of the plurality of document data, wherein the step of calculating a score for each of the searched plurality of document data includes: calculating an activity score for each of the plurality of document data; and calculating a score for each of the plurality of document data based on the activity score, wherein the step of calculating the activity score includes: identifying a researcher identifier included in a specific document data; counting the number of other researcher identifiers listed as co-authors with the researcher identifier in the searched plurality of document data; determining a value obtained by dividing the counted number of other researcher identifiers by the total number of researchers as a cooperative network index; and calculating an activity score for the specific document data based on the cooperative network index. Claim 2 A data retrieval method according to claim 1, wherein the step of calculating a score for each of the searched plurality of document data comprises: a step of calculating a similarity between the features of the query and the features of the plurality of document data; and a step of calculating a score for each of the plurality of document data based on the calculated similarity and the activity score. Claim 3 delete Claim 4 A data retrieval method according to claim 1, wherein the step of calculating an activity score for the specific document data comprises: a step of selecting document data having a publication time included in a predetermined period among the searched plurality of document data; a step of identifying researcher identifiers included in the specific document data and counting the number of researcher identifiers in the selected document data; a step of determining the counted number of researcher identifiers as a research productivity indicator of the specific document data; and a step of calculating an activity score for the specific document data based on the cooperation network indicator and the research productivity indicator. Claim 5 delete Claim 6 A data retrieval method performed by a computing system comprises: receiving a query from a user terminal; searching for a plurality of researcher data related to the characteristics of the query in a first database; searching for a plurality of document data related to the plurality of researcher data in a second database; calculating a score for each of the searched plurality of document data; and determining one or more document data to be used to generate a response to the query based on the calculated score of the plurality of document data, wherein the step of calculating a score for each of the searched plurality of document data includes: calculating an activity score for each of the plurality of document data; and calculating a score for each of the plurality of document data based on the activity score, wherein the step of calculating the activity score includes: identifying a keyword included in specific document data; calculating an entropy indicating how much the keyword is distributed in other document data other than the specific document data; determining the calculated entropy as an expertise index for the specific document data; and calculating an activity score for the specific document data based on the expertise index. Claim 7 A data retrieval method according to claim 1, wherein the step of searching for a plurality of document data in a second database comprises: a step of obtaining a plurality of researcher identifiers included in the plurality of researcher data; and a step of searching for a plurality of document data including at least one of the plurality of researcher identifiers in the second database. Claim 8 A data retrieval method according to claim 1, further comprising, after the step of determining one or more document data used to generate a response to the above query, the step of transmitting a response generated using the determined one or more document data to the user terminal. Claim 9 A data retrieval method according to claim 8, wherein the step of transmitting the generated response to the user terminal includes the step of inputting the query and one or more determined document data into a large-scale language model to obtain the response. Claim 10 A data retrieval method according to claim 1, further comprising, prior to the step of retrieving the plurality of researcher data in the first database, the step of applying document or researcher information to an artificial intelligence model to obtain researcher data including an institution name and a researcher identifier; and the step of storing the obtained researcher data in the first database. Claim 11 A data retrieval method according to claim 1, further comprising, prior to the step of retrieving the plurality of researcher data in the first database, the step of applying the plurality of documents to an artificial intelligence model to obtain the plurality of document data including a title, abstract, and researcher identifier; and the step of storing the obtained plurality of document data in the second database. Claim 12 A data retrieval method according to claim 11, wherein the step of storing the plurality of document data in the second database comprises: the step of dividing the plurality of document data into a plurality of items; and the step of storing the divided plurality of items in the second database in units of a predetermined batch size. Claim 13 A data search method according to claim 12, wherein the step of dividing each of the plurality of document data into a plurality of items includes the step of determining a weight for each of the divided plurality of items, and the weight is reflected when calculating the similarity between the query and the document data. Claim 14 A data retrieval method by a computing system comprises: receiving a query from a user terminal; searching a plurality of document data related to the features of the query in a database; calculating an activity score for each of the plurality of document data; calculating a score for each of the plurality of document data based on the similarity between the features of the query and the features of the document data and the activity score; and determining one or more document data to be used to generate a response to the query based on the calculated score of the plurality of document data, wherein the step of calculating the activity score comprises: identifying a researcher identifier included in a specific document data; counting the number of other researcher identifiers listed as co-authors with the researcher identifier in the searched plurality of document data; determining a value obtained by dividing the counted number of other researcher identifiers by the total number of researchers as a cooperative network index; and calculating an activity score for the specific document data based on the cooperative network index. Claim 15 delete Claim 16 In claim 14, the step of calculating an activity score for the specific document data comprises: a step of selecting document data having a publication time included in a predetermined period among the searched plurality of document data; a step of identifying researcher identifiers included in the specific document data and counting the number of researcher identifiers in the selected document data; a step of determining the counted number of researcher identifiers as a research productivity indicator for the specific document data; and a step of calculating an activity score for the specific document data based on the cooperation network indicator and the research productivity indicator. Claim 17 delete Claim 18 A data retrieval method by a computing system comprises: receiving a query from a user terminal; searching a plurality of document data related to the features of the query in a database; calculating an activity score for each of the plurality of document data; calculating a score for each of the plurality of document data based on the similarity between the features of the query and the features of the document data and the activity score; and determining one or more document data to be used to generate a response to the query based on the calculated scores of the plurality of document data, wherein the step of calculating the activity score includes: identifying a keyword included in specific document data; calculating an entropy indicating how much the keyword is distributed in other document data other than the specific document data; determining the calculated entropy as an expertise indicator for the specific document data; and calculating an activity score for the specific document data based on the expertise indicator. Claim 19 One or more processors; and a memory for storing a computer program executed by the one or more processors, wherein the computer program includes instructions for performing the operation of receiving a query from a user terminal; the operation of searching for a plurality of researcher data related to the characteristics of the query in a first database; the operation of searching for a plurality of document data related to the plurality of researcher data in a second database; the operation of calculating a score for each of the searched plurality of document data; and the operation of determining one or more document data used to generate a response to the query based on the calculated score of the plurality of document data, wherein the operation of calculating a score for each of the searched plurality of document data includes the operation of calculating an activity score for each of the plurality of document data; and the operation of calculating a score for each of the plurality of document data based on the activity score, wherein the operation of calculating an activity score includes the operation of identifying a researcher identifier included in a specific document data; the operation of counting the number of other researcher identifiers listed as co-authors with the researcher identifier in the searched plurality of document data; and the operation of determining a value obtained by dividing the total number of researchers by the counted number of other researcher identifiers as a cooperation network indicator. A computing system comprising an operation to calculate an activity score for the specific document data based on the cooperation network indicators. Claim 20 A computing system comprising: one or more processors; and a memory for storing a computer program executed by the one or more processors, wherein the computer program comprises instructions for performing operations such as: receiving a query from a user terminal; searching a plurality of document data related to the features of the query in a database; calculating an activity score for each of the plurality of document data; calculating a score for each of the plurality of document data based on the similarity between the features of the query and the features of the document data and the activity score; and determining one or more document data used to generate a response to the query based on the calculated score of the plurality of document data, wherein the operation of calculating the activity score comprises: identifying a researcher identifier included in a specific document data; counting the number of other researcher identifiers listed as co-authors with the researcher identifier in the searched plurality of document data; determining a value obtained by dividing the counted number of other researcher identifiers by the total number of researchers as a cooperative network index; and calculating an activity score for the specific document data based on the cooperative network index. Claim 21 A method is combined with a computing system and stored in a computer-readable, non-transient recording medium to execute the steps of: receiving a query from a user terminal; searching for a plurality of researcher data related to the characteristics of the query in a first database; searching for a plurality of document data related to the plurality of researcher data in a second database; calculating a score for each of the searched plurality of document data; and determining one or more document data used to generate a response to the query based on the calculated score of the plurality of document data. The step of calculating a score for each of the searched plurality of document data includes: calculating an activity score for each of the plurality of document data; and calculating a score for each of the plurality of document data based on the activity score. The step of calculating an activity score includes: identifying a researcher identifier included in a specific document data; counting the number of other researcher identifiers listed as co-authors with the researcher identifier in the searched plurality of document data; and determining a value obtained by dividing the total number of researchers by the counted number of other researcher identifiers as a collaborative network indicator. A computer program comprising the step of calculating an activity score for the specific document data based on the cooperation network indicators.

Images