System for analyzing data by using large language model and method for implementing same

Abstract

The present invention relates to a technology pertaining to a system and method for analyzing data by using a large language model, and the system for analyzing data by using a large language model, according to an embodiment of the present invention, comprises: an orchestrator that provides a query message including a natural language query to an LLM server and receives an analysis code from the LLM server; and a code handler that receives the analysis code from the orchestrator and provides an execution result obtained by executing the analysis code to the orchestrator.

Description

A system for analyzing data using a large language model and a method for implementing the same.

[0001] The present invention relates to a system and method for analyzing data using a large language model.

[0002] Recently, various large-scale language models have been developed and are increasingly being used in practice, leading to an increase in the number of companies incorporating them into their operations. However, large-scale language models are separate components from the company's computer system and, as a result, are provided by a third party, posing a security risk.

[0003] Accordingly, some companies are restricting the input of corporate confidential information or data into large language models. However, these restrictions reduce the utility of large language models. Therefore, companies need a way to utilize large language models while maintaining the security of their internal data.

[0004] This specification aims to address the aforementioned issues and improve the accessibility of data analysis by using various large-scale language models.

[0005] Additionally, this specification seeks to enhance data security and confidentiality when applying various large-scale language models.

[0006] In addition, this specification seeks to increase user understanding and improve the efficiency of data analysis and decision-making by applying various schema data including metadata to data analysis results.

[0007] The objectives of the present invention are not limited to those mentioned above. Other objectives and advantages of the present invention not mentioned above can be understood through the following description and will be more clearly understood through the embodiments of the present invention. Furthermore, it will be readily apparent that the objectives and advantages of the present invention can be realized by the means and combinations thereof set forth in the claims.

[0008] A method for analyzing data using a large language model according to one embodiment of the present invention includes a first step in which an orchestrator provides a query message including a natural language query to an LLM server and receives an analysis code from the LLM server, a second step in which the orchestrator provides the analysis code to a code handler and receives an execution result of the analysis code from the code handler, and a third step in which the orchestrator provides the execution result of the analysis code to the LLM server and receives a response message.

[0009] A method for analyzing data using a large language model according to another embodiment of the present invention includes a first step in which an orchestrator provides a query message including a natural language query to an LLM server and receives an analysis code and a response format corresponding to the query message from the LLM server, a second step in which the orchestrator provides an analysis code to a code handler and receives an execution result of the analysis code from the code handler, and a third step in which the orchestrator combines the execution result of the analysis code with the response format to generate a response message.

[0010] A system for analyzing data using a large language model according to one embodiment of the present invention includes an orchestrator that provides a query message including a natural language query to an LLM server and receives analysis code from the LLM server, and a code handler that receives analysis code from the orchestrator and provides the execution result of executing the analysis code to the orchestrator.

[0011] When the present invention is applied, non-expert users can perform data analysis, while at the same time satisfying confidentiality and security requirements necessary for corporate data security.

[0012] When the present invention is applied, the accessibility of data analysis can be improved by using a large language model.

[0013] When the present invention is applied, the security and confidentiality of data can be strengthened when applying various large-scale language models.

[0014] When the present invention is applied, various schema data including metadata can be applied to data analysis results to increase user understanding and improve the efficiency of data analysis and decision making.

[0015] By applying the present invention, the efficiency of a company's decision-making process can be improved based on data.

[0016] The effects of the present invention are not limited to the effects described above, and various effects of the present invention can be easily derived from the configuration of the present invention.

[0017] Figure 1 is a drawing showing the entire components according to one embodiment of the present invention.

[0018] FIG. 2 is a diagram showing a process of analyzing a question entered in natural language and generating a result corresponding to the question according to one embodiment of the present invention.

[0019] Figure 3 is a drawing showing detailed operations according to an embodiment of the present invention.

[0020] FIG. 4 is a diagram showing a process of producing dryer-related information by querying it in natural language according to one embodiment of the present invention.

[0021] FIG. 5 is a diagram showing the configuration of schema data of a database related to a dryer according to one embodiment of the present invention.

[0022] FIG. 6 is a diagram showing a process using a layered structure according to one embodiment of the present invention.

[0023] FIG. 7 is a diagram showing a configuration in which schema data according to one embodiment of the present invention includes authority information.

[0024] FIG. 8 is a diagram showing a process for confirming authority and answering a question according to one embodiment of the present invention.

[0025] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings so that those skilled in the art can easily implement the present invention. The present invention may be implemented in various different forms and is not limited to the embodiments described herein.

[0026] In order to clearly explain the present invention, parts that are not related to the description have been omitted, and the same or similar components are designated by the same reference numerals throughout the specification. In addition, some embodiments of the present invention will be described in detail with reference to exemplary drawings. When adding reference numerals to components in each drawing, the same components may have the same numerals as much as possible even if they are shown in different drawings. In addition, when explaining the present invention, if it is determined that a specific description of a related known configuration or function may obscure the gist of the present invention, the detailed description may be omitted.

[0027] When describing components of the present invention, terms such as first, second, A, B, (a), (b), etc. may be used. These terms are only intended to distinguish the components from other components, and the nature, order, sequence, or number of the components are not limited by the terms. When it is described that a component is "connected," "coupled," or "connected" to another component, it should be understood that the component may be directly connected or connected to the other component, but that other components may also be "interposed" between each component, or that each component may be "connected," "coupled," or "connected" through another component.

[0028] In addition, in implementing the present invention, components may be described in detail for convenience of explanation, but these components may be implemented in one device or module, or one component may be implemented by being divided into multiple devices or modules.

[0029] Recently, research and real-world industrial applications of AI models based on Large Language Models (LLMs) that process natural language input and produce output are on the rise. LLMs (hereafter abbreviated as LLMs) process information entered by the user in natural language and provide corresponding results. Therefore, when users input questions appropriately for LLM processing, the accuracy of the results increases.

[0030] Systems that build databases to store and process diverse data use these databases to retrieve data. However, entering queries to retrieve data stored in the database or analyzing the results requires advanced technical skills. This can be a barrier that reduces the usability of data stored in the database.

[0031] Accordingly, this specification examines a configuration that performs analysis on user input of natural language, such as a large language model, and generates a response including the results in a database.

[0032] Figure 1 is a drawing showing the entire components according to one embodiment of the present invention.

[0033] A client device (1) is a device for entering questions and checking results, and is exemplified by various devices used by users, such as computers, laptops, tablets, and mobile devices. Users can use the client device (1) to enter questions about data and receive and check the corresponding analysis results.

[0034] The front server (100) receives a question composed of a natural language query from a client device (1). The front server (100) can communicate with the client device (1) based on various interfaces.

[0035] The front server (100) receives a question composed in natural language from a client device (1), transmits it to an orchestrator (200), and provides the analysis results returned from the orchestrator (200) to the client device (1). According to one embodiment of the present invention, the front server (100) may provide a chatbot interface to enhance user convenience. The front server (100) may provide a web interface and may also be implemented as a chat app such as Slack or MS Teams.

[0036] For example, if a client device (1) transmits a question using the web, the front server (100) may implement a web interface. Alternatively, if the client device (1) transmits a question using a predetermined application, the front server (100) may be implemented as a server that communicates in response to the application. The front server (100) may provide a query message including a natural language query to the orchestrator (200).

[0037] An orchestrator (200) is, in one embodiment, a hardware component configured independently or software installed on a server. The orchestrator (200) analyzes a user's question, requests data analysis code corresponding to the question from the LLM server (300), interprets the output of the LLM server (300), and executes the analysis code on the code handler (400).

[0038] In one embodiment, the orchestrator (200) provides a query message to the LLM server (Large Language Model, 300). At this time, the orchestrator (200) provides the schema of the database (1000) it holds to the LLM server (300).

[0039] The LLM server (300) generates an analysis code capable of processing a query message and provides it to the orchestrator (200). At this time, the analysis code is an analysis code required to execute a database query (DB query) for the database (1000).

[0040] The LLM server (300) generates appropriate data analysis code based on the user's questions and schema data provided by the Orchestrator (200). The LLM server (300) is, in one embodiment, ChatGPT or another LLM or various natural language processing servers.

[0041] The analysis code is generated by the LLM server (300) to reflect the schema of the database (1000). The orchestrator (200) provides the analysis code to the code handler (400). Thereafter, the code handler (400) applies and executes the analysis code to the database (1000) and provides the results to the orchestrator (200).

[0042] That is, the code handler (400) executes the analysis code provided by the orchestrator (200) and returns the result to the orchestrator (200).

[0043] Then, the orchestrator (200) provides the execution result back to the LLM server (300), and the LLM server (300) provides the orchestrator (200) with a response message corresponding to the previous query message. The orchestrator (200) transmits the response message to the front server (100), and the front server (100) provides it to the client device (1).

[0044] In the configuration of FIG. 1, the front server (100), orchestrator (200), code handler (400), and database (1000) may be configured as internal systems. Additionally, the LLM server (300) may be a separate external system deployed externally. Of course, the LLM package may be included in the internal system.

[0045] The orchestrator (200) can be implemented as a single component including a code handler (400). In addition, the orchestrator (200) can receive a query message directly from a client device (1) without a front server (100).

[0046] In the configuration of FIG. 1, the LLM server (300) does not access data stored in the database (1000), but only uses the schema of the database (1000) to produce results, so the security of the data stored in the database (1000) can be maintained.

[0047] In addition, in the process of checking data stored in the database (1000), even if the user cannot handle DB queries related to the database (1000) or cannot write code, the user can search for data stored in the database (1000) or check the results.

[0048] In the configuration of FIG. 1, in one embodiment of the present invention, the orchestrator (200) and the code handler (400) may be configured as a single system. In another embodiment, the orchestrator (200), the code handler (400), and the DB (1000) may be configured as a single system.

[0049] Alternatively, the front server (100) may be included in one of the aforementioned systems.

[0050] Additionally, the LLM server (300) can be placed externally, separate from the orchestrator (200), to process natural language queries.

[0051] Alternatively, the LLM server (300) can be placed in the same internal system area as the orchestrator (200) to process natural language queries.

[0052] However, since the LLM server (300) cannot access the database (1000) regardless of where the LLM server (300) is located, the security of the data stored in the DB (1000) can be maintained even when the LLM server (300) is used as an external service or when the LLM server (300) is leased.

[0053] Various external companies can provide LLM servers (300). Therefore, a company owning a database (1000) can enhance both the data security and data usability of its database (1000) by incorporating the orchestrator (200) into various LLM services.

[0054] Examples of LLM services include services provided by various companies such as chatCGP, bard, and bing, but the present invention is not limited thereto, and also includes cases where a company purchases and customizes an LLM module.

[0055] The orchestrator (200) effectively processes the user's questions, performs appropriate analysis, provides the results to the user, and maintains the security and confidentiality of the data.

[0056]

[0057] FIG. 2 is a diagram showing a process of analyzing a question entered in natural language and generating a result corresponding to the question according to one embodiment of the present invention.

[0058] A client device (1) transmits a natural language query to a front server (100) (S11). The natural language query is, in one embodiment, text, but may also include non-text data such as voice.

[0059] The front server (100) generates a query message including the received query and transmits it to the orchestrator (200) (S12). Alternatively, in one embodiment, the client device (1) generates a query message including a natural language query or non-text data and then transmits the query message to the front server (100), and the front server (100) then forwards it to the orchestrator (200).

[0060] Additionally, depending on the system structure, the orchestrator (200) can receive a query message from the front server (100) or directly from the client device (1) without the front server (100).

[0061] Thereafter, the orchestrator (200) provides a query message including a natural language query to the LLM server (300) (S13) and receives an analysis code from the LLM server (300) (S15). In this process, the LLM server (300) can generate an analysis code based on the schema of the database (1000), the scope or characteristics of the data stored in the database (1000), the hierarchical structure, etc. (S14).

[0062] Thereafter, the orchestrator (200) provides the analysis code to the code handler (400) (S16). The code handler (400) executes the analysis code to derive results from the database (1000) (S17), and then provides the execution result of the analysis code to the orchestrator (200) (S18).

[0063] The orchestrator (200) provides the execution results of the analysis code to the LLM server (300) (S19). The LLM server (300) generates a response message composed of natural language based on the results (S20) and provides the response message to the orchestrator (200) (S21).

[0064] Thereafter, the orchestrator (200) provides a response message to the front server (100) (S22), and the front server (100) transmits the response message to the client device (1) (S23).

[0065] The embodiment of FIG. 2 can be applied as follows. When analyzing a customer's electronic device usage behavior within a company, a natural language query can be entered without writing a database query or code for the database where the customer's electronic device usage data is stored, thereby extracting information stored in the database and deriving results. Furthermore, the LLM server (300) can provide a response message reflecting the database schema for the information stored in the database.

[0066] As the information processing process of a company that has introduced an IT system is digitized and connected online, the company can accumulate large amounts of data in a database (1000).

[0067] Effectively utilizing this accumulated data can automate business processes, improve the quality of management decisions, and impact product / service planning and development. However, previously, information stored in the database (1000) could only be accessed by skilled database administrators or programmers. In other cases, access was limited to in-house infrastructure tools.

[0068] Additionally, due to the size and complexity of the data, big data processing and data analysis capabilities were required, and due to the strengthening of personal information protection laws, data accessibility for company members was low, so data utilization by field members was limited.

[0069] However, when the orchestrator (200) of the present invention described above is applied, even users without any background knowledge can intuitively perform data analysis through a chatbot-style interface. Furthermore, customer data, which should be maintained and searched internally, can be analyzed without having to be downloaded directly from external sources, thereby enhancing safety and security in terms of personal information and security.

[0070]

[0071] FIG. 3 is a diagram illustrating detailed operations according to an embodiment of the present invention. In addition to the embodiment of FIG. 2, this embodiment is further refined to include data retrieval authority and schema verification.

[0072] S11 and S12 are the same as in Fig. 2 above. Here, an example of a natural language query transmitted by the client device (1) is assumed to be "Tell me the average daily usage time of the dryer."

[0073] The orchestrator (200) requests message analysis from the LLM server (300) (S31). The LLM server (300) analyzes the intent of the message (S32) and provides an intent analysis response message (S33). The intent analysis response message can be structured as follows.

[0074] Intent Analysis Response Message = {"Product=Dryer", "Content:Daily Average Usage"}

[0075] Alternatively, the intent analysis response message can be further refined as follows:

[0076] Intent Analysis Response Message = {"Product=Dryer", "Content:Usage", "Criteria=Average", "Period:Daily"}

[0077] The orchestrator (200) uses the client device's identification information based on the intent analysis response message to determine whether the client device (1) has the authority to retrieve data (S33). Furthermore, the orchestrator (200) determines whether the query message contains the data it seeks to obtain (S35). For example, if the dryer's usage data is not stored, it cannot respond to the query message.

[0078] And if it is confirmed that the data is held, the orchestrator (200) verifies the schema of the database (1000) (S36). Then, the orchestrator (200) requests an analysis code from the LLM server (300) (S37). In this process, the orchestrator (200) can provide the schema of the database (1000) to the LLM server (300).

[0079] Alternatively, if the orchestrator (200) previously provided schema data of the database (1000) to the LLM server (300), the schema data provision process may be omitted. The LLM server (300) performs intent analysis of the query message of S31 or generates analysis code using the analysis results of S32 (S38). Then, the LLM server (300) provides the generated analysis code to the orchestrator (200) (S39).

[0080] The orchestrator (200) verifies the data path of the database (1000) to which the analysis code will be applied (S40) and requests the code handler (400) to execute the code (S41). During the code execution request process, the orchestrator (200) can transmit the analysis code received at S39 and the data path confirmed at S40 to the code handler (400).

[0081] The code handler (400) is not limited to Python, Java, Perl, etc. as an example.

[0082] The code handler (400) verifies whether the analysis code is executable and then executes the analysis code (S42). The analysis code executed by the code handler (400) is the result of executing the natural language query of S11, and the code handler (400) derives a code execution result value. Accordingly, the code handler (400) provides the execution result to the orchestrator (200) (S43).

[0083] When the orchestrator (200) receives the execution result, it goes through a process of preparing to generate a response (S44) so ​​that it can provide the result value as a response, and then requests the LLM server (300) to generate a response (S45). At this time, the orchestrator (200) can provide the query message of S31 and the execution result of S43 together to the LLM server (300) during the response generation request process.

[0084] Alternatively, the orchestrator (200) and the LLM server (300) share identification information for each query message. Accordingly, the LLM server (300) can identify which query message the execution result corresponds to. Alternatively, the orchestrator (200) and the LLM server (300) maintain threads corresponding to query messages, and the orchestrator (200) can request analysis code or response generation through the thread corresponding to a single query message.

[0085] After S45, processes S20 to S23 of Fig. 2 are performed.

[0086] In the aforementioned embodiment, the analysis code is executed in the company's internal analysis / development environment based on the data set of the company's database (1000) by the code handler (400). The generated analysis results are then converted into a user-understandable answer via the LLM server (300) (S45, S20).

[0087] The process of a user asking a question in natural language and providing the resulting analysis results back to the user in text format can be implemented through API communication between the components of the system. However, the present invention is not limited to this. Each component can be implemented in an encapsulated form, and the LLM server (300) can be replaced or deployed in various models as needed.

[0088] When the above-described embodiment is applied, it provides convenience that surpasses existing data analysis methods, allowing users to ask questions in natural language and check the corresponding data analysis results, thereby expanding the possibilities and efficiency of information processing based on natural language.

[0089] The response message to the natural language query “Tell me the average daily usage time of the dryer” presented earlier could be “The average daily usage time of the dryer is 80 minutes.”

[0090] For example, if the execution result provided in S43 is “80”, the LLM server (300) can generate a response message for the result “80” in conjunction with the content of the query message in processes S44 and S45.

[0091] If, for example, the natural language query is “Tell me the number of users who use the dryer every day” and the execution result provided in S43 is “80”, the LLM server (300) can generate a response message in S44 and S45, “80 users use the dryer every day” for the result “80” in connection with the content of the query message.

[0092] That is, the LLM server (300) can understand the query message provided by the orchestrator (200) and automatically generate appropriate analysis code to resolve the query message according to the schema of the data provided by the orchestrator (200). In addition, the orchestrator (200) and the LLM server (300) can share information related to the context, synonyms, and similarity of the query message.

[0093] For example, the orchestrator (200) can provide the LLM server (300) with department information of the client device (1) that sent the query message, identification information of query messages sent by the client device (1) in the past, etc., and the LLM server (300) can increase the accuracy in analyzing the intent of the query message. As a result, data results can be easily confirmed within the enterprise without having to add a description of the data each time.

[0094] In addition, the orchestrator (200) provides the schema of the database (1000) to the LLM server (300) and the data stored in the database (1000) is not shared with the LLM server (300), so the company can maintain the confidentiality of the data it holds.

[0095] In other words, the company's data itself is not directly leaked to external or purchased modules, such as the LLM server (300), thereby preventing the possibility of leaks of the company's original customer or management data. Furthermore, when a user requests analysis, the orchestrator (200) verifies the user's access to the data, thereby controlling members' access to data (such as profit and loss, sales, and sales volume) that must be kept secure even within the company.

[0096] In particular, after providing a query message to the LLM server (300), the orchestrator (200) can receive keywords for the query message from the LLM server (300) (S33). In this process, the orchestrator (200) verifies whether the client device (1) has the authority to correspond to the keyword, and if so, requests and receives an analysis code corresponding to the query message from the LLM server (300) (S37 to S39).

[0097] For example, if a client device (1) has only basic inquiry rights related to an air conditioner, and the client device (1) requests information related to a dryer, the orchestrator (200) may determine that the client device (1) does not have the rights.

[0098] Below we will look at the process of processing query data for dryers.

[0099] First, let's make two assumptions about the database tables related to the dryer.

[0100] The structure of the table "Dryer_Info", which contains information about the dryer's identification information, information about the dryer user, the city or country where the dryer is located, and the dryer model, is as shown in Table 1.

[0101] D_Machine_IDUser_AgeCity_LocCountry_LocMachine_ModelVersion

[0102] The columns that make up the Dryer_Info table are described below. These descriptions are also metadata and can constitute the schema data of the database.

[0103] D_Machine_ID identifies the dryer, and User_Age identifies the dryer user's age. City_Loc identifies the city where the dryer is located, and Country_Loc identifies the country where the dryer is located. Machine_Model identifies the dryer's model name, and Version indicates the software version installed on the dryer.

[0104] Meanwhile, the configuration of the table "Dryer_Usage" that generates data for each dryer use is as shown in Table 2.

[0105] D_Machine_IDStart_TENd_TDurationCourseLevelErrorCode

[0106] The columns that make up the Dryer_Usage table are described below. These descriptions are also metadata and can constitute the schema data of the database.

[0107] D_Machine_ID is the identification information of the dryer, Start_T is the time when the dryer started operating, End_T is the time when the dryer ended operating, Duration is the time it took for the dryer to perform one operation, Course and Level are the operation course and intensity of the dryer, and ErrorCode is the error code that occurred during the dryer operation.

[0108] The orchestrator (200) can store the names of each table in Tables 1 and 2, metadata about the information stored in the tables, and the format and information of the data stored in each table in advance as a schema of the database and provide the schema of the database to the LLM server (300).

[0109] Unlike FIG. 2, if the LLM server (300) provides a response format, such as that shown in FIG. 4, which will be described later, the security of the analysis results can be maintained. Accordingly, the orchestrator (200) may request the LLM server (300) to generate a response message in the same manner as FIG. 2, or request the LLM server (300) to provide a response format, depending on whether the result of the query message is secure.

[0110] The response format, as an example, configures the text of the response, which includes the results of executing the analysis code, to indicate areas where the results may be included using special characters. For example, the results of executing the analysis code may be indicated to be included in variable names between the "%" characters.

[0111] When applying the aforementioned embodiment, the orchestrator (200) and the LLM server (300) can process a user's question and generate analysis code. The LLM server (300) can parse and understand the user's question and generate data analysis code that answers the user's question using the schema data of the database provided by the orchestrator (200).

[0112] The orchestrator (200) does not provide the entire database, but only provides the LLM server (300) with schema data including information about the tables that make up the database, information about the columns (fields) that make up the tables, and metadata about these, so that the LLM server (300) can generate analysis code without accessing internal data.

[0113] The orchestrator (200) analyzes the user's question and requests the LLM server (300) to generate analysis code. The orchestrator (200) processes the analysis code returned by the LLM server (300) by executing it directly or through a code handler (400). The orchestrator (200) can then return the execution results of the analysis code to the user.

[0114]

[0115] Figure 4 is a diagram illustrating a process for obtaining dryer-related information through a natural language query according to one embodiment of the present invention. Since the processes of Figures 2 and 3 are applied, Figure 4 will focus on the information shared between the orchestrator (200) and the LLM server (300).

[0116] The orchestrator (200) provides the LLM server (300) with database schema data (S50). The orchestrator (200) can provide the LLM server (300) with database schema data including the names of the dryer-related tables in Tables 1 and 2, the names of each column (field) constituting the tables, and metadata regarding the values ​​stored in each column.

[0117] Thereafter, when a client device (1) inputs a query such as "What columns are there in the dryer data?", the query is transmitted directly to the orchestrator (200) with or without passing through the front server (100) (S51). The orchestrator (200) provides the query of S51 to the LLM server and requests message analysis (S52).

[0118] Then, the LLM server (300) analyzes the intent of the message (S53) and generates an intent analysis response message querying information constituting a database related to the dryer, and provides the message to the orchestrator (200) (S54). This process may correspond to S13 to S21 of FIG. 2 or S31 to S45 and S20, S21 of FIG. 3.

[0119] If a response is performed on the schema data of the database, the LLM server (300) can produce a result using the schema data provided in step S50 without generating a separate analysis code.

[0120] An example of an intent analysis response message may be a listing of the names and column information of the dryer tables provided in S50 above. The intent analysis response message may be "Dryer data includes "D_Machine_ID, User_Age, City_Loc, Country_Loc, Machine_Model, Version, Start_T, End_T, Duration, Course, Level, ErrorCode." Then, the orchestrator (200) transmits this message to the client device (1) or the front server (100). As a result, the client device (1) can check the columns of the dryer data as in S55.

[0121] Afterwards, when the client device (1) inputs a query such as “Please tell me the number of uses per course,” the query is transmitted directly to the orchestrator (200) with or without going through the front server (100) (S56).

[0122] The LLM server (300) generates a DB query (analysis code) that counts the number of uses per course in the table in Table 2 and generates a response format to include the results (S58). For example, the analysis code executes a DB query statement that counts by "Course" in the "Dryer_Usage" table, as one example.

[0123] For example, the orchestrator (200) may provide {"BEDDING", "COTTON", "CARE", "AI"} for the value of "Course" in the schema data above. In this case, the LLM server (300) may generate four analysis codes (Code1, Code2, Code3, Code4).

[0124] Code1= "SELECT COUNT (*) as cnt1 FROM Dryer_Usage WHERE Course = "BEDDING"

[0125] Code2= "SELECT COUNT (*) as cnt2 FROM Dryer_Usage WHERE Course = "COTTON"

[0126] Code3= "SELECT COUNT (*) as cnt3 FROM Dryer_Usage WHERE Course = "CARE"

[0127] Code4= "SELECT COUNT (*) as cnt4 FROM Dryer_Usage WHERE Course = "AI"

[0128] In one embodiment, the response format comprises a response message containing the number of responses corresponding to each response. For example, the LLM server (300) can generate the response format as follows.

[0129] "The number of uses per course is as follows: BEDDING: %cnt1%, COTTON: %cnt2%, CARE: %cnt3%, AI: %cnt4%"

[0130] The LLM server (300) provides the analysis code and response format to the orchestrator (200) (S59). The orchestrator (200) transmits the analysis code to the code handler, receives the execution results, and applies the results to the response format to generate a response message (S60). For example, assume that the code handler (400) executes Code1, Code2, Code3, and Code3 4, and the returned results are 35832 for cnt1, 13592 for cnt2, 41372 for cnt3, and 9870 for cnt4.

[0131] The orchestrator (200) inputs the results previously returned in the response format, generates a response message as follows, and transmits it to the client device (1) or the front server (100). As a result, the client device (1) can display the number of uses of each course of the dryer, as in S61.

[0132] "The number of uses per course is as follows: BEDDING: 35832, COTTON: 13592, CARE: 41372, AI: 9870"

[0133] If S57 to S61 of Fig. 4 are classified in detail, the orchestrator (200) provides a query message including a natural language query to the LLM server (300) and receives an analysis code and a response format corresponding to the query message from the LLM server (300) (S57, S58, S59).

[0134] The process includes a process (S60) in which the orchestrator (200) provides analysis code to the code handler (400) and receives the execution result of the analysis code from the code handler (400), and then the orchestrator (200) combines the execution result of the analysis code with the response format to generate a response message.

[0135] When using the response format, the process of the orchestrator (200) requesting a response message according to the execution result from the LLM server (300) (S19 to S21 of FIG. 2 or S44, S45, S21 of FIG. 3) may be included in S58.

[0136] The embodiment of FIG. 4 illustrates the results of a user query counting the types or number of data stored in a database. This embodiment can be applied to processing a client device (1) asking about the amount of time a user uses a dryer based on their age, or asking about the number of times the user uses the dryer based on their city or country of residence.

[0137] Additionally, Tables 1 and 2 are tables regarding dryers, and the database (1000) may maintain tables regarding air conditioners, washing machines, dishwashers, etc. in addition to dryers. In this case, the client device (1) may inquire about the number of users who own both air conditioners and dryers, and the orchestrator (200) may provide an answer thereto.

[0138] As shown in Fig. 4, when the LLM server (300) dynamically generates analysis code and the code handler (300) executes the analysis code to provide an answer to a user's question, the execution result of the analysis code can be an answer corresponding to the user's question. This answer is included in a response message and transmitted to the client device (1), and the user can check the data analysis results.

[0139] Applying the aforementioned examples allows corporate employees to perform data analysis without complex statistical techniques, programming knowledge, or database queries. This increases the analysis and usability of data accumulated within the company.

[0140] Additionally, since the orchestrator (200) provides only the database schema data to the LLM server (300), the data is protected within the company's system. Consequently, analysis can be performed without exposing sensitive corporate data to external parties, allowing the LLM service to be used while maintaining data security and confidentiality.

[0141] In the embodiment of Figure 4, a user can ask questions about data in natural language. Users can access the chatbot and ask questions about the data, and these questions can take a variety of forms. Users can request analysis to resolve questions about specific data, identify relationships between data, or identify specific patterns. These requests are delivered to the orchestrator (200) in the form of a query message containing a natural language question.

[0142] The orchestrator (200) and the LLM server (300) process the query and generate analysis code. The LLM server (300) determines the intent and meaning of the user's query and, based on this, generates appropriate analysis code tailored to the data. During this process, the orchestrator (200) provides only schema data for the database to the LLM server (300) and does not provide the LLM server (300) with the actual stored data (the actual stored values). As a result, data security and confidentiality can be maintained.

[0143] The LLM server (300) generates analysis codes that respond to user queries using only schema data (columns and descriptions of each column) that defines or describes each table and column that constitutes the database, as well as their relationships and access rights. The orchestrator (200) communicates with the LLM server (300) and provides the analysis codes provided by the LLM server (300) to the code handler (400) to obtain code execution results.

[0144] The analysis code generated by the LLM server (300) is executed in an in-house analysis / development environment based on a dataset stored in a database within the company. The code handler (400) executes the analysis code to derive analysis results tailored to the user's requirements.

[0145] The analysis results can be processed into a user-understandable format, and the LLM server (300) can generate a response message containing the results. Alternatively, the orchestrator (200) can insert the results produced by the code handler (400) into the response format provided by the LLM server (300) to generate a response message.

[0146] The orchestrator (200) provides the analysis code generated by the LLM server (300) to the code handler (400), and the code handler (400) provides the result of executing the analysis code to the client device (1).

[0147] In one embodiment of a result delivery method, the orchestrator (200) provides the results of executing the analysis code to the LLM server (300). The LLM server (300) converts the analysis results into a response message in a form understandable to the user. The response message is provided to the client device (1) via a chatbot interface or the like. This allows the user to obtain an answer to the original question.

[0148] In addition, as another embodiment of the result delivery method, when the LLM server (300) provides the analysis code and the response format to the orchestrator (200), the orchestrator (200) can use this to input the actual results of the analysis code into the response format to generate a response message.

[0149] In an embodiment of the present invention, the orchestrator (200) can secure data security and confidentiality by providing only the schema (schema data) that constitutes the database to the LLM server (300) without providing actual data. Since the LLM server (300) is capable of responding based on the schema data regarding the structure and characteristics of tables and columns that constitute the database, the code handler (400) can generate the analysis code necessary to extract information from the database.

[0150]

[0151] FIG. 5 is a diagram showing the configuration of schema data of a database related to a dryer according to one embodiment of the present invention.

[0152] The schema data generated by the orchestrator (200) for Table 1 above is as shown in 65. The orchestrator (200) includes the table name as "Dryer_Info" and the column names as "D_Machine_ID, User_Age, City_Loc, Country_Loc, Machine_Model, Version" and descriptions for each column in the "ColumnName" and "ColumnInfo" items, respectively.

[0153] Additionally, the orchestrator (200) includes examples of values ​​that can be stored in the "Country_Loc" column in the Country_Loc DataRange item. As illustrated in FIG. 5, the data that can be entered in the "Country_Loc" column are two alphabetic characters indicating each country. The orchestrator (200) includes country names corresponding to each alphabetic character in the schema data as follows.

[0154] "KR: Korea | US: United States | CN: China | JP: Japan | GB: United Kingdom | FR: France | AU: Australia | DE: Germany"

[0155] If a country is added, the orchestrator (200) can update the schema data and provide it to the LLM server (300).

[0156] Additionally, the orchestrator (200) can include a hierarchical structure of the Country_Loc value in the schema data. For example, "Dryer_Info" only has columns for countries, but since queries may arise by continent in the future, the orchestrator (200) hierarchizes the "Country_Loc" value.

[0157] This is presented in the "Country_Loc Hierarchy" entry, and the codes for countries included in Asia are "KR, CN, JP" and those included in Europe are "GB, FR, DE".

[0158] The orchestrator (200) can provide schema data (65) configured as shown in Fig. 5 to the LLM server (300). That is, the schema data can include a description of each column constituting the table, or a range or value of data that can be stored in each column.

[0159] Additionally, schema data may include a hierarchical structure of data stored in columns, as described in "Country_Loc Hierarchy".

[0160] FIG. 6 is a diagram showing a process using a layered structure according to one embodiment of the present invention.

[0161] The orchestrator (200) provides the schema data (65) of FIG. 5 to the LLM server (300) (S70). Thereafter, when the client device (1) enters a query such as "Please tell me the number of dryers installed in Europe," the query is transmitted directly to the orchestrator (200) with or without passing through the front server (100) (S71). The orchestrator (200) provides the query message of S71 to the LLM server (300) (S72).

[0162] The LLM server (300) uses schema data (65) to generate a DB query (analysis code) that provides information on countries within Europe and the number of dryers installed in those countries as results, and generates a response format to include the results (S73). For example, the analysis code executes a DB query statement regarding the number of dryers in the "Dryer_Info" table where "Country_Loc" corresponds to "GB," "FR," and "DE," as one example.

[0163] For example, the LLM server (300) can generate an analysis code, Code, using schema data (65).

[0164] Code= "SELECT COUNT (*) as cnt FROM Dryer_Info WHERE Country_Loc = "GB" OR Country_Loc = "FR" OR Country_Loc = "DE"

[0165] And the LLM server (300) can generate the answer format as follows.

[0166] "The number of dryers installed in Europe is %cnt%."

[0167] The LLM server (300) provides the analysis code and response format to the orchestrator (200) (S74). The orchestrator (200) transmits the analysis code to the code handler, receives the execution results, and applies the results to the response format to generate a response message (S75). For example, assume that the code handler (400) executes the code and returns a result with a cnt value of 3215.

[0168] The orchestrator (200) inputs the results previously returned in the response format, generates a response message as follows, and transmits it to the client device (1) or front server (100). As a result, the client device (1) can display a message such as "The number of dryers installed in Europe is 3215" as in S76.

[0169]

[0170] Figure 7 is a diagram showing a configuration in which schema data according to one embodiment of the present invention includes authorization information. Schema data (80) lists each table name (Dryer_Info, Dryer_Usage) and individual columns of the corresponding table.

[0171] Schema data (80) presents each user's access rights for each table in Table_User_Grade. If the user's authority level is high / middle / low, Dryer_Info has a "low" level and Dryer_Usage has a "middle" level. In addition, the orchestrator (200) can additionally set the level of access rights for specific columns in the table in the schema data (80). In Fig. 7, access to the "Course" column of Dryer_Usage is set to a "high" level.

[0172] Accordingly, the orchestrator (200) can provide the LLM server (300) with grade information corresponding to the identification information of the user logged in to the client device (1) by including it in a query message.

[0173] The LLM server (300) compares the user's grade information with the schema data, and if the user attempts to access information not assigned to him / her, a response message such as "You do not have permission to ask that question" may be output.

[0174] The embodiment of FIG. 7 is an example in which schema data is set including access rights to tables or fields that constitute a database.

[0175] FIG. 8 is a diagram showing a process for confirming authority and answering a question according to one embodiment of the present invention.

[0176] The orchestrator (200) provides the schema data (80) of FIG. 7 to the LLM server (300) (S81). Thereafter, when the client device (1) enters a query such as "Tell me the most frequently used dryer course," the query is transmitted directly to the orchestrator (200) with or without passing through the front server (100) (S82). The orchestrator (200) provides the query message of S82 to the LLM server (300) (S83).

[0177] The LLM server (300) uses schema data (80) to compare and verify the user's authority with the course authority (S84). If the user's authority is "low," it is lower than the "high" authority for the course column, and therefore the user who asked the question in S82 does not have the authority to verify the information.

[0178] That is, the LLM server (300) uses schema data (80) to check the authority and provides a response indicating that there is no authority (S85).

[0179] The orchestrator (200) provides the answer back to the client device (1), and as a result, the client device (1) may display a message such as "You do not have permission to ask that question", as in S86.

[0180] In one embodiment of the present invention, the orchestrator (200) may extract information about the database (1000) and convert it into a form that the LLM server (300) can understand.

[0181] For example, in addition to the embodiment of FIG. 5, the orchestrator (200) can generate sentences such as the following:

[0182] "The table named 'Dryer_Info' has six columns."

[0183] "The names of the columns that make up the table called 'Dryer_Info' are D_Machine_ID, User_Age, City_Loc, Country_Loc, Machine_Model, and Version."

[0184] " D_Machine_ID is the identification information of the dryer, User_Age is the age information of the dryer user. City_Loc is the identification information of the city where the dryer is located, and Country_Loc is the identification information of the country where the dryer is located. Machine_Model is the model name of the dryer, and Version is the version of the software installed on the dryer."

[0185] In this way, the orchestrator (200) can generate schema data in the form of natural language, and this information is transmitted to the LLM server (300) and used in the process of the LLM server (300) generating an analysis code that resolves the user's question.

[0186] Applying the aforementioned embodiments allows non-expert users to perform data analysis while simultaneously satisfying corporate data security requirements, such as confidentiality and security. Furthermore, since the results are provided in natural language, similar to chatbots, users can quickly and easily obtain analysis results through the chatbot interface without having to undergo complex analysis processes.

[0187] Applying the aforementioned embodiments can increase the accessibility of data analysis using large-scale language models. Implementing the system and method of the present invention facilitates the data analysis process, enabling many people to perform data analysis using natural language. This helps non-specialists and non-data professionals leverage data insights to add value to their work. Various departments can access data accumulated within the company and make data-driven decisions.

[0188] Applying the aforementioned embodiments can enhance data security and confidentiality when applying various large-scale language models. When implementing the system and method of the present invention, the orchestrator (200) provides only the data schema to the LLM server (300), thereby maintaining the security of a company's critical data and preventing the external leakage of sensitive information requiring confidentiality. As a result, companies can more safely utilize large-scale language models for data analysis.

[0189] Applying the aforementioned embodiments can enhance user understanding and improve the efficiency of data analysis and decision-making by applying various schema data, including metadata, to data analysis results. Implementing the system and method according to the present invention can translate data analysis results into user-friendly natural language and provide them. This allows users to more easily understand and utilize the analysis results. This reduces the time it takes for users to understand the analysis results, enabling them to make more informed decisions more quickly.

[0190] Applying the aforementioned embodiments can enhance the efficiency of a company's data-based decision-making process. Implementing the system and method of the present invention provides a user-friendly interface and immediate analysis results, enhancing the efficiency of information-based decision-making within a company. This increases the speed and accuracy of the company's decision-making process, thereby enhancing the company's overall productivity and efficiency. Consequently, companies can utilize data analysis more easily, safely, and effectively.

[0191] In particular, the results of applying analysis code to a database and executing it can be included in a response message generated in a user-friendly format according to the user's query. As illustrated in FIGS. 2 and 3 , the orchestrator (200) provides the response result to the LLM server (300), and the response message can be received from the LLM server (300).

[0192] Alternatively, as seen in FIGS. 4 and 6, the orchestrator (200) may receive a response format from the LLM server (300), and the orchestrator (200) may apply the analysis code and include the executed result in the response format to generate a response message.

[0193] Although all components constituting the embodiments of the present invention have been described as being combined or operating in combination, the present invention is not necessarily limited to such embodiments, and within the scope of the present invention, all components may be selectively combined and operated one or more times. In addition, although all of the components may be implemented as individual independent hardware, some or all of the components may be selectively combined and implemented as a computer program having program modules that perform some or all of the functions combined in one or more hardware pieces. The codes and code segments constituting the computer program may be easily inferred by those skilled in the art of the present invention. Such a computer program may be stored in a computer-readable storage medium and read and executed by a computer, thereby implementing the embodiments of the present invention. Storage media for computer programs include magnetic recording media, optical recording media, and storage media including semiconductor recording devices. In addition, a computer program implementing an embodiment of the present invention includes a program module that is transmitted in real time through an external device.

[0194] While the above description focuses on specific embodiments of the present invention, various modifications and variations can be made within the scope of those skilled in the art. Therefore, it should be understood that such modifications and variations are within the scope of the present invention, as long as they do not depart from its scope.

[0195] Description of the symbol -

[0196] 100: Front-end server 200: Orchestrator

[0197] 300: LLM Server 400: Code Handler

Claims

1. A first step in which an orchestrator provides a query message including a natural language query to an LLM server and receives an analysis code from the LLM server; A second step in which the orchestrator provides the analysis code to the code handler and receives the execution result of the analysis code from the code handler; and A method for analyzing data using a large language model, comprising a third step in which the orchestrator provides the execution result of the analysis code to the LLM server and receives a response message.

2. A first step in which the orchestrator provides a query message including a natural language query to the LLM server and receives an analysis code and a response format corresponding to the query message from the LLM server; A second step in which the orchestrator provides the analysis code to the code handler and receives the execution result of the analysis code from the code handler; and A method for analyzing data using a large language model, comprising a third step in which the orchestrator combines the execution result of the analysis code with the response format to generate a response message.

3. In either paragraph 1 or paragraph 2, A method for analyzing data using a large language model, wherein the query message is generated by a client device, and further comprises a step of receiving the query message through the client device or a front server connected to the client device prior to the first step.

4. In paragraph 3, The above first step is A step in which the orchestrator provides the query message to the LLM server and then receives a keyword for the query message from the LLM server; and A method for analyzing data using a large language model, comprising the step of: the orchestrator checking whether the client device has the authority corresponding to the keyword, and if so, requesting and receiving an analysis code corresponding to the query message from the LLM server.

5. In either paragraph 1 or paragraph 2, A method for analyzing data using a large language model, wherein the orchestrator further comprises a step of providing schema data of a database to the LLM server.

6. In paragraph 5, A method for analyzing data using a large language model, wherein the schema data includes the names of tables constituting the database and columns constituting the tables.

7. In paragraph 6, The above schema data includes a description of the column or a range of data that can be stored in the column, A method for analyzing data using a large language model, wherein the schema data includes a hierarchical structure of data stored in the column.

8. In paragraph 6, A method for analyzing data using a large language model, wherein the schema data includes access rights to tables or fields constituting the database.

9. An orchestrator that provides a query message containing a natural language query to an LLM server and receives an analysis code from the LLM server; and A system for analyzing data using a large language model, including a code handler that receives the analysis code from the orchestrator and provides the execution result of executing the analysis code to the orchestrator.

10. In paragraph 9, A system for analyzing data using a large language model, in which the orchestrator provides the execution result of the analysis code to the LLM server and receives a response message.

11. In paragraph 9, In the process of providing the above analysis code to the above orchestrator, the LLM server also provides a response format corresponding to the above query message. A system for analyzing data using a large language model, wherein the orchestrator combines the execution result with the response format to generate a response message.

12. In paragraph 9, A system for analyzing data using a large language model, wherein the query message is generated by a client device and the orchestrator receives the query message through the client device or a front server connected to the client device.

13. In paragraph 12, A system for analyzing data using a large language model, wherein the orchestrator provides the query message to the LLM server, and then receives a keyword for the query message from the LLM server, and then the orchestrator verifies whether the client device has the authority to correspond to the keyword, and if so, requests and receives an analysis code corresponding to the query message from the LLM server.

14. In paragraph 9, The above orchestrator is a system that analyzes data using a large language model, providing schema data of the database to the LLM server.

15. In paragraph 14, A system for analyzing data using a large language model, wherein the schema data includes the names of the tables constituting the database and the columns constituting the tables.

16. In paragraph 15, The above schema data includes a description of the column or a range of data that can be stored in the column, A system for analyzing data using a large language model, wherein the schema data includes a hierarchical structure of data stored in the column.

17. In paragraph 15, A system for analyzing data using a large language model, wherein the schema data includes access rights to tables or fields that constitute the database.

Images