Method and device for data conversion and computer program

Abstract

To provide a data conversion method, device, and program.SOLUTION: A method includes: defining a first large language model agent and a second large language model agent as an agent system by transmitting at least one instruction to at least one large language model; receiving a first structured data file and a question set provided by a user; providing the agent system with an instruction for generating a second structured data file on the basis of the first structured data file and the question set; and obtaining a verified second structured data file of the second large language model agent, the second structured data file being capable of answering all questions in the question set from the agent system.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to data processing, and in particular to a method for transforming data, an apparatus for transforming data, and a computer program product. [Background technology]

[0002] Task-driven automated data transformation is an advanced technology that combines cutting-edge technologies in fields such as data mining, natural language processing (NLP), and machine learning to improve the efficiency and accuracy of data processing. The core of this technology is understanding complex data structures and user needs, and then automatically transforming data accordingly to meet specific analytical and business goals. The data to be transformed can come from a variety of technical fields, such as transportation, sales, industrial production, public opinion, environmental monitoring, remote sensing, meteorology, finance, and social media.

[0003] Data mining techniques aim to extract useful information and patterns from large datasets. These techniques involve processes such as, but not limited to, classification, regression, clustering, and association rule mining. All of these processes are powerful tools for recognizing the underlying structure of data. For example, clustering algorithms can automatically group data into sets with similar characteristics, which is crucial for understanding the essential characteristics of the data and for subsequent data transformation. Association rule mining can provide rule-based guidance for data transformation by revealing interesting relationships between data attributes.

[0004] NLP is a technology that enables computers to understand, interpret, and generate human language. It includes a series of subfields, such as grammatical analysis, semantic analysis, and sentiment analysis, all of which are essential technologies for automated data transformation. By applying NLP, computers can understand queries and commands in natural language and then perform corresponding transformations and processing on data based on this. For example, semantic analysis helps systems understand the meaning of queries, ensuring that the results of data transformation match the user's intent.

[0005] Machine learning algorithms enable computers to learn from data and improve their performance. In the context of automated data transformation, machine learning can be used to guide the data transformation process by recognizing patterns and structures in the data. In particular, large-scale language models such as Generative Pre-Trained Transformer (GPT) and Bidirectional Encoder Representations from Transformers (BERT) have proven powerful in text understanding and generation by being trained on large amounts of text data. The emergence of these models opens up new possibilities for the automated processing and transformation of text-based data, making the extraction of structured information from text data more effective and accurate.

[0006] Modern data transformation methods are evolving toward a more integrated and intelligent approach, which requires a comprehensive framework structure that closely integrates technologies such as data mining, NLP, and machine learning. Such a comprehensive framework can not only understand complex data and user queries, but also automatically train and optimize the data transformation process. This approach allows the system to adaptively improve its performance, providing users with more accurate and personalized data processing services.

[0007] By combining these technologies, task-driven data automation methods can play an important role in various fields, such as financial analysis, market research, and social sciences. Some fields have particularly high demands for data processing and analysis. Furthermore, as the amount and types of data continue to increase, such automation methods will become even more important in order to provide more powerful and flexible support for data-driven decision-making.

[0008] In summary, task-driven data automatic transformation methods are an interdisciplinary field that brings together various advanced technologies and aims to improve the efficiency and intelligence level of data processing. With the advancement of technology and the expansion of application fields, such methods can be expected to play a more important role in future data processing and analysis.

[0009] Existing AI language models are inadequate in terms of data transformation. For example, when a user asks a question about an input data set, the model may output an "error" due to issues such as data format, data type, or data duplication. In other words, the model may notify the user that an error has occurred and that the question cannot be answered. Summary of the Invention [Problem to be solved by the invention]

[0010] The inventor has conducted research and experiments on automatic data conversion using existing AI (Artificial Intelligence) models and proposed the solution of the present invention, which improves the automation level of data conversion and the quality of converted data, ensuring that the converted data fully meets the needs of users. [Means for solving the problem]

[0011] According to one aspect of the present invention, there is provided a method of data transformation, the method comprising: The first large-scale language model agent and the second large-scale language model agent are defined as an agent system by providing two instructions to at least one large-scale language model, wherein the two instructions are: causing the agent system to generate, through cooperation between the first large scale language model agent and the second large scale language model agent, a second structured data file based on the first structured data file that can answer all questions in the question set, including questions about the first structured data file; A first large scale language model agent is defined to perform the following data transformation task: extract, from the first structured data file, data to answer all questions in the question set as a second structured data file if the unanswered question information indicates that there are unanswered questions; and A second large-scale language model agent is defined to perform the following verification task: updating the unanswered question information by verifying whether each question in the question set can be answered based on the second structured data file; It is configured as follows: receiving a first structured data file and a set of questions provided by a user; providing instructions to the agent system for generating a second structured data file based on the first structured data file and the question set; and Obtaining from the agent system a second structured data file that has been validated by the second large scale language model agent and that is capable of answering all questions in the question set.

[0012] According to another aspect of the present invention, there is provided an apparatus for transforming data, the apparatus including: a memory storing instructions; and at least one processor coupled to the memory, the processor configured to execute the instructions to implement the aforementioned method of data transformation.

[0013] According to another aspect of the present invention, there is provided a non-transitory computer-readable storage medium storing a program, which, when executed by a computer, causes the computer to perform the following operations (steps): The first large-scale language model agent and the second large-scale language model agent are defined as an agent system by providing two instructions to at least one large-scale language model, wherein the two instructions are: causing the agent system to generate, through cooperation between the first large scale language model agent and the second large scale language model agent, a second structured data file based on the first structured data file that can answer all questions in the question set, including questions about the first structured data file; A first large scale language model agent is defined to perform the following data transformation task: extract, from the first structured data file, data to answer all questions in the question set as a second structured data file if the unanswered question information indicates that there are unanswered questions; and A second large-scale language model agent is defined to perform the following verification task: updating the unanswered question information by verifying whether each question in the question set can be answered based on the second structured data file; It is configured as follows: receiving a first structured data file and a set of questions provided by a user; providing instructions to the agent system for generating a second structured data file based on the first structured data file and the question set; and A second structured data file is obtained from the agent system that has been validated by the second large scale language model agent and is capable of answering all questions in the question set.

[0014] According to another aspect of the present invention, there is provided a computer program which, when executed on a computer, causes the computer to carry out the method for data conversion described above. [Effects of the Invention]

[0015] The advantageous effects of the method, apparatus, and computer program of the present invention include at least one of the following: improving the automation level of data conversion and improving the quality of converted data, so that the converted data can fully meet the needs of users, increasing the efficiency of data conversion, improving the accuracy of data conversion, and improving the effectiveness and reliability of data conversion. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is an exemplary flowchart of a data transformation method according to one embodiment of the present invention. [Figure 2] 1 illustrates an interaction scenario of a data transformation method according to one embodiment of the present invention; [Figure 3] 1 is an exemplary block diagram of a data conversion device according to one embodiment of the present invention. [Figure 4] 1 is an exemplary block diagram of an information processing device according to one embodiment of the present invention. [Figure 5] 1 is an exemplary block diagram of a data conversion device according to one embodiment of the present invention; [Figure 6] 10 is a screenshot of the interactive interface when ChatGPT cannot successfully answer an example question. [Figure 7] A screenshot of the interactive interface when ChatGPT successfully answers an example question. DETAILED DESCRIPTION OF THE INVENTION

[0017] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings. Note that the following embodiments are merely illustrative and are not intended to limit the scope of the present invention.

[0018] Computer program code for carrying out operations of aspects of embodiments of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, Smalltalk, C++, etc., as well as conventional procedural programming languages ​​such as the C programming language or similar programming languages.

[0019] The method of the present invention can be implemented by circuits having corresponding functional configurations, including circuits for a processor.

[0020] According to one aspect of the present invention, a data conversion method is provided. The method can be implemented by a computer. The inventors studied and tested the ability of existing large language models (LLMs) to extract desired data (i.e., converted data) from structured data files. The tests revealed that, while the LLMs were sometimes able to extract desired data from a file based on a given question and answer the relevant question, they were also sometimes unable to answer (failed to answer) (i.e., the answer to the given question included an error message indicating that no answer to the question could be found). Particularly when the data file is complex, even if the meaning of the elements in the data file is provided to the model, the LLMs may still be unable to accurately understand the data file, resulting in data conversion errors and the inability to extract the desired data. To address this issue, engineers typically manually filter or screen the data, for example, by creating customized code to achieve the filtering or screening. Such data conversion is time-consuming and inconvenient. Therefore, the inventors devised the data conversion method of the present invention. It solves the above-mentioned problems, improves data conversion efficiency, and enhances data conversion accuracy by using two task agents (forming a dual-agent system) that cooperate with each other: a large-scale language model-based data conversion agent and a verification agent.

[0021] The method will now be described by way of example with reference to FIGS.

[0022] FIG. 1 is an exemplary flowchart of a method 100 of data transformation according to one embodiment of the present invention. FIG. 2 is a diagram illustrating an interaction scenario of the data transformation method according to one embodiment of the present invention. As shown in FIG. 2, the method 100 is realized by an object Obj. The object Obj may be a program, a script, or the like. The program or script includes instructions for performing operations (also called steps) Op101, Op103, Op105, and Op107. As an example, a program or script corresponding to the method 100 is executed by a computer to realize the method 100.

[0023] In operation Op101, a first large-scale language model agent Ag1 (i.e., a "data conversion agent") and a second large-scale language model agent Ag2 (i.e., a "verification agent") are defined as an agent system aSys by providing (sending) two instructions to at least one large-scale language model, the two instructions being configured to cause the agent system aSys, through cooperation between the first large-scale language model agent Ag1 and the second large-scale language model agent Ag, to generate, based on the first structured data file dataF1, a second structured data file dataF2 that can answer all questions in a question set Qs, including questions about the first structured data file dataF1. Of the two instructions, the instruction defining the first large-scale language model agent Ag1 is further configured to define the first large-scale language model agent Ag1 to perform the following data conversion task: when the unanswered question information unQ_Inf indicating the unanswered questions indicates that there are unanswered questions, extract data for answers to all questions in the question set Qs1 from the first structured data file dataF1 as a second structured data file dataF2 (i.e., perform a "generation operation" to generate dataF2). Of the two instructions, the instruction defining the second large-scale language model agent Ag2 is further configured to define the second large-scale language model agent Ag2 to perform the following verification task (also referred to as a "question answering task" or "update task"): update the unanswered question information unQ_Inf by verifying whether each question in the question set Qs can be answered based on the second structured data file dataF2. As can be understood, when two or more instructions are required to define two task agents of the present invention, the definition of the two task agents can be achieved by providing two or more corresponding instructions.

[0024] The first large-scale language model agent Ag1 is a large-scale language model-based task agent. After inputting the function definition into the large-scale language model, the large-scale language model can provide a corresponding task agent that can realize the defined function for user use or invocation. As described above, in the present invention, Ag1 is defined to complete a data transformation task. The second large-scale language model agent Ag2 is also a large-scale language model-based task agent. As described above, in the present invention, Ag2 is defined to update the unanswered question information unQ_Inf by completing a question answering task. As can be understood, Ag2 can actually complete a verification task to verify whether all questions in Qs can be answered based on dataF2. Therefore, Ag2 may be referred to as a "verification agent." For example, suppose there are eight questions in the question set QS. During the verification operation, if Ag2 successfully answers questions 1-7 and generates an answer for question 8 that is "answer failed" or "error occurred," the unanswered question information unQ_Inf is updated to indicate that there is no answer for question 8. If the answer result of Ag2 indicates that all questions have been answered (regardless of whether the answer content is actually correct, i.e., no "answer failure" or "error occurrence" occurs), unQ_Inf is updated to indicate that there are no unanswered questions (e.g., the unanswered question list it contains is set to an "empty" list). Ag1 and Ag2 may be based on the same large-scale language model or different large-scale language models. In one example, the large-scale language model is a generative pre-trained transformer model, such as GPT (including ChatGPT) or BERT. As an example of performing operation Op101 when Ag1 and Ag2 are based on the same large-scale language model, a user account logged in to the large-scale language model LLM provides (sends) instructions defining Ag1 and Ag2 to the server of the large-scale language model LLM. As can be understood, if an account is not required to use the large-scale language model, a user may log in to the large-scale language model directly without using a user account and send instructions defining Ag1 and Ag2 to the server of the large-scale language model LLM.After completing the definition operations of Ag1 and Ag2, Obj may send instructions to Ag1 and Ag2 to complete a predetermined task, for example, instruct Ag1 to perform a data transformation task. If Ag1 and Ag2 are based on different large-scale language models, Ag1 and Ag2 may be obtained by providing definition instructions to two different large-scale language models, respectively.

[0025] The computer that executes the program or script corresponding to the method 100 and the computer that executes the large-scale language models associated with Ag1 and Ag2 may be the same computer or different computers.

[0026] The second structured data file, dataF2, may be a WORD file, a text (TXT) file, an EXCEL file, a JSON file, etc. DataF2 may include "field information" for the data and corresponding data for each field. That is, DataF2 has a predetermined data structure including structured data. The specific data structure of the second structured data file, dataF2, differs from that of the first structured data file, dataF1. For example, DataF2 may have fewer or more fields, a smaller or larger file size, and more or fewer data types than DataF1. For example, the data type of a certain field in DataF2 may differ from that of the same field in DataF1. For example, DataF2 has a user-friendly data layout format that allows users to easily view DataF2 directly to obtain desired information. The second structured data file, DataF2, may be a data file related only to Qs and may not include data unrelated to Qs. For example, if Qs does not include "gender," DataF2 will not include data related to "gender."

[0027] In operation Op103, a first structured data file dataF1 and a question set Qs provided by a user U are received. The first structured data file dataF1 is data in a predetermined field. The predetermined field may be, for example, one of traffic management, sales management, inventory management, industrial production, social opinion monitoring, environmental monitoring, remote sensing, weather, finance, social networking services, video surveillance, logistics management, material transportation, image analysis, and audio analysis. Note that in method 100, there are no particular limitations on the order in which operations Op101 and Op103 are performed, and Op101 may be performed before or after Op103.

[0028] The first structured data file dataF1 may be a WORD file, a text (TXT) file, an EXCEL file, a JSON file, etc. DataF1 may include "field information" for the data and corresponding data for each field. That is, dataF1 has a predetermined data structure and is a data file, not a natural language file. DataF1 contains structured data, not a natural language file consisting of natural language phrases, sentences, or paragraphs. In one example, the first structured data file dataF1 does not include natural language sentences describing (explaining) the data.

[0029] The number of questions in the question set Qs may be one, two, or more. The questions may be questions of user interest, and may be questions to which answers can be expected based on the data in dataF1. For example, dataF1 is today's customer flow data, and one example question in Qs is "What is the gender distribution of customers entering this supermarket today?"

[0030] In operation Op105, an instruction Ins is sent to the agent system aSys to generate a second structured data file dataF2 based on the first structured data file dataF1 and the question set Qs (i.e., a "data conversion instruction" is sent). Ins may include path information for dataF1 so that Ag1 can read the file. Ins includes Qs. After receiving the instruction (e.g., Ag1 receives the instruction), according to the above definitions of Ag1 and Ag2, Ag1 and Ag2 can work together to generate a second structured data file that can answer all the questions in the question set Qs. Ag2 verifies whether dataF2 generated by Ag1 can answer all the questions in Qs. Ag2 also updates the unanswered question information unQ_Inf based on the question answer status. Whether Ag1 performs the generation operation is determined depending on whether the unanswered question information unQ_Inf indicates that there are no unanswered questions. For example, when the unanswered question information unQ_Inf indicates that there are no unanswered questions, Ag1 no longer performs the generation operation, and when the unanswered question information unQ_Inf indicates that there are unanswered questions, Ag1 performs the generation operation. For example, Ag1 may be defined to repeatedly (cyclically) perform the generation operation until the unanswered question information unQ_Inf indicates that there are no unanswered questions. Before the first execution of the generation operation, unQ_Inf is initialized to indicate that all questions in Qs are unanswered questions. This initialization may be performed by Obj (Ins may include the initialized unQ_Inf) or may be included in the instructions for defining Ag1 (i.e., the initialization is performed by Ag1).To realize the cooperation between Ag1 and Ag2, Ag1 and Ag2 need to be able to directly or indirectly interact with each other (e.g., transmit dataF2 and unQ_Inf) to realize the following: Ag2 can obtain the generated dataF2 and start an update operation after learning that Ag1 has generated dataF2; Ag1 can perform a conversion operation when it learns that Ag2 has completed updating unQ_Inf and that this indicates that there is an unanswered question. If the associated large-scale language model supports direct interaction between Ag1 and Ag2, the instructions for defining Ag1 and Ag2 may include instructions regarding direct interaction between Ag1 and Ag2. If the associated large-scale language model does not support direct interaction between Ag1 and Ag2, indirect interaction between Ag1 and Ag2 may be realized via Obj. For example, the program corresponding to Obj may include instructions for obtaining dataF2 and unQ_Inf, and instructions for providing the obtained dataF2 and unQ_Inf to Ag2 and Ag1, respectively.

[0031] In operation Op107, a second structured data file dataF2 that has been verified (passed verification) for the second large-scale language model agent Ag2 and can answer all questions in the question set Qs is obtained from the agent system aSys (denoted as "vdataF2" in FIG. 2 to emphasize that it has passed verification). The agent systems (i.e., Ag1 and Ag2) can perform a loop of conversion and verification until vdataF2 is generated in accordance with the data conversion instruction sent to Ag1. In one example, after obtaining vdataF2, a notification that "data conversion was successful" may be displayed to the user. Furthermore, in method 100, if vdataF2 is configured to be automatically stored in a default location, the user may be informed of the storage location of vdataF2 and / or vdataF2 may be automatically opened and displayed to the user. Additionally, in method 100, if vdataF2 is not configured to be automatically stored (e.g., configured not to be stored on a local hard disk or non-volatile memory), after obtaining vdataF2, user U may be prompted to specify a storage path for storing vdataF2.

[0032] As can be seen, in method 100, the data conversion agent and the verification agent are repeatedly invoked until all questions are answered. As shown in FIG. 2 , method 100 receives a first structured data file dataF1 and a question set Qs as input, and sends a plurality of instructions Inss (instructions for defining Ag1 and Ag2 and data conversion instructions) to an agent system aSys, causing the agent system to generate a second structured data file vdataF2 that has been verified as being able to answer all questions in Qs. Method 100 also acquires vdataF2 from the agent system for use by the user. Method 100 improves the method for converting data using a large-scale language model, without improving or changing the large-scale language model itself. Method 100 is a type of task-driven automatic data conversion method (i.e., "answering user questions"). Method 100 may acquire vdataF2 by performing a single unit operation, including a "data conversion task" and a "question answering task," or may require multiple unit operations to acquire vdataF2.

[0033] By performing method 100, a user can obtain a second structured data file, dataF2, corresponding to the questions of interest to the user. The second structured data file, dataF2, has a format that can directly answer the set of questions of interest to the user, Qs. Ag1 converts dataF1 to obtain dataF2. The conversion process may include restructuring, formatting, and simplifying dataF1. Ag2 checks the data in dataF2 to ensure that it accurately reflects the needs of the questions and can provide satisfactory answers. The final second structured data file can be used for display (including data visualization), statistics, or further processing (e.g., using a drawing application (app) to draw graphs related to the data in the data file) related to the questions of interest to the user.

[0034] In method 100, since both Ag1 and Ag2 have dataF2, obtaining from the agent system aSys the second structured data file dataF2 that has been verified for the second large-scale language model agent Ag2 and can answer all questions in the question set Qs may be obtained from Ag1 or from Ag2.

[0035] The purpose of method 100 is to use intelligent processing to transform domain data into a format that can directly answer relevant questions. The core process involves two large-scale language model-based agents: a data transformation agent and a verification agent. First, the data transformation agent receives the original data and the relevant question, analyzes and understands these inputs, and can recognize how to effectively extract and transform the data to meet the needs of the question. This involves restructuring, formatting, and simplifying the data to ensure that the new data set generated adheres to the core requirements of the question. Next, the verification agent checks this newly generated data to ensure that it accurately reflects the needs of the question and provides a convincing answer. If the new data fails verification, the data transformation agent can adjust based on the feedback, and this process is repeated until the data passes verification. With this highly automated and intelligent flow, the present invention not only significantly simplifies data and its storage, but also ensures that the data transformation process can accurately target specific questions, thereby improving the efficiency and effectiveness of data processing.

[0036] Application fields of method 100 include, but are not limited to, traffic, transportation, sales, industrial production, public opinion, environmental monitoring, remote sensing, weather, finance, online social, video analysis, image analysis, audio analysis, etc. The first structured data file may be a structured data file of one of the following types of data: traffic data (e.g., traffic volume data), sales data (e.g., retail data), inventory data, industrial data, social opinion data, environmental data, remote sensing data, weather data, financial data, social media data, video data, logistics data, transportation data, image data, audio data, etc.

[0037] In one embodiment, the instruction Ins in operation Op105 of method 100 may include information about dataF1, dataF1_Inf, information about the question set, Qs_inf, and optional additional information, add_Inf. Therefore, the instruction Ins may exemplarily be expressed as "Ins(dataF1_Inf, Qs_inf, add_Info)". dataF1_Inf may be the path of dataF1. Qs_inf may be the path of the file of question set Qs or a list of all questions in Qs. The additional information, add_Inf, may include, for example, the path required for saving vdataF2, an interpretation (explanation) of the contents of dataF1 (e.g., a description of at least some of the fields included therein), etc. Providing such an interpretation is beneficial for the agent system (e.g., Ag1) to better understand dataF1 and generate vdataF2 more quickly, thereby improving the conversion efficiency of the entire system.

[0038] To enhance the translation performance of the agent system, Ag2 may be configured with a question expansion capability. In one embodiment, instructions for defining Ag2 in the agent system are configured to have Ag2 generate expansion questions based on dataF2 and Qs, and update Qs by adding the expansion questions to Qs, where the expansion questions are different from the original questions in Qs provided by the user. Defining Ag2 causes Ag2 to generate expansion questions before performing a verification operation. In one example, the updated Qs are provided to Ag1. These expansion questions can cover some situations that cannot be covered by the original questions. The instructions for defining Ag2 may also include the definition of the following function: dynamically adjust the expansion question generation strategy based on previous verification results. For example, if a question is found in a particular data set, Ag2 may generate more and more detailed verification questions for this set. This means that the verification process is iterative and can adaptively respond to and provide feedback on discovered questions, improving overall verification efficiency and quality. This self-reinforcing query expansion mechanism allows the verification agent Ag2 to not only strengthen the initial verification of file content, but also further mine questions as they are discovered, providing users with a more comprehensive and in-depth data checking service. This ultimately achieves a more complete verification process and ensures that users can trust the converted data for subsequent analysis or decision-making processes.

[0039] To prevent method 100 from entering an "infinite loop" due to an improper setting of Qs or other reasons, the number of loops may be counted, and if the number of loops exceeds a threshold, the user may be notified that the data conversion task has failed and the method may be terminated. In one embodiment, method 100 may include counting the number of times the data conversion task or update task has been executed, and if the count value exceeds a threshold, the user may be notified that the data conversion has failed and the method 100 may be terminated.

[0040] To improve the data transformation performance of Ag1, the method 100 may include an interaction function between the user and Ag1. In one embodiment, the method 100 may further include providing an interface for interaction between the user U and the agent Ag1 regarding data extraction and / or the data arrangement format of the second structured data file dataF2. In one example, the instructions defining Ag1 may include instructions such as Ag1 spontaneously posing a question based on dataF1 and Qs. The posed question is provided to the user U via the interaction interface, and the user U provides an answer to the question to Ag1 via the interaction interface. Providing such an interaction interface is advantageous for Ag1 to accurately understand dataF1, accurately and quickly extract useful data, and arrange the extracted data in a desired format.

[0041] Also, according to one aspect of the present invention, there is provided an apparatus for data conversion. Figure 3 is an exemplary block diagram of an apparatus 300 for data conversion according to one embodiment of the present invention.

[0042] The apparatus 300 includes a memory 301 in which instructions Inst are stored, and also includes at least one processor 303 configured to execute the instructions Inst to implement the method 100.

[0043] According to another aspect of the present invention, there is provided a non-transitory computer-readable storage medium storing a program, which, when executed by a computer, causes the computer to perform the following operations: provide two instructions to at least one large-scale language model to define a first large-scale language model agent and a second large-scale language model agent as an agent system, the two instructions causing the agent system to generate, based on the first structured data file, a second structured data file that can answer all questions in a question set that includes a question related to the first structured data file through cooperation between the first large-scale language model agent and the second large-scale language model agent; and define the first large-scale language model agent to perform the following data transformation tasks: If the second large scale language model agent is defined to perform the following verification tasks, the second large scale language model agent is configured to update unanswered question information by verifying whether each question in the question set can be answered based on the second structured data file; receive the first structured data file and the question set provided by the user; provide instructions to the agent system for generating a second structured data file based on the first structured data file and the question set; and obtain from the agent system the verified second structured data file of the second large scale language model agent that can answer all questions in the question set. For more details of the program, please refer to the description of method 100.

[0044] Furthermore, according to one aspect of the present invention, there is further provided an information processing device.

[0045] 4 is an exemplary block diagram of an information processing device 40 according to one embodiment of the present invention. In FIG. 4, a central processing unit (CPU) 401 performs various processes based on a program stored in a ROM 402 or a program loaded from a storage device 408 into a RAM 403. The RAM 403 may store data required when the CPU 401 executes various processes as needed.

[0046] The CPU 401, ROM 402, and RAM 403 are connected to one another via a bus 404. An input / output interface 405 is also connected to the bus 404.

[0047] The following components may also be connected to the input / output interface 405: an input device 406 including a soft keyboard, an output device 407 including a display such as an LCD and a speaker, a storage device 408 such as a hard disk, and a communication device 409 including a network interface card, for example, a LAN card, a modem, etc. The communication device 409 can realize communication processing via a network such as the Internet, a LAN, a mobile network, or a combination thereof.

[0048] The drive 410 is also connected to the input / output interface 405 as needed. A removable medium 411, such as a semiconductor memory, can be set in the drive 410 as needed, allowing a program read from the removable medium 411 to be installed in the storage device 408 as needed.

[0049] The CPU 401 can execute a program or script corresponding to a data conversion method.

[0050] According to another aspect of the present invention, there is provided a data conversion apparatus, which will be exemplarily described below with reference to FIG. 5. FIG. 5 is an exemplary block diagram of a data conversion apparatus 500 according to one embodiment of the present invention. The apparatus 500 includes a definition unit 501, a receiving unit 503, a task sending unit 505, and an acquisition unit 507. The definition unit 501 is configured to define a first large-scale language model agent Ag1 and a second large-scale language model agent Ag2 as an agent system aSys by providing two instructions to at least one large-scale language model. The two instructions configure the agent system to generate, based on the first structured data file, a second structured data file dataF2 that can answer all questions in a question set Qs, including questions about the first structured data file dataF1, through cooperation between the first large-scale language model agent and the second large-scale language model agent. The first large-scale language model agent Ag1 is defined to perform the following data conversion task: extract data for answering all questions in the question set from the first structured data file as a second structured data file when the unanswered question information unQ_Inf indicating the unanswered questions indicates that there are unanswered questions. The second large-scale language model agent is defined to perform the following verification task: update the unanswered question information by verifying whether each question in the question set can be answered based on the second structured data file. The receiving unit 503 is configured to receive the first structured data file dataF1 and the question set Qs provided by the user U. The task sending unit 505 is configured to send an instruction to the agent system to generate a second structured data file based on the first structured data file and the question set. The acquiring unit 507 is configured to acquire from the agent system the second structured data file verified by the second large-scale language model agent and capable of answering all questions in the question set. For further configuration of the device 500, please refer to the description of the method 100 of the present invention.

[0051] To verify the effectiveness of method 100, the inventors tested the actual effectiveness of method 100 using an exemplary structured data file "test.json" containing customer information. Ag1 and Ag2 are data conversion and verification agents based on ChatGPT. Figure 6 shows a screenshot of the interactive interface when ChatGPT fails to answer an exemplary question. Specifically, the structured data file "test.json" is provided to ChatGPT, and ChatGPT is prompted to "show me the age range distribution of all customers." ChatGPT provides "Error analyzing" feedback, indicating that using a traditional single agent (without data conversion using the dual agents included in method 100) ChatGPT fails to accurately understand the data and extract the data necessary to answer the question. Figure 7 shows a screenshot of the interactive interface when ChatGPT successfully answers an exemplary question. Specifically, the structured data file “test.json” is provided to the method 100, and the method 100 provides the converted data dataF2, based on which ChatGPT successfully provides the “customer age distribution”.

[0052] The beneficial effects of the method, apparatus, and computer program of the present invention include at least one of the following: increasing the automation level of data conversion; improving the quality of converted data so that the converted data fully meets the needs of users; increasing the efficiency of data conversion; improving the accuracy of data conversion; and improving the effectiveness and reliability of data conversion.

[0053] Furthermore, the above-mentioned embodiments are further disclosed as supplementary notes as follows.

[0054] (Appendix 1) 1. A method of transforming data, comprising: providing two instructions to at least one large-scale language model, the first large-scale language model agent and the second large-scale language model agent being defined as an agent system, the two instructions being: the agent system generates, through cooperation between the first large scale language model agent and the second large scale language model agent, a second structured data file based on the first structured data file that can answer all questions in a question set that includes questions about the first structured data file; The first large scale language model agent is defined to perform the following data transformation task: if the unanswered question information indicates that there are unanswered questions, extract data from the first structured data file to answer all questions in the question set as the second structured data file; and The second large scale language model agent is defined to perform the following verification task: updating the unanswered question information by verifying whether each question in the question set can be answered based on the second structured data file; It is configured as follows: receiving the first structured data file and the set of questions provided by a user; sending instructions to the agent system to generate the second structured data file based on the first structured data file and the question set; and obtaining from the agent system the second structured data file that has been verified by the second large scale language model agent to be capable of answering all questions in the question set.

[0055] (Appendix 2) The method of claim 1, further comprising: presenting the user with a designation of a storage path for storing the retrieved second structured data file.

[0056] (Appendix 3) 2. The method of claim 1, comprising: Of the two instructions, the instructions defining the first large scale language model agent include a description of the contents of the first structured data file.

[0057] (Appendix 4) The method of claim 1, further comprising: providing an interface for the user to interact with the first large scale language model agent regarding data extraction and / or data arrangement of the second structured data file.

[0058] (Appendix 5) 2. The method of claim 1, comprising: Of the two instructions, the instructions defining the second large scale language model agent are configured to cause the second large scale language model agent to generate expanded questions based on the second structured data file and the question set, and to update the question set by adding the expanded questions to the question set.

[0059] (Appendix 6) 2. The method of claim 1, comprising: The at least one large-scale language model is a generative pre-trained transformer model.

[0060] (Appendix 7) 2. The method of claim 1, comprising: The data arrangement format of the second structured data is different from the data arrangement format of the first structured data.

[0061] (Appendix 8) 2. The method of claim 1, comprising: Of the two instructions, the instructions defining the first large scale language model agent configure the first large scale language model agent to generate the second structured data file having a user-friendly data arrangement format.

[0062] (Appendix 9) 2. The method of claim 1, comprising: The at least one large-scale language model is ChatGPT.

[0063] (Appendix 10) 2. The method of claim 1, comprising: The first structured data file does not include natural language sentences describing the data.

[0064] (Appendix 11) 1. An apparatus for transforming data, comprising: a memory device storing instructions (computer programs); and at least one processor coupled to the memory; The processor is configured to execute the instructions to implement the method of any one of claims 1 to 10.

[0065] (Appendix 12) A non-transitory computer-readable storage medium storing a computer program, When the computer program is executed by a computer, the computer program causes the computer to: providing two instructions to at least one large-scale language model, the first large-scale language model agent and the second large-scale language model agent being defined as an agent system, the two instructions being: the agent system generates, through cooperation between the first large scale language model agent and the second large scale language model agent, a second structured data file based on the first structured data file that can answer all questions in a question set that includes questions about the first structured data file; The first large scale language model agent is defined to perform the following data transformation task: if the unanswered question information indicates that there are unanswered questions, extract data from the first structured data file to answer all questions in the question set as the second structured data file; and The second large scale language model agent is defined to perform the following verification task: updating the unanswered question information by verifying whether each question in the question set can be answered based on the second structured data file; Configure it as follows: receiving the first structured data file and the set of questions provided by a user; sending instructions to the agent system to generate the second structured data file based on the first structured data file and the question set; and obtaining from the agent system the second structured data file that has been verified by the second large scale language model agent to be capable of answering all questions in the question set.

[0066] (Appendix 13) 13. The storage medium of claim 12, The computer program further causes the computer to prompt the user to specify a storage path for storing the obtained second structured data file.

[0067] (Appendix 14) 13. The storage medium of claim 12, Of the two instructions, the instructions defining the first large scale language model agent include a description of the contents of the first structured data file.

[0068] (Appendix 15) 13. The storage medium of claim 12, The computer program further causes the computer to: providing an interface for the user to interact with the first large scale language model agent regarding data extraction and / or data arrangement of the second structured data file;

[0069] (Appendix 16) 13. The storage medium of claim 12, Of the two instructions, the instructions defining the second large scale language model agent are configured to configure the second large scale language model agent to generate expanded questions based on the second structured data file and the question set, and to update the question set by adding the expanded questions to the question set.

[0070] (Appendix 17) 13. The storage medium of claim 12, The at least one large-scale language model is a generative pre-trained transformer model.

[0071] (Appendix 18) 13. The storage medium of claim 12, The data arrangement format of the second structured data is different from the data arrangement format of the first structured data.

[0072] (Appendix 19) 13. The storage medium of claim 12, The two instructions configure the first large scale language model agent to generate the second structured data file having a user-friendly data arrangement format.

[0073] (Appendix 20) 13. The storage medium of claim 12, The first structured data file does not include natural language sentences describing the data.

[0074] Although the preferred embodiment of the present invention has been described above, the present invention is not limited to this embodiment, and any modification to the present invention falls within the technical scope of the present invention as long as it does not depart from the spirit of the present invention.

Claims

1. 1. A method of transforming data, comprising: providing two instructions to at least one large-scale language model, the first large-scale language model agent and the second large-scale language model agent being defined as an agent system, the two instructions being: the agent system generates, through cooperation between the first large scale language model agent and the second large scale language model agent, a second structured data file based on the first structured data file that can answer all questions in a question set that includes questions about the first structured data file; The first large scale language model agent is defined to perform the following data transformation task: if the unanswered question information indicates that there are unanswered questions, extract data from the first structured data file to answer all questions in the question set as the second structured data file; and The second large scale language model agent is defined to perform the following verification task: updating the unanswered question information by verifying whether each question in the question set can be answered based on the second structured data file. configured so that: receiving the first structured data file and the set of questions provided by a user; sending instructions to the agent system to generate the second structured data file based on the first structured data file and the question set; and obtaining from the agent system the second structured data file that has been validated by the second large scale language model agent and that can answer all questions in the question set.

2. 10. The method of claim 1, The method, wherein the two instructions defining the first large scale language model agent include a description of the contents of the first structured data file.

3. 10. The method of claim 1, The method further comprising providing an interface for the user to interact with the first large scale language model agent regarding data extraction and / or data arrangement of the second structured data file.

4. 10. The method of claim 1, wherein the two instructions defining the second large scale language model agent are configured to cause the second large scale language model agent to generate expansion questions based on the second structured data file and the question set, and to add the expansion questions to the question set to update the question set.

5. 10. The method of claim 1, The method, wherein the at least one large-scale language model is a generative pre-trained transformer model.

6. 10. The method of claim 1, wherein the two instructions defining the first large scale language model agent are configured to cause the first large scale language model agent to generate the second structured data file having a user-friendly data arrangement format.

7. 10. The method of claim 1, A method wherein the data layout format of the second structured data file is different from the data layout format of the first structured data file.

8. 10. The method of claim 1, The method, wherein the first structured data file does not include natural language sentences describing the data.

9. 1. An apparatus for converting data, comprising: a memory storing the program; and at least one processor coupled to the memory; 9. Apparatus, wherein the processor is configured to execute the program to implement the method of any one of claims 1 to 8.

10. A program for causing a computer to carry out the method according to any one of claims 1 to 8.

Images