Data cleaning device and data cleaning method

Abstract

The invention provides a data cleaning device and a data cleaning method, the data cleaning device executes a data cleaning method including executing an application program and executing a plug-in hint management program, the execution of the plug-in hint management program includes generating a hint instruction according to an unformatted content through a first hint module and transmitting the hint instruction to a first device, and receiving formatted content generated by a large language model according to the prompt instruction from the first device. According to the invention, the operation process of converting the unformatted content into the formatted content is greatly reduced. By managing and executing the prompt module in a mode of externally hanging the program in the application program, the influence on the original core system program operation of the application program is avoided, and a user does not need to change the original application program use habit.

Description

Technical Field

[0001] A data cleaning device and a data cleaning method, in particular, a data cleaning device and a data cleaning method combined with a large language model. Background Art

[0002] Data cleaning technology refers to processing unformatted content to generate formatted content that meets application requirements. Traditionally, unformatted content is usually identified manually, and then the identification results are further converted into formatted content that meets application requirements. However, manual processing of unformatted content is not only time-consuming and inefficient, but also the speed of manual data processing is far slower than the speed of data generation.

[0003] In summary, data cleaning technology for unformatted content urgently needs to be further developed to improve the above problems. Summary of the invention

[0004] In view of the limitations of existing non-formatted text content processing technology, the present invention provides a data cleaning device and method to solve the above problems.

[0005] The present invention provides a data cleaning device, comprising:

[0006] A memory configured to store an application, a prompt management plug-in and a prompt template registry;

[0007] a processor, connected to the memory, configured to read the application from the memory and execute the application, wherein executing the application comprises:

[0008] Executing the prompt plug-in management program to access a first prompt template from at least one prompt template indicated by the prompt template registration table;

[0009] Generate a prompt instruction according to a non-formatted content through the first prompt template, and transmit the prompt instruction to a first device; and

[0010] A formatted content is received from the first device, wherein the first device generates the formatted content according to the prompt instruction through a large language model (LLM).

[0011] The present invention also provides a data cleaning method, which is executed by a processor of the data cleaning device and includes the following steps:

[0012] An application is read from a memory and executed, wherein executing the application comprises:

[0013] Executing a prompt plug-in management program to access a first prompt template from at least one prompt template indicated by the prompt template registration table;

[0014] Generate a prompt instruction according to a non-formatted content through the first prompt template, and transmit the prompt instruction to a first device; and

[0015] A formatted content is received from the first device, wherein the first device generates the formatted content according to the prompt instruction through a large language model (LLM).

[0016] The data cleaning device of the present invention provides the management, presentation and execution functions of the prompt template by executing the prompt plug-in management program under the application program. When the prompt plug-in management program is executed, the application accesses the first prompt template from the multiple prompt templates provided by the prompt template registry, and further generates a complete prompt instruction according to the non-formatted content through the first prompt template. The prompt instruction is transmitted by the data cleaning device to a first device, so that the large language model of the first device processes the text content of the non-formatted content therein in accordance with the prompt instruction to meet the requirements of the first prompt template, and generates the specified formatted content. Finally, the data cleaning device receives the completed formatted content.

[0017] The present invention only needs to start the prompt plug-in management program and the prompt template registry to provide the unformatted content to the first prompt module to generate prompt instructions, and transmit the prompt instructions to the large language model of the first device to generate formatted content, from generating unformatted content to receiving formatted content. This greatly reduces the operation process of converting unformatted content into formatted content. By managing and executing the prompt template in the form of a plug-in program in the application, it avoids affecting the original core system program operation of the application, and the user does not need to change the original habit of using the application, for example, there is no need to open an additional data cleaning application or device. In addition, by using the multiple prompt templates presented in the prompt template registry, when prompt instructions that meet specific application requirements are required, the prompt templates that are pre-defined by the prompt project to meet the specific application requirements can be directly applied to generate prompt content that meets the specific application requirements, thereby ensuring that the formatted content generated by the large language model has a data type and format that meets the specific application requirements, so as to improve the problems of the background technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 FIG. 4 is a block diagram of a data cleaning system according to an embodiment of the present invention.

[0019] Figure 2 1 is a flowchart of a data cleaning method according to an embodiment of the present invention.

[0020] Figure 3 It is a block diagram of an application state of a data cleaning device executing an application program according to an embodiment of the present invention.

[0021] Figure 4 It is an exemplary prompt instruction generated by the data cleaning device according to an embodiment of the present invention.

[0022] Figure 5 is an exemplary formatted content received by the data cleaning device according to the embodiment of the present invention.

[0023] Figure 6 It is another exemplary prompt instruction generated by the data cleaning device according to an embodiment of the present invention.

[0024] Figure 7 is another exemplary formatted content received by the data cleaning device according to the embodiment of the present invention. DETAILED DESCRIPTION

[0025] See also Figure 1 As shown, Figure 11 is a block diagram of a data cleaning system 1 according to an embodiment of the present invention. The data cleaning system 1 includes a data cleaning device 10 and a first device 20. The data cleaning device 10 includes a memory 11 and a processor 12, and is connected (e.g., communication connected) to the first device 20. The memory 11 is configured to store an application 13, a prompt plug-in management program 14, a prompt template registry 15, and at least one prompt template 150. The at least one prompt template 150 may include a first prompt template 150A. The at least one prompt template 150 may also include other prompt templates (not shown). The processor 12 is connected to the memory 11 to read the application 13, the prompt plug-in management program 14, and the prompt template registry 15 from the memory 11. The prompt plug-in management program 14 is configured as a plug-in module of the application 13. When the processor 12 executes the application 13, the prompt plug-in management program 14 may be synchronously executed to perform the management function of at least one prompt template 150. The prompt template registry 15 is used to manage (e.g., track or update) at least one prompt template 150 and / or store relevant information of at least one prompt template 150, such as template name, template function, template version and / or required resource definition, etc., but not limited thereto. The first device 20 includes (e.g., stores) a large language model 21 (LLM). The large language model 21 can be a natural language model that completes a large amount of text training based on artificial intelligence, such as a chat generative pre-trained transformer (chatGPT) or a model derived therefrom (e.g., Generative Pre-Trained Transformer 4 (GPT-4)), a large language model meta-artificial intelligence (LLM Meta AI, LLaMA), a pathways language model (PaLM) or a model derived therefrom (e.g., PaLM 2 or Bard), any model having similar functions to the above models, or a combination of the above models, but not limited thereto.

[0026] In one embodiment, the data cleaning device 10 can also be connected (e.g., connected by communication) to a second device 30, and the second device 30 can be used to store at least one public prompt template 31, and the at least one public prompt template 31 can include a second prompt template 150B. The at least one public prompt template 31 can also include other public prompt templates (not shown). Since the second device 30 is optional in the present invention, the second device 30 is shown in dotted lines. The second device 30 is, for example, an external device (e.g., a server) for storing at least one public prompt template 31, so that engineers who perform prompt engineering can upload at least one public prompt template 31 of the completed prompt engineering, and for the prompt plug-in management program 14 to connect and access when it is activated to perform the following operations: synchronize and check the prompt templates available in the at least one public prompt template 31, view the information of the prompt template, and download the prompt template that meets the application requirements (e.g., the second prompt template 150B therein) to the data cleaning device 10, in response to the activation of the prompt plug-in management program 14. Correspondingly, the data cleaning device 10 can install and deploy the prompt templates downloaded by it.

[0027] It should be noted that Figure 1 In the figure, at least one prompt template 150 is exemplarily depicted in a dotted line in the memory 11 of the data cleaning device 10, to indicate that at least one prompt template 150 in the present invention can be temporarily stored in the memory 11. The application 13 or the prompt plug-in management program 14 can download and store at least one prompt template 150 from an external device (such as the second device 30) in the memory 11 during execution or startup. The memory 11 can be any data storage device that can be read and executed by the processor 12, such as a memory, a hard disk, a solid state drive, a flash memory, or other data storage device suitable for storing programs, but is not limited thereto.

[0028] Please also read Figure 2 As shown, Figure 2 FIG. 1 is a flow chart of a step of a data cleaning method 100 according to an embodiment of the present invention. The data cleaning method 100 may be applicable to Figure 1The data cleaning device 10 can be compiled into a program to be executed by the processor 12, and includes the following steps: step S101: reading the application 13 from the memory 11 and executing the application 13; step S102: executing (e.g., starting) the prompt plug-in management program 14 to access a first prompt template 150A from at least one prompt template 150 indicated (e.g., pointed to) by the prompt template registration table 15; step S103: generating a prompt instruction according to the unformatted content through the first prompt template 150A, and transmitting the prompt instruction to the first device 20; step S104: receiving formatted content from the first device 20, wherein the first device 20 generates formatted content according to the prompt instruction through the large language model 21. That is, the prompt plug-in management program 14 generates a prompt instruction from the unformatted content through the predefined first prompt template 150A, and transmits the prompt instruction to the first device 20 so that the large language model 21 of the first device 20 formats the unformatted content, which can greatly reduce the human resources and time consumption, errors and inefficiency caused by manually processing the unformatted content.

[0029] Please also read Figure 3 As shown, Figure 31 is a block diagram of an application state of a data cleaning device 10 executing an application according to an embodiment of the present invention. The application 13 may include a core system program 131 and a prompt plug-in management program 14. The prompt plug-in management program 14 may include a prompt template registry 15 and a loader 17. The prompt template registry 15 may load at least one prompt template 150 from a prompt template directory 16 at a preset address. The core system program 131 may be any program having the function of inputting or receiving user input content (e.g., any data such as text, numerical value or symbol) (e.g., a text editor, a medical information integration system program, a financial information integration system program, any professional information integration system program, an online conference or an online chat room, etc., but not limited thereto), which may be the main function program of the application 13. The core system program 131 may generate unformatted content according to the user's input content. The user's input content may be, for example, any content input by the user through a text editor, or any content input by the user recognizing an image file. In one embodiment, the non-formatted content may be medical data, such as patient data, medical record data, test data, diagnosis data, medical image diagnosis data, or diagnosis book data handwritten by medical personnel, but not limited thereto. In one embodiment, the non-formatted content may be financial data, online conference chat content, and online chat room chat content, but not limited thereto. In some embodiments, the application 13 or the core system program 131 may define the functions, resources, services, and / or data exchange methods provided by the application 13 or the core system program to at least one prompt template 150 and / or at least one public prompt template 31 through a service interface (not shown in the figure, such as a data exchange protocol). In some embodiments, the prompt plug-in manager 14 can pre-define the purpose (e.g., for which application requirement and / or professional field) of each (e.g., required to be provided) of at least one prompt template 150 and / or at least one public prompt module 31 through a prompt template service interface (not shown in the figure, such as a service provision protocol) so that at least one prompt template 150 and / or at least one public prompt template 31 can be loaded and executed by the application 13 (the prompt plug-in manager 14 therein).

[0030] In some embodiments, according to the content of the input field in the non-formatted content, the application 13 may select a corresponding (e.g., application-compliant) prompt template (e.g., but not limited to, the first prompt template 150A and / or the second prompt template 150B) from at least one prompt template 150 indicated by the prompt template registry 15 to generate a prompt instruction according to the non-formatted content. In other embodiments, according to the user's selection (e.g., a prompt template selected from a drop-down form presented by the application 13), the application 13 may execute a corresponding (e.g., application-compliant) prompt template (e.g., but not limited to, the first prompt template 150A and / or the second prompt template 150B) to generate a prompt instruction according to the non-formatted content.

[0031] Each of the at least one prompt template 150 and the at least one public prompt template 31 may include prompt content pre-generated by an engineer performing a prompt project. Each of the at least one prompt template 150 may provide different formatting functions corresponding to different application requirements to generate different formatted content. The formatting function is pre-defined according to different expected application scenarios, expected input information, expected output formats, and expected output data types when performing a prompt project.

[0032] In more detail, each of the at least one prompt template 150 and the at least one public prompt template 31 may have a plurality of prompt content blocks, and different prompt content blocks may have different prompt contents, wherein one of the plurality of prompt content blocks may include an input content block. When the processor 12 generates a prompt instruction according to the non-formatted content through one of the at least one prompt templates 150 (e.g., the first prompt template 150A), the non-formatted content is embedded into the input content block of the one, thereby generating a prompt instruction. In addition, other blocks of the plurality of prompt content blocks may include at least one of output data type definition (e.g., using true or false values ​​to indicate whether the restriction in the categories is met, and using restriction explanation to define each restriction), output format definition (e.g., json format, FHIR format), output example definition (e.g., output example including categories and explanations), and output restriction definition (e.g., using explanations to define restrictions, using numerical values ​​to represent telephone numbers, and using YYYY-MM-DD to represent dates).

[0033] In one embodiment, generating a prompt instruction according to the unformatted content by using at least one of the prompt templates 150 (e.g., the first prompt template 150A) further includes performing preprocessing on the unformatted content to generate preprocessed unformatted content, and then generating a prompt instruction according to the preprocessed unformatted content. The preprocessing may at least include data cleaning processing, and the data cleaning processing may include at least one of wrong word processing (e.g., correction), illegal word processing (e.g., correction or deletion), missing value processing (e.g., value addition), and data type processing (e.g., data type change), but is not limited thereto.

[0034] In one embodiment, the data cleaning device 10 can be connected to the large language model 21 via an application programming interface (API). When the first device 20 receives the prompt instruction and inputs the prompt instruction to the large language model 21, the large language model 21 generates formatted content in the instruction format of at least one prompt template 150 (e.g., the first prompt template 150A) according to the unformatted content in the prompt instruction and the information such as the output data type definition, the output format definition, the output example definition, and / or the restriction condition definition.

[0035] In one embodiment, when the data cleaning device 10 receives the formatted content from the first device 20, the data cleaning device 10 stores the formatted content in the memory 11 or a destination device (not shown), and / or can display the formatted content. The destination device is, for example, an external server at the back end of the application 13 for uniformly storing formatted content from different source devices.

[0036] In one embodiment, the processor 12 may further clear resources used to execute (eg, load) the plug-in management program 14 , wherein the resources may include at least one storage space, but is not limited thereto.

[0037] Please also read Figure 1 and Figure 3As shown, in one embodiment, the loader 17 can be used to download at least one prompt template (for example, the second prompt template 150B) in at least one public prompt template module 31 from the second device 30, and store the downloaded prompt template in the prompt template directory 16 at a preset address in response to starting the prompt plug-in management program 14. The loader 17 can also be used to load the downloaded prompt template into the application 13 for execution by the application 13. In more detail, when the data cleaning device 10 executes the prompt plug-in management program 14, the prompt template registry 15 further scans the prompt template directory 16 at a preset address, and loads the prompt template in the prompt template directory 16 into the application 13 through the loader 17. The loader 17 can load the prompt template from the prompt template directory 16 when the prompt plug-in management program 14 is started (or executed). For the sake of simplicity of the diagram, Figure 1 and Figure 3 Only part of the prompt templates are shown in the at least one prompt template 150, but the number of the at least one prompt template 150 may not be limited to Figure 1 and Figure 3 The quantity shown in .

[0038] Furthermore, the data cleaning device 10 executes the prompt plug-in management program 14 and also includes updating the prompt template registry 15 according to the prompt template in the prompt template directory 16 (for example, the second prompt template 150B), thereby providing the updated prompt template registry 15 in the application 13, for example, displaying it in a drop-down menu to provide user selection, or providing the plug-in management program 14 to automatically trigger execution when it is detected that the input field includes non-formatted content.

[0039] Please also read Figure 1 and Figure 4 As shown, Figure 4 It is an exemplary prompt instruction Prt1 generated by the data cleaning device 10 according to an embodiment of the present invention, and its purpose is to instruct the large language model 21 of the first device 20 to format the medical image tag content and present the "categories" and "explanations" of the tag content in json format. The exemplary prompt instruction Prt1 includes an output format definition P1 (i.e., json format) provided by a prompt template that meets the medical image application requirements, an output example definition P2 (i.e., including categories and using true and false values ​​to indicate whether the categories meet the requirements), and also includes explanations to define each category), a restriction definition P3 (i.e., defining which categories), and an input content block P4 (i.e., input based on unformatted medical image tag content).

[0040] Please read further Figure 5 As shown, Figure 5is an exemplary formatted content Rsp1 received by the data cleaning device 10 according to an embodiment of the present invention, which is based on Figure 4 An exemplary prompt instruction Prt1 is provided by Figure 2 The data cleaning method 100 is generated. Figure 5 As shown, the exemplary formatting content Rsp1 is Figure 4 The output format definition P1, output example definition P2, constraint definition P3 and input content block P4 defined in the exemplary prompt instruction Prt1 are generated.

[0041] Please also read Figure 1 and Figure 6 As shown, Figure 6 Another exemplary prompt instruction Prt2 generated by the data cleaning device 10 according to an embodiment of the present invention is used to instruct the large language model 21 of the first device 20 to generate a meeting notice with a specific format and column content for the conversation record of the multi-person group. The exemplary prompt instruction Prt2 includes an output format definition and an example definition P5 (i.e., titles such as "basic data", "participants", "topics" and their corresponding designated columns) provided by a prompt template that meets the application requirements of the meeting notification, and an input content block P6 (i.e., input based on the non-formatted conversation record of the multi-person group).

[0042] Please read further Figure 7 As shown, Figure 7 is another exemplary formatted content Rsp2 received by the data cleaning device 10 according to an embodiment of the present invention, which is based on Figure 6 An exemplary prompt instruction Prt2 is passed Figure 2 The data cleaning method 100 is generated. Figure 7 As shown, the exemplary formatted content Rsp2 is represented by Figure 6 The output format definition and example definition P5 and the input content block P6 defined in the exemplary prompt instruction Prt2 are generated.

[0043] From the above two exemplary prompt instructions Prt1 and Prt2r and the corresponding exemplary formatting contents Rsp1 and Rsp2, it can be seen that the data cleaning device 10 and the data cleaning method 100 of the present invention can be effectively applied to any application requiring data cleaning (formatting) of unformatted contents.

[0044] In summary, the data cleaning device and method of the present invention have at least the following effects:

[0045] 1. The plug-in management program 14 does not affect the execution of the original application 13, and the user does not need to change the way of executing the original application 13.

[0046] 2. By executing prompt templates of different functions through the prompt plug-in management program 14, the user can select the prompt template for a specific application requirement according to the application requirement, or the data cleaning device can automatically execute the prompt template for a specific application requirement. The user does not need to generate prompt instructions by himself, which can improve the efficiency of generating prompt instructions and generate efficient prompt instructions.

[0047] 3. When efficient prompt instructions can be generated, a large language model is further applied to clean the non-formatted content to generate formatted content.

[0048] The above effects can be used to improve the problem of data cleaning technology of non-formatted content in the background art.

[0049] The above description is only an embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as above by the embodiment, it is not used to limit the present invention. Any technical personnel in this field can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present invention without departing from the content of the technical solution of the present invention still fall within the scope of the technical solution of the present invention.

Claims

1. A data cleaning device, characterized in that: include: a memory configured to store an application, a prompt plug-in management program and a prompt template registry; a processor, connected to the memory, configured to read the application from the memory and execute the application, wherein executing the application comprises: Executing the prompt plug-in management program to access a first prompt template from at least one prompt template indicated by the prompt template registration table; Generate a prompt instruction according to a non-formatted content through the first prompt template, and transmit the prompt instruction to a first device; and A formatted content is received from the first device, wherein the first device generates the formatted content according to the prompt instruction through a large language model.

2. The data cleaning device according to claim 1, characterized in that: Executing the prompt plug-in management program also includes: Scanning a reminder template directory at a preset address; and At least one second prompt module is downloaded from a second device through a loader, and the at least one second prompt template is stored in the prompt template directory of the preset address in response to the activation of the prompt plug-in management program.

3. The data cleaning device according to claim 2, characterized in that: Executing the prompt plug-in management program also includes: The prompt template registry is updated according to the at least one second prompt template.

4. The data cleaning device according to claim 2, characterized in that: Through a prompt template service interface, a purpose of each of the at least one second prompt template and the at least one prompt template is predefined.

5. The data cleaning device according to claim 1, characterized in that: Executing the application also includes: The non-formatted content is generated according to input content of a user.

6. The data cleaning device according to claim 1, characterized in that: The prompt command includes at least one of an output data type definition, an output format definition, and an output example definition.

7. The data cleaning device according to claim 1, characterized in that: Generating the prompt instruction according to the non-formatted content by using the first prompt template includes: performing a preprocessing on the unformatted content to generate a preprocessed unformatted content; and The prompt instruction is generated according to the pre-processed unformatted content, wherein the pre-processing includes a data cleaning process.

8. The data cleaning device according to claim 6, characterized in that: The data cleaning process includes at least one of a wrong word process, an illegal word process, a missing value process and a data type process.

9. The data cleaning device according to claim 1, characterized in that: Executing the application also includes: storing the formatted content or displaying the formatted content; and A resource for executing the prompt plug-in management program is cleared, wherein the resource includes at least a storage space.

10. A data cleaning method, used in a data cleaning device, characterized in that: The following steps are involved: Reading an application and executing the application, wherein executing the application comprises: Executing a prompt plug-in management program to access a first prompt template from at least one prompt template indicated by a prompt template registration table; Generate a prompt instruction according to a non-formatted content through the first prompt template, and transmit the prompt instruction to a first device; and A formatted content is received from the first device, wherein the first device generates the formatted content according to the prompt instruction through a large language model.

Images