Data set conversion method and system
Through the AI agent based on the LangChain large language model, using Prompt prompt words and automation tools, efficient data set conversion is achieved, solving the problems of low efficiency and high professional threshold of traditional methods, reducing development workload and cost, and meeting diverse Demand.
Patent Information
- Application Number
- CN202411118400.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-15
- Publication Date
- 2024-11-12
AI Technical Summary
Traditional dataset conversion methods are inefficient and require professionals to develop specific tooling programs, resulting in limited conversion capabilities and high barriers to high conversion.
Using an AI agent based on the LangChain large language model, the source data set is preprocessed, converted and post-processed through Prompt prompt words and built-in automation tools, and solutions are generated and automation tools are called to perform conversion operations, which lowers professional thresholds and development Workload.
It improves the data set conversion efficiency, reduces labor and technical costs, meets the conversion needs of different data sets, has strong scalability and low technical threshold, and does not require the development of new conversion programs.
Smart Images

Figure CN118940722A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of artificial intelligence, and in particular relates to a data set conversion method and system. Background Art
[0002] Before training a deep learning model, algorithm engineers need to prepare a training data set. There may be some ready-made source data sets in the local database, and they only need to convert the source data sets before they can be used. However, since traditional data set conversion work is performed on the local server, algorithm developers are generally required to use specific rule conversion tools or develop custom tool programs to convert data sets. This method has great limitations: not only is the conversion capability limited and the conversion efficiency is very low, but also specific tool programs must be developed for specific data set conversions. This requires personnel who are familiar with tool writing and data set rules to do this work, and places high demands on the professional quality of the operators. Summary of the invention
[0003] In view of this, an embodiment of the present invention provides a data set conversion method and system, which are used to solve the problem that the existing data set conversion efficiency is low and requires professionals to develop specific tool programs.
[0004] In a first aspect of an embodiment of the present invention, a data set conversion method is provided, comprising:
[0005] Obtain source data sets and preprocess the source data sets;
[0006] The preprocessed source data set is input into the AI agent. Based on the prompt words and built-in automation tools, the prompt words are analyzed by the large language model to generate solutions. The AI agent makes decisions to select solutions and calls the corresponding automation tools to perform conversion operations to obtain the converted data set.
[0007] The AI agent is an application developed based on the LangChain large language model;
[0008] Post-process the transformed dataset and output the transformed dataset.
[0009] In a second aspect of an embodiment of the present invention, a system for data set conversion is provided, comprising:
[0010] A data set input module is used to obtain a source data set, preprocess the source data set, and input the preprocessed source data set into the AI agent;
[0011] The data set conversion module is used to analyze the prompt words through the large language model to generate solutions based on the prompt words and the built-in automation tools. The AI agent makes decisions to select solutions and calls the corresponding automation tools to perform conversion operations to obtain the converted data set.
[0012] The AI agent is an application developed based on the LangChain large language model;
[0013] The data set output module is used to post-process the converted data set and output the converted data set.
[0014] In a third aspect of an embodiment of the present invention, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described in the first aspect of the embodiment of the present invention when executing the computer program.
[0015] In a fourth aspect of an embodiment of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method provided in the first aspect of the embodiment of the present invention are implemented.
[0016] In the embodiment of the present invention, based on the prompt word and the built-in automation tool, a solution is generated through a large language model, and the AI intelligent model makes a decision and calls the corresponding automation tool to convert the input source data set. Not only is the data set conversion efficiency high, but also the professional threshold of data set conversion work is lowered. There is no need to specially develop a new conversion program, which greatly reduces the development workload and manpower and material costs. At the same time, it can meet the conversion needs of different data sets, with low technical threshold and strong scalability. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0018] Figure 1 A flowchart of a data set conversion method provided by an embodiment of the present invention;
[0019] Figure 2 A schematic diagram of the structure of a system for data set conversion provided by one embodiment of the present invention;
[0020] Figure 3A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0021] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0022] It should be understood that the term "including" and other similar expressions in the specification or claims of the present invention and the above-mentioned drawings are intended to cover non-exclusive inclusions, such as a process, method, system, or device including a series of steps or units is not limited to the listed steps or units. In addition, "first" and "second" are used to distinguish different objects, not to describe a specific order.
[0023] See also Figure 1 , a flowchart of a data set conversion method provided by an embodiment of the present invention includes:
[0024] S101, obtaining a source data set, and preprocessing the source data set;
[0025] The source dataset refers to a dataset that needs to be converted, which can be a local dataset used for specific model training or an open source dataset.
[0026] The preprocessing is used to screen and deduplicate the data in the source data set. Preprocessing can reduce the impact of erroneous and redundant data on the intelligent agent conversion. The preprocessed data set can be directly input into the AI intelligent agent for data conversion.
[0027] S102, input the preprocessed source data set into the AI agent, analyze the prompt word through the large language model based on the prompt word and the built-in automation tool to generate a solution, select the solution through the AI agent decision and call the corresponding automation tool to perform the conversion operation, and obtain the converted data set;
[0028] The AI agent is an application developed based on the LangChain large language model;
[0029] The AI agent module is an application based on a large language model developed using LangChain (an open source framework for developing language model-driven applications) in Python. The AI agent can interact with the large language model.
[0030] Based on the large language model, the prompt words are deeply analyzed to generate possible solutions. The AI agent makes decision plans based on the candidate solutions and calls the built-in automation tools based on the decision plan to obtain the converted data set. When there is no corresponding automation tool, the AI agent can directly convert the data set.
[0031] The Prompt words include common prompt words and custom prompt words;
[0032] The general prompt word structure is: role, role skills, task core keywords, task objectives, task background, task scope, task solution judgment, task restriction conditions, output form and output amount.
[0033] Users can input prompt words according to the general structure as key information to guide AI agents to convert data sets, and can customize the prompt word structure for input.
[0034] The built-in automated tools are commonly used dataset format conversion tools in the field of dataset conversion, such as LabelConvert, a dataset format conversion tool for target detection and image segmentation, which supports conversion between labelme, labelImg and YOLO, VOC and COCO dataset formats. With built-in common conversion tools, these tools have higher conversion efficiency and accuracy in specific situations, and can be used together as tools for intelligent agents and called when needed.
[0035] By applying the prompt word technology, users can describe the original format, target format, attribute correspondence, etc. of the data set in detail. Since the AI agent has natural language processing and logical reasoning capabilities and is oriented towards generating target results, the AI agent can meet different data set conversion requirements based on prompt words and built-in tools without having to worry about the details of intermediate data set conversion. It has extremely high scalability and does not require the development of new conversion programs, which reduces the threshold and workload of staff.
[0036] S103: post-process the converted data set and output the converted data set.
[0037] The post-processing refers to specific processing of the data after the data set conversion, which may include data error correction, content extraction, etc.
[0038] Among them, the pre-processing includes at least data screening, deduplication and content cutting, and the post-processing includes at least valuable content extraction, data error correction and data combination.
[0039] Screening may include removing empty abnormal data and removing irrelevant redundant data, etc. Deduplication refers to removing duplicate data; content cutting refers to the fact that the large language model can only receive a limited amount of text at one time, so the input long text content must be cut into reasonable lengths and provided to the large language model in batches for processing. Valuable content extraction refers to the data set text directly generated by the large language model, which may not only contain target data set fragments but also redundant logical explanation description information. The target data set fragment content needs to be extracted and the redundant content needs to be ignored; data error correction refers to correcting obvious spelling errors and grammatical errors in the data set generated by the large language model, and finally obtaining the correct data set format; data combination is a processing process that matches the content cutting performed in the preprocessing process. Multiple short texts cut from a long text are provided to the large language model to obtain multiple output results. The multiple output results need to be combined to obtain the final complete data set.
[0040] In this embodiment, the data set conversion is performed based on the AI agent, which has strong scalability and high conversion efficiency. The user only needs to provide the original data set format and the target data set format, and clearly describe the meaning of each attribute in the data set. The user does not need to care about the intermediate conversion details, and the data set conversion can be completed intelligently. There is no need to develop a new conversion program, which greatly reduces the development workload and improves work efficiency. At the same time, the professional threshold for data set conversion work is lowered, so that people who do not know how to write tools and technical details can also complete the data set conversion work, greatly reducing labor costs and technical costs.
[0041] In one embodiment, an AI agent image is built through Dockerfile, and the AI agent image is deployed on a cloud server.
[0042] Specifically, first use the official Python as the base image, set the working directory, copy the current directory content to the working directory in the container, then install dependencies, expose ports, and finally run the main program of the AI agent.
[0043] Deploying data set conversion on a cloud platform can provide stronger computing power for data set conversion, better respond to massive data needs, and effectively improve the efficiency of data set conversion.
[0044] It should be understood that the serial numbers of the steps in the above embodiments do not imply a sequence of execution, and the execution sequence of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
[0045] Figure 2 A schematic diagram of a system for data set conversion provided by an embodiment of the present invention, the system comprising:
[0046] The data set input module 210 is used to obtain a source data set, preprocess the source data set, and input the preprocessed source data set into the AI agent;
[0047] The data set conversion module 220 is used to analyze the prompt word through the large language model to generate a solution based on the Prompt prompt word and the built-in automation tool, select the solution through the AI agent decision and call the corresponding automation tool to perform the conversion operation to obtain the converted data set;
[0048] The Prompt words include common prompt words and custom prompt words;
[0049] The general prompt word structure is: role, role skills, task core keywords, task objectives, task background, task scope, task solution judgment, task restriction conditions, output form and output amount.
[0050] The data set output module 230 is used to post-process the converted data set and output the converted data set.
[0051] Among them, the pre-processing includes at least data screening, deduplication and content cutting, and the post-processing includes at least valuable content extraction, data error correction and data combination.
[0052] Preferably, the data set conversion module 220 includes:
[0053] The model deployment module is used to build an AI agent image through Dockerfile and deploy the AI agent image on a cloud server.
[0054] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working process of the above-described system and module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.
[0055] Figure 3 1 is a schematic diagram of a structure of an electronic device provided by an embodiment of the present invention. The electronic device is used for data set conversion. Figure 3 As shown, the electronic device 3 of this embodiment includes: a memory 310, a processor 320 and a system bus 330, and the memory 310 includes an executable program 3101 stored thereon. Those skilled in the art can understand that Figure 3 The electronic device structure shown in the figure does not constitute a limitation of the electronic device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.
[0056] Combine the following Figure 3 A detailed introduction to the various components of electronic equipment:
[0057] The memory 310 can be used to store software programs and modules, and the processor 320 executes various functional applications and data processing of the electronic device by running the software programs and modules stored in the memory 310. The memory 310 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device (such as cache data), etc. In addition, the memory 310 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.
[0058] The memory 310 includes an executable program 3101 of a data set conversion method, and the executable program 3101 can be divided into one or more modules / units, which are stored in the memory 310 and executed by the processor 320 to perform automatic data set conversion, etc. The one or more modules / units can be a series of computer program instruction segments that can complete specific functions, and the instruction segments are used to describe the execution process of the executable program 3101 in the electronic device 3. For example, the executable program 3101 can be divided into functional modules such as a data set input module, a data set conversion module, and a data set output module.
[0059] The processor 320 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. By running or executing software programs and / or modules stored in the memory 310, and calling data stored in the memory 310, it performs various functions of the electronic device and processes data, thereby monitoring the overall status of the electronic device. Optionally, the processor 320 may include one or more processing units; preferably, the processor 320 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, application programs, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 320.
[0060] The system bus 330 is used to connect the various functional components inside the computer, and can transmit data information, address information, and control information. Its types can be, for example, PCI bus, ISA bus, CAN bus, etc. The instructions of the processor 320 are transmitted to the memory 310 through the bus, and the memory 310 feeds back data to the processor 320. The system bus 330 is responsible for the data and instruction exchange between the processor 320 and the memory 310. Of course, the system bus 330 can also be connected to other devices, such as network interfaces, display devices, etc.
[0061] In the embodiment of the present invention, the executable program executed by the processor 320 included in the electronic device includes:
[0062] The preprocessed source data set is input into the AI agent. Based on the prompt words and built-in automation tools, the prompt words are analyzed by the large language model to generate solutions. The AI agent makes decisions to select solutions and calls the corresponding automation tools to perform conversion operations to obtain the converted data set.
[0063] Obtain source data sets and preprocess the source data sets;
[0064] The preprocessed source data set is input into the AI agent. Based on the prompt words and built-in automation tools, the prompt words are analyzed by the large language model to generate solutions. The AI agent makes decisions to select solutions and calls the corresponding automation tools to perform conversion operations to obtain the converted data set.
[0065] The AI agent is an application developed based on the LangChain large language model;
[0066] Post-process the transformed dataset and output the transformed dataset.
[0067] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and modules described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0068] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0069] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A dataset conversion method, It is characterized in that include: Obtain source data sets and preprocess the source data sets; The preprocessed source data set is input into the AI agent. Based on the prompt words and built-in automation tools, the prompt words are analyzed by the large language model to generate solutions. The AI agent makes decisions to select solutions and calls the corresponding automation tools to perform conversion operations to obtain the converted data set. The AI agent is an application developed based on the LangChain large language model; Post-process the transformed dataset and output the transformed dataset.
2. The method according to claim 1, It is characterized in that The pre-processing includes at least data screening, deduplication and content cutting, and the post-processing includes at least valuable content extraction, data error correction and data combination.
3. The method according to claim 1, It is characterized in that The Prompt words include general prompt words and custom prompt words; The general prompt word structure is: role, role skills, task core keywords, task objectives, task background, task scope, task solution judgment, task restriction conditions, output form and output amount.
4. The method according to claim 1, It is characterized in that The method of selecting a solution through AI agent decision-making and calling a corresponding automated tool to perform the conversion operation to obtain a converted data set also includes: Build the AI agent image through Dockerfile and deploy the AI agent image on the cloud server.
5. A system for data set conversion, It is characterized in that include: A data set input module is used to obtain a source data set, preprocess the source data set, and input the preprocessed source data set into the AI agent; The data set conversion module is used to analyze the prompt words through the large language model to generate solutions based on the prompt words and the built-in automation tools. The AI agent makes decisions to select solutions and calls the corresponding automation tools to perform conversion operations to obtain the converted data set. The AI agent is an application developed based on the LangChain large language model; The data set output module is used to post-process the converted data set and output the converted data set.
6. The system according to claim 5, It is characterized in that The pre-processing includes at least data screening, deduplication and content cutting, and the post-processing includes at least valuable content extraction, data error correction and data combination.
7. The system according to claim 5, It is characterized in that The Prompt words include general prompt words and custom prompt words; The general prompt word structure is: role, role skills, task core keywords, task objectives, task background, task scope, task solution judgment, task restriction conditions, output form and output amount.
8. The system according to claim 5, It is characterized in that The data set conversion module includes: The model deployment module is used to build an AI agent image through Dockerfile and deploy the AI agent image on a cloud server.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the computer program, the steps of a data set conversion method as described in any one of claims 1 to 4 are implemented.
10. A computer-readable storage medium storing a computer program. It is characterized in that When the computer program is executed, the steps of a data set conversion method as described in any one of claims 1 to 4 are implemented.
Citation Information
Cited By
Intelligent agent-based office automation method and related device
CN119180620A
Preprocessing method of pre-training data and electronic equipment
CN119312399A
