Data processing pipeline method, system and equipment based on large language model and storage medium

Abstract

The invention discloses a data processing pipeline method, system and device based on a large language model and a storage medium. The method comprises the steps of collecting data; processing the data; analyzing the data by using a large model in combination with semantic analysis, text clustering and a classification strategy to obtain hierarchical data; establishing an intelligent data processing pipeline based on the hierarchical data; and performing information extraction and conversion on the data of the intelligent data processing pipeline to obtain a labeled data set. According to the method, the defects and pain points of existing enterprise large model data processing are overcome, the data processing and synthesis efficiency is improved, and data processing can be intelligently and efficiently realized.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data processing pipeline method, system, device and storage medium based on a large language model. Background Art

[0002] In this digital age, data has become the core resource for enterprise big model competition. With the popularization of the Internet and the continuous development of artificial intelligence technology, the amount of data that needs to be processed to build enterprise big models has shown an explosive growth trend. These data contain valuable information such as the company's operations, customer behavior, and corporate knowledge, and contain huge commercial value. However, traditional data processing methods can no longer meet the stringent requirements of enterprise big models for the quality, diversity, and accuracy of training data. There is an urgent need for an efficient and intelligent data intelligent processing method and system that can quickly process a large amount of enterprise data and extract valuable information from it. It is hoped that this method and system can intelligently identify and analyze data patterns and trends, provide decision support and business insights for enterprises, and guide the production and operation process of enterprises.

[0003] The difference between the patent "A method and device for data classification prediction based on a large language model" (CN202410405867.9) and the present invention is: the patent processes data through a large language model, first decomposing the task into multiple levels, then constructing interactive request prompt information according to the levels, and finally determining and outputting the predicted category path to improve the accuracy. The present invention is a method and system for intelligent processing of enterprise data based on a large language model and a series of data intelligent processing algorithm strategies. Data classification prediction is a link in the data processing pipeline and a technology for implementing a data stratification module. The present invention has a higher dimension and includes a series of processes such as data intelligent classification and data intelligent processing pipeline. It has the characteristics of process orchestration and dynamic expansion. The goal is to provide a platform tool for data intelligent processing, realize field data intelligent processing for several enterprises, and support the implementation of enterprise large models for multiple enterprises.

[0004] The differences between "Data governance system, method, equipment and medium based on natural language processing" (patent number: CN202410465219.2) and the present invention in terms of intelligent data processing are as follows: This patent is a technology for medical data governance, which aims to improve data reading capabilities and reduce misrecognition of NLP solutions when processing medical texts. It includes four trained natural language processing models: part-of-speech recognition, entity similarity calculation, medical unified coding and variable extraction. After reading the original medical data, these models are used to obtain specific results of word segmentation data sets and various types of data. The present invention is a method and system for intelligent processing of enterprise data based on a large language model and a series of data intelligent processing algorithm strategies. It also has the characteristics of process orchestration, dynamic expansion, etc., and is more scalable.

[0005] Although there are a variety of data processing tools and technologies in the industry, they still face many challenges and defects when processing large enterprise model data:

[0006] Insufficient intelligence: Existing technologies often rely on manually preset rules or models for data processing, lack the ability to learn and adapt to changes independently, and find it difficult to accurately capture the complex relationships and potential value of data.

[0007] Lack of domain data construction solutions: The data required by the big model is different from the data stored in the traditional data center. There is a lack of solutions for intelligent construction of domain data, and it is impossible to truly integrate enterprise data into big model training. At the same time, there is a lack of data quality processing and evaluation tools, which affects the accuracy and effectiveness of subsequent models.

[0008] Poor scalability of data processing modules: As the scale of enterprises expands and the business develops, the demand for data processing continues to increase, and there will be scenarios where data processing pipelines need to be expanded and reused. However, existing systems are often difficult to expand flexibly and are difficult to support large-scale data processing tasks. Summary of the invention

[0009] In view of the shortcomings existing in the above problems, the present invention provides a data processing pipeline method, system, device and storage medium based on a large language model.

[0010] To achieve the above object, the present invention provides a data processing pipeline method based on a large language model, comprising:

[0011] Collect data;

[0012] Processing the data;

[0013] Analyze the data using a large model combined with semantic analysis, text clustering and classification strategies to obtain hierarchical data;

[0014] Establishing an intelligent data processing pipeline based on the hierarchical data;

[0015] Information is extracted and converted from the data of the intelligent data processing pipeline to obtain a labeled data set.

[0016] Preferably, collecting the data includes supporting access to multiple data sources, performing hierarchical processing upon access, distinguishing knowledge and task data in general / industry / enterprise fields, and implementing hierarchical management of data sources.

[0017] Preferably, processing the data includes cleaning, format conversion and deduplication of the data, and performing knowledge extraction, label stratification and question classification on the data.

[0018] Preferably, establishing an intelligent data processing pipeline based on the hierarchical data includes:

[0019] performing knowledge extraction, knowledge enhancement, and knowledge evaluation on the hierarchical data;

[0020] Performing data annotation, data enhancement, and data evaluation on the hierarchical data;

[0021] The intelligent data pipeline is established for the two types of data.

[0022] The present invention also provides a data processing pipeline system based on a large language model, comprising:

[0023] Training module, used to collect data;

[0024] A processing module, used for processing the data;

[0025] An analysis module, for analyzing the data using a large model combined with semantic analysis, text clustering and classification strategies to obtain hierarchical data;

[0026] An establishment module, for establishing an intelligent data processing pipeline based on the hierarchical data;

[0027] The labeling module is used to extract and convert information from the data of the intelligent data processing pipeline to obtain a labeled data set.

[0028] The present invention also provides a device, comprising at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the program is executed by the processing unit, the processing unit executes the above method.

[0029] The present invention also provides a storage medium storing a computer program executable by an electronic device, and when the program runs on the electronic device, the electronic device executes the above method.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] The present invention makes up for the deficiencies and pain points of existing enterprise large model data processing, improves data processing and synthesis efficiency, and can realize data processing intelligently and efficiently. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a flow chart of the data processing pipeline method based on a large language model of the present invention. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] Reference Figure 1 The present invention provides a data processing pipeline method based on a large language model, comprising:

[0035] Collect data;

[0036] Specifically, it supports access to multiple data sources, realizes real-time or batch data collection, ensures the comprehensiveness and timeliness of data, and at the same time, performs hierarchical processing during access, distinguishes knowledge and task data in general / industry / enterprise fields, and realizes hierarchical management of data sources.

[0037] Process the data;

[0038] Specifically, it integrates basic steps such as data cleaning, format conversion, deduplication, and intelligent processing components such as knowledge extraction, label stratification, and question classification to achieve secondary processing and classification of raw data, which is used to extract potential information and knowledge and realize the classification of knowledge / task data.

[0039] Use big models combined with semantic analysis, text clustering and classification strategies to analyze data and obtain hierarchical data;

[0040] Specifically, we use big model technology, combined with question semantic analysis, text clustering and intelligent classification strategies, to intelligently divide tasks and data systems, realize data hierarchical analysis, and provide prerequisites for subsequent data selection / matching.

[0041] Establish intelligent data processing pipeline based on hierarchical data;

[0042] Extract and convert information from the data of the intelligent data processing pipeline to obtain a labeled data set.

[0043] Specifically, for the data after knowledge classification, knowledge extraction, knowledge enhancement, and knowledge evaluation are performed respectively; and for the data after task classification, data annotation, data enhancement, and data evaluation are performed. An intelligent data processing pipeline is established for the two types of data, combined with low-code tools to achieve graphical processing flow orchestration, automatically extract and build effective data features based on business needs and data characteristics, extract and convert data information, and intelligently and automatically build high-quality field annotation data sets.

[0044] Intelligent data processing pipeline: Knowledge / data system: Provides an intuitive data visualization interface and powerful analysis tools to help enterprise users quickly gain insights into data distribution and quality scores, and provide data input for subsequent data selection / proportioning.

[0045] Specifically, knowledge data and knowledge system construction: data access, data intelligent processing, knowledge hierarchical system construction, knowledge extraction, knowledge enhancement, knowledge evaluation and knowledge system construction. Task data and task system construction: data access, data intelligent processing, task hierarchical system construction, data labeling, data enhancement, data evaluation and data system construction.

[0046] The specific objectives of the present invention include:

[0047] Improve data processing efficiency: By optimizing the data processing process, realize layered processing of knowledge / task system, achieve rapid data collection, cleaning, conversion and loading, and shorten the data processing cycle.

[0048] Enhance the level of intelligence: Introduce advanced large models and data processing technologies, and improve the accuracy and diversity of data processing through intelligent data screening, evaluation, processing and other strategies.

[0049] Ensure data quality: Build a complete data cleaning and verification mechanism and an intelligent data quality assessment solution to effectively remove noise, redundant information and sensitive information, and ensure that the quality of processed data meets the enterprise's large model training requirements.

[0050] Improve data processing scalability: Design a modular, pluggable system architecture, combined with low-code tools, to support flexible adjustment of data processing processes and resource allocation according to business needs, meet the needs of enterprises for building large model training data in different scenarios, improve the efficiency of building large model training data, and reduce data processing investment costs.

[0051] The present invention provides an integrated solution for field data processing: integrating multiple links such as data processing, analysis, and application, establishing a full-link data management and control link, and realizing the maximum utilization of data value.

[0052] The present invention makes up for the deficiencies and pain points of existing enterprise large model data processing, improves data processing and synthesis efficiency, and can realize data processing intelligently and efficiently.

[0053] Intelligent data processing pipeline: By introducing advanced large model technology and algorithm strategies and arranging multiple processes through strategy, an intelligent data processing link is established for hot models such as rearrangement models, search models, classification models, and large language models, which improves the support efficiency of enterprise large model construction. Through low-code tools, non-technical professionals can also easily participate in the design and optimization of data processing processes, which reduces the technical threshold and improves team collaboration efficiency. Compared with traditional automated data processing scripts and manual support, the efficiency is improved by an average of 60%, and the data quality is improved by about 30%.

[0054] Data hierarchical analysis: Using big model technology, combined with problem semantic analysis and intelligent classification strategies, we automatically divide tasks and data systems to achieve data hierarchical analysis. We have established nearly 1,000+ task data systems and 50+ knowledge data systems, and have efficiently and intelligently realized enterprise data processing, analysis, and management, improving the support efficiency of enterprise big models and providing sufficient data supply for the models.

[0055] Promote in-depth mining and utilization of data value: The established data processing chain and value analysis system make the entire process from data collection to application traceable and evaluable, which helps enterprises better grasp data assets, maximize the utilization of data value, and create an enterprise big model ecosystem with data as the core driving force.

[0056] Example

[0057] Based on a project case practice scenario, we conducted intelligent data processing on nearly 1,000 customer service knowledge articles. Through multiple links such as knowledge extraction, knowledge enhancement, knowledge evaluation, and intelligent annotation, we constructed a high-quality customer service knowledge rearrangement data set. The results are as follows:

[0058]

[0059] By comparing the three solutions of manual processing, automated data processing methods, and intelligent data processing pipelines, it can be found that although the automated processing method has a higher processing efficiency and can complete data construction in nearly 2 days, its data quality is extremely poor and cannot meet the requirements of rearrangement model training. The intelligent data processing pipeline, although it takes 5 man-days / GPU (A40), can linearly shorten the processing time as resources grow, replacing manpower with resources, and the data processing quality is high, which is basically the same as the manual processing level. It greatly improves the efficiency of rearrangement data construction, provides an efficient and intelligent data link for rearrangement model training, and can also be flexibly expanded to search, recommendation and other scenario applications.

[0060] The present invention also provides a data processing pipeline system based on a large language model, comprising:

[0061] Training module, used to collect data;

[0062] A processing module, used for processing data;

[0063] The analysis module is used to analyze data using a large model combined with semantic analysis, text clustering and classification strategies to obtain hierarchical data;

[0064] Establishing a module for establishing an intelligent data processing pipeline based on hierarchical data;

[0065] The labeling module is used to extract and convert information from the data of the intelligent data processing pipeline to obtain a labeled data set.

[0066] The present invention also provides a device, comprising at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the program is executed by the processing unit, the processing unit executes the above method.

[0067] The present invention also provides a storage medium storing a computer program executable by an electronic device. When the program runs on the electronic device, the electronic device executes the above method.

[0068] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data processing pipeline method based on a large language model, characterized in that: include: Collect data; Processing the data; Analyze the data using a large model combined with semantic analysis, text clustering and classification strategies to obtain hierarchical data; Establishing an intelligent data processing pipeline based on the hierarchical data; Information is extracted and converted from the data of the intelligent data processing pipeline to obtain a labeled data set.

2. The data processing pipeline method based on a large language model according to claim 1, characterized in that: Collecting the data includes supporting access to multiple data sources, performing hierarchical processing during access, distinguishing knowledge and task data in general / industry / enterprise fields, and implementing hierarchical management of data sources.

3. The data processing pipeline method based on a large language model according to claim 2, characterized in that: Processing the data includes cleaning, format conversion and deduplication of the data, and performing knowledge extraction, label stratification and question classification on the data.

4. The data processing pipeline method based on a large language model according to claim 3, characterized in that: Establishing an intelligent data processing pipeline based on the hierarchical data includes: performing knowledge extraction, knowledge enhancement, and knowledge evaluation on the hierarchical data; Performing data annotation, data enhancement, and data evaluation on the hierarchical data; The intelligent data pipeline is established for the two types of data.

5. A data processing pipeline system based on a large language model, characterized in that: include: Training module, used to collect data; A processing module, used for processing the data; An analysis module, for analyzing the data using a large model combined with semantic analysis, text clustering and classification strategies to obtain hierarchical data; An establishment module, for establishing an intelligent data processing pipeline based on the hierarchical data; The labeling module is used to extract and convert information from the data of the intelligent data processing pipeline to obtain a labeled data set.

6. A device, characterized in that: The method comprises at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the program is executed by the processing unit, the processing unit executes the method according to any one of claims 1 to 4.

7. A storage medium, characterized in that: It stores a computer program executable by an electronic device. When the program runs on the electronic device, the electronic device executes the method according to any one of claims 1 to 4.

Images