Multi-format data intelligent structured platform

Abstract

The invention relates to the technical field of artificial intelligence, in particular to a multi-format data intelligent structured platform. The method comprises the following steps: step 1, acquiring local storage data, and performing source data verification to verify the format validity of the source data; 2, if the data format is valid, verifying the representation result according to the source data, and carrying out data preprocessing on the representation data; step 3, according to a preprocessing representation result, using an intelligent partitioning technology to perform partitioning processing on the text, and converting the partitioned text into vectors; 4, outputting the data to a data storage library according to a document vectorization representation result, and performing target data verification to verify the format validity of the target data; and 5, if the data format is valid, transmitting and storing the data to a data storage library set by a user according to a target data representation result. The invention aims to provide a multi-format data intelligent structured platform which can provide a comprehensive solution to support the development of a generative artificial intelligence application and reduce the threshold of technical application.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a multi-format data intelligent structuring platform. Background Art

[0002] At present, generative artificial intelligence (such as large language models, LLM) and related technical applications are developing rapidly, and the demand for processing multi-format unstructured data is growing. These data include documents, images, audio and video, etc. They usually lack unified format and structured information in their original state, making them difficult to be directly used for analysis and processing by AI models.

[0003] Although there are many text entity relationship extraction technologies based on machine learning and deep learning on the market, these technologies have significant limitations when processing unstructured data. They often focus on processing a single or limited type of data format, cannot cope with data sources from multiple formats, and lack the ability to uniformly process multi-format data. For example, natural language processing (NLP) technology has been widely used in extracting key information from text data, while optical character recognition (OCR) technology is used to convert text content in images and PDF files into machine-readable formats. For audio files, speech recognition technology is the main means to achieve text processing. It should be noted that in enterprise-level applications, it is very common for various data formats (including but not limited to text, images, PDF, Office documents, and audio files, etc.) to appear at the same time, which usually requires the project team to have corresponding expertise, and they often operate independently and fail to form an integrated solution to adapt to the comprehensive processing needs of generative artificial intelligence applications for multi-format unstructured data;

[0004] In addition, the existing technology still has the following shortcomings:

[0005] 1. Lack of unified processing flow: Although different data formats have their own corresponding processing methods, there is no platform that can integrate these technologies and implement unified processing flow operations such as data cleaning, slicing, and vectorization.

[0006] 2. Difficulty in integration with generative AI: Existing processing flows often cannot be directly applied to generative AI models (such as LLM), resulting in the need for additional steps or tools for data conversion, which increases the complexity of data processing. In addition, due to the lack of automated and standardized processes, data conversion is difficult to support the requirements of generative AI models.

[0007] 3. High application threshold: The existing image recognition, audio recognition, text relationship extraction and other technologies have high application thresholds, requiring the team to have certain professional knowledge

[0008] These limitations not only limit the development potential of generative AI applications, but also increase the complexity and cost of technology applications.

[0009] Therefore, how to design a multi-format data intelligent structuring platform that can solve the above technical problems is a technical problem that needs to be solved. Summary of the invention

[0010] In order to solve the above problems, the purpose of the present invention is to provide a multi-format data intelligent structuring platform that can provide a comprehensive solution to support the development of generative artificial intelligence applications and lower the threshold for technology application.

[0011] To achieve the above object, the present invention adopts the following technical solution: Step 1: Ingesting locally stored data and performing source data verification to verify the format validity;

[0012] Step 2: If the data format is valid, verify the representation result based on the source data, and use partitioning and cleaning techniques to pre-process the representation data; if the data format is wrong and / or invalid, the source data enters error processing;

[0013] Step 3: Based on the preprocessing representation results, use the intelligent segmentation technology to segment the text, and use the sentence-transformers model to convert the segmented text into vectors;

[0014] Step 4: Based on the document vectorization result, output the data to the data repository and perform target data verification to verify the format validity;

[0015] Step 5: If the data format is valid, the result is expressed according to the target data and transmitted and stored in the data repository set by the user; if the data format is wrong and / or invalid, the target data enters error processing.

[0016] Furthermore, according to the data format in the data repository, NLP is used to process text data, OCR technology is used to parse PDF and images, Whisper model is used to parse audio, Pandoc tool is used to process Office documents to extract text, and the text is partitioned. Based on the logical structure of the document, the document is split into independent structural elements, such as "title, paragraph, table, etc.", and the redundant information of the partition result is cleaned up, such as "header, footer and blank content, etc." redundant information, and other words without specific document category meaning are deleted.

[0017] Furthermore, the data preprocessing is specifically as follows: according to the data format in the data repository, NLP is used to process text data, OCR technology is used to parse PDF and images, Whisper model is used to parse audio, Pandoc tool is used to process Office documents to extract text, the text is partitioned, and based on the logical structure of the document, the document is split into independent structural elements, such as "title, paragraph, table, etc.", and the redundant information of the partition result is cleaned up, such as "header, footer and blank content, etc." redundant information, and other words without specific document category meanings are deleted.

[0018] Furthermore, the error handling is specifically as follows: the platform monitors the source data verification representation results and the target data representation results, marks the error and / or invalid data as "invalid", triggers the error handling process, and reprocesses or discards the data according to the above steps

[0019] The present invention has the following beneficial effects:

[0020] 1-Multimodal data support: The present invention can process unstructured data in various formats, including but not limited to text, images, PDF, Office documents, and audio files. According to the different formats of the input data, the system will automatically select the most suitable parsing method to ensure that the data can be processed efficiently. Text data is parsed using NLP technology, image and PDF data are extracted using OCR technology, and audio files are converted using speech recognition technology. This multimodal data support function enables the platform to cope with various complex data scenarios, significantly improving the versatility and applicability of the platform.

[0021] 2- Unified structured data generation process: This invention integrates multiple technologies such as NLP, OCR, and speech recognition into one platform to establish a unified data structuring process. This process greatly simplifies the conversion steps of unstructured data to structured data, reduces the complexity of manual intervention and multi-tool switching, and improves the accuracy and consistency of data analysis. Through this unified process, not only the data processing speed is accelerated, but also the uniformity of various data processing results is ensured, providing high-quality input data for subsequent generative AI applications.

[0022] 3- Seamlessly connect generative AI applications: The structured data processed by the present invention through steps such as cleaning, slicing, and vectorization can be directly used in the application of generative AI models (such as large language models LLM), including RAG (retrieval enhanced generation) and fine-tuning operations. By deeply cleaning and slicing the data, this platform ensures that the model can learn from high-quality, clean data, thereby improving the application effect and accuracy of the AI ​​model, and providing a solid data foundation for users to efficiently model in generative AI application scenarios.

[0023] 4- Automation and efficiency improvement: This invention achieves a high level of automation in the field of unstructured data processing, especially in the analysis of multimodal data, and can quickly and accurately convert complex text, images, audio and other data into structured information. With the help of NLP and deep learning technology in the platform, the system can intelligently identify key information in various types of data and automatically complete the data structuring process. This automated design greatly reduces the user's operating burden, improves data processing efficiency, and reduces the risk of errors in data processing.

[0024] 5- Low threshold visual operation: The present invention provides a user-friendly visual interface, which allows users to easily operate without having a deep technical background. On the interface, users can complete operations such as data set input, workflow definition, data cleaning, data slicing and vectorization. Through the concise and clear graphical interface design, users can intuitively see the various steps of data processing, further lowering the threshold for using the system, so that even users with non-technical backgrounds can efficiently use the platform for data processing.

[0025] 6- Local deployment and data security assurance: To cope with scenarios with high data privacy and security requirements, the present invention supports local deployment of the platform. Data is processed in the local environment, avoiding the process of transmitting to external servers through the network, thereby effectively reducing the risk of data leakage. The present invention provides users with a highly secure data processing environment through the local deployment function, ensuring the security and compliance of data during the processing process. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a schematic diagram of the present invention.

[0027] Description of reference numerals: DETAILED DESCRIPTION

[0028] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0029] Step 1: Ingest locally stored data and perform source data verification to verify its format validity;

[0030] Step 2: If the data format is valid, verify the representation result based on the source data, and use partitioning and cleaning techniques to pre-process the representation data; if the data format is wrong and / or invalid, the source data enters error processing;

[0031] Step 3: Based on the preprocessing representation results, use the intelligent segmentation technology to segment the text, and use the sentence-transformers model to convert the segmented text into vectors;

[0032] Step 4: Based on the document vectorization result, output the data to the data repository and perform target data verification to verify the format validity;

[0033] Step 5: If the data format is valid, the result is expressed according to the target data and transmitted and stored in the data repository set by the user; if the data format is wrong and / or invalid, the target data enters error processing.

[0034] Furthermore, according to the data format in the data repository, NLP is used to process text data, OCR technology is used to parse PDF and images, Whisper model is used to parse audio, Pandoc tool is used to process Office documents to extract text, and the text is partitioned. Based on the logical structure of the document, the document is split into independent structural elements, such as "title, paragraph, table, etc.", and the redundant information of the partition result is cleaned up, such as "header, footer and blank content, etc." redundant information, and other words without specific document category meaning are deleted.

[0035] Furthermore, the data preprocessing is specifically as follows: according to the data format in the data repository, NLP is used to process text data, OCR technology is used to parse PDF and images, Whisper model is used to parse audio, Pandoc tool is used to process Office documents to extract text, the text is partitioned, and based on the logical structure of the document, the document is split into independent structural elements, such as "title, paragraph, table, etc.", and the redundant information of the partition result is cleaned up, such as "header, footer and blank content, etc." redundant information, and other words without specific document category meanings are deleted.

[0036] Furthermore, the error handling is specifically as follows: the platform monitors the source data verification representation results and the target data representation results, marks the erroneous and / or invalid data as "invalid", triggers the error handling process, and reprocesses or discards the data according to the above steps.

[0037] Furthermore, for the workflow definition, the platform provides a code-free interface to set rules for ingesting data; it implements regular extraction of new data and performs structured processing.

[0038] Furthermore, the platform also provides a code-free interface, and users can customize workflows according to their needs. For example, users can set a weekly schedule to extract new data from the local database and perform structured processing. The system automatically executes the process according to the rules defined by the user. Through code-free interface operations, the task automation process of multiple links such as data ingestion, preprocessing, data cleaning, vectorization, and output can be completed. The system can set task scheduling according to user needs to ensure that each link can be executed and completed as planned.

[0039] Furthermore, the platform monitors the verification phase of data processing in real time. Once data errors or invalid data appear, the error handling process will be automatically triggered, marking them as "invalid" and reprocessing or discarding them to ensure the quality of the output data.

[0040] In this embodiment, the details are as follows:

[0041] First, the system uses specific parsing technologies according to different data formats. For example, NLP is used to process text data, OCR technology is used to parse PDF and images, Whisper model is used to parse audio, and Pandoc tool is used to process Office documents to process unstructured data in various formats.

[0042] Secondly, the document is logically segmented through intelligent segmentation technology, and data cleaning technology is used to remove irrelevant noise information (such as headers, footers, redundant content, etc.) to clean and preprocess the unstructured data. This automated data cleaning mechanism not only improves the quality of structured data, but also reduces the complexity of subsequent processing.

[0043] Secondly, by using an embedding model to convert text content into vector representation, it provides support for subsequent similarity search, machine learning model training, and AI applications such as fine-tuning and RAG. This intelligent segmentation and vectorization based on content logic not only makes the data have a high-quality structured form, but also lays the foundation for in-depth data analysis and efficient model processing.

[0044] Finally, users can define the output format and target location of the data according to their needs to ensure that the generated structured data can be seamlessly connected to subsequent AI application systems.

[0045] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0046] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0047] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0048] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0049] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.

Claims

1. A multi-format data intelligent structuring platform, characterized in that: The following steps are involved: Step 1: Ingest locally stored data and perform source data verification to verify its format validity; Step 1: If the data format is valid, the representation result is verified according to the source data, and the representation data is preprocessed using partitioning and cleaning techniques; if the data format is wrong and / or invalid, the source data enters error processing; Step 2: Based on the preprocessing representation results, use intelligent segmentation technology to segment the text, and use the sentence-transformers model to convert the segmented text into vectors; Step 3: Based on the document vectorization result, output the data to the data repository and perform target data verification to verify the format validity; Step 4: If the data format is valid, the result is expressed according to the target data and transmitted and stored in the data repository set by the user; If the data format is malformed and / or invalid, the target data enters error processing.

2. A multi-format data intelligent structuring platform according to claim 1, characterized in that: The data preprocessing is specifically as follows: according to the data format in the data repository, NLP is used to process text data, OCR technology is used to parse PDF and images, Whisper model is used to parse audio, Pandoc tool is used to process Office documents to extract text, the text is partitioned, and based on the logical structure of the document, the document is split into independent structured elements, such as "title, paragraph, table, etc.", and the redundant information of the partition result is cleaned up, and redundant information such as "header, footer and blank content, etc." and other words without specific document category meanings are deleted.

3. The multi-format data intelligent structuring platform according to claim 1, characterized in that: The error handling is specifically as follows: the platform monitors the source data verification representation results and the target data representation results, marks the erroneous and / or invalid data as "invalid", triggers the error handling process, and reprocesses or discards the data according to the above steps.

Images