Method for processing unstructured information of text by calling artificial intelligence API (Application Program Interface) by utilizing python

Abstract

The invention provides a method for processing text unstructured information by calling an artificial intelligence API (Application Program Interface) by utilizing Python. The method comprises the following steps: S1, automatically extracting text, table and picture information from a PDF (Portable Document Format) file by utilizing Python and a related library; s2, performing a data analysis and storage technology based on a GPT model by using a function, and outputting a CSV file; and S3, performing table merging and data cleaning on the CSV file. According to the method for processing the unstructured information of the text by calling the artificial intelligence API interface through the python, an automatic process for extracting the text data in batches and a full-automatic process design from PDF document extraction to unstructured data generation are provided, manual intervention is reduced, and the processing efficiency and accuracy are improved. Especially for processing of large-scale documents, the working efficiency is remarkably improved through the process.

Description

Technical Field

[0001] The present invention relates to the technical field of large language model natural language processing, and in particular to a method for processing text unstructured information by calling an artificial intelligence API interface using Python. Background Art

[0002] At present, with the advent of the big data era, data extraction and processing are becoming increasingly important in various fields. Especially in the academic and research fields, extracting effective information from a large number of PDF documents is crucial for tasks such as data analysis, knowledge discovery and information retrieval. However, due to the complexity and layout diversity of the PDF format, automatic extraction of text and data from it faces certain challenges. Traditional text extraction methods are difficult to accurately identify and extract text content from these elements, while data extraction methods often rely on manual operations, which are not only time-consuming and labor-intensive, but also inefficient. There are many shortcomings in existing PDF data extraction technologies. First, many existing tools cannot effectively handle these complex structures, resulting in poor accuracy and completeness of extracted data. Secondly, existing tools usually lack intelligent and automated processing capabilities, rely on rule or template matching methods, are difficult to adapt to PDF documents of different types and formats, and are difficult to meet batch text processing. Therefore, developing a technology that can automatically process PDF files and can accurately extract the required data is of great significance to improving research efficiency and data utilization. The emergence of ChatGPT in recent years has brought great possibilities to artificial intelligence, among which text mining in API models has also brought a more convenient way to natural language processing. The text mining technology in the API model can more efficiently process and analyze the text information in PDF documents. These models have powerful semantic understanding and text processing capabilities, and can automatically identify and extract key data from PDF documents.

[0003] For example, the invention patent with application number 201811211967.9 discloses a complaint responsibility identification system based on text mining technology, including data layer, platform layer, middleware, service layer and business layer; the data layer includes relational database, distributed file system and non-relational database, and the data layer regularly extracts data from the 95598 business support system and marketing business application system of the State Grid headquarters, and stores the data in categories; the platform layer includes public business components, business components, business flow engine components and Spark; the middleware includes Zookeeper, image recognition algorithm plug-in and decision tree algorithm plug-in, which calculate the data of the data layer and model it through data modeling and data mining to realize data analysis and mining; the service layer includes unified coding service, unified cache service, bidding model calculation service, process service and file storage service; the business layer includes responsibility identification, image recognition and automatic generation of work orders. However, this method has poor accuracy and complex structure. Summary of the invention

[0004] In order to solve the technical problems of poor accuracy and difficulty in extracting unstructured information in automated batch processing of PDF documents, the present invention proposes a method for processing unstructured information of text by using Python to call an artificial intelligence API interface, and uses Python programming language, ChatGPT API model and multiple powerful libraries, such as PyPDF2, PDFminer, Pytesseract and PIL, to realize efficient batch data extraction of PDF files. In order to achieve the above purpose, the technical solution of the present invention is implemented as follows: A method for processing unstructured information of text by using Python to call an artificial intelligence API interface, the steps comprising:

[0005] S1: Automatically extract text, table and image information from PDF files using Python and related libraries;

[0006] S2: Use functions to perform data parsing and preservation technology based on the GPT model and output CSV files;

[0007] S3: Merge tables and clean data in CSV files.

[0008] The related libraries described in step S1 include: PyPDF2, PDFMiner, Pytesseract, Pdfplumber and tqdm libraries.

[0009] The method for extracting text, table and image information from a PDF file in step S1 is:

[0010] S11: Use PyPDF2 to read the text and extract each text block from the PDF;

[0011] S12: extracting the page layout in combination with the PDFMiner library to obtain the text information in the text block;

[0012] S13: extracting text information on the page according to the page layout, and extracting text from tables and images in a structured format in the PDF file;

[0013] S14: Integrate the complete text content on the page and convert the text information into JSON format according to the logical structure of the web page content;

[0014] S15: Encode the generated JSON data and save it as a .json file.

[0015] The method of extracting page layout in combination with the PDFMiner library in step S12 is:

[0016] If PDFMiner reads successfully, first, use the extract_pages function to traverse each page of the PDF file and extract the page layout information from the specified PDF file; second, during the traversal process, check each page layout element, such as a text container, and extract its text content; finally, organize the extracted text content into paragraphs to obtain text information;

[0017] If PDFMiner fails to read, the name of the PDF file is extracted and saved in TXT format.

[0018] The method of extracting text from the table and image of the PDF file in a structured format in step S13 is:

[0019] Convert the given PDF file to text using the text_extraction function to extract text and formatting information by parsing text blocks, images, and tables. Convert the given PDF file to text using the PyPDF2 library.

[0020] The method of performing data analysis and storage technology based on the GPT model in step S2 is:

[0021] S21: Use the correct_line_breaks and remove_special_characters functions to fix the continuous spaces and incorrectly segmented words in the text and complete the preprocessing of the .json file;

[0022] S22: Set keywords, use the nltk library to obtain synonyms of keywords, and filter out text fragments containing specific keywords from the extracted text through regular expressions and string operations;

[0023] S23: Segment the text and call the OpenAI API model to obtain the GPT model response;

[0024] S24: parsing the GPT model response, extracting the table data that conforms to the preset format and organizing the data;

[0025] S25: Store the sorted data to obtain a CSV file.

[0026] The functions described in step S2 include re function, os function, tiktoken function, pandas function, csv function, json function, nltk function, openai function, tqdm function and numpy function.

[0027] The method for segmenting the text in step S23 is as follows:

[0028] Use the split_strings and merge_contents functions to split the string into at least one part with the number of tokens, and merge the split parts into at least one fragment, where the number of tokens is less than the number of tokens set to read by the API. Then, use the get_gpt_response function to call the OpenAI API model to obtain the GPT response, and filter and obtain the corresponding unstructured information based on the keyword matching algorithm and the GPT response.

[0029] The method of extracting the table data conforming to the preset format and arranging the data in step S24 is as follows:

[0030] Parse the GPT model response and extract tabular data that conforms to the preset format. Secondly, split the long text to adapt to the token limit of the GPT model and merge multiple responses into a single data set. Use the Pandas library to further organize and format the parsed single data set, including operations such as expanding rows and rearranging, and finally generate a structured DataFrame.

[0031] The method for merging tables and cleaning data in CSV files in step S3 is:

[0032] Using Python code, merge all CSV tables together, import Pandas and re libraries for data cleaning, delete NaN values, remove data containing garbled characters, remove brackets and their contents, and convert data into specific units.

[0033] The beneficial effects of the present invention are as follows: the method provided by the present invention for processing text unstructured information by using Python to call an artificial intelligence API interface provides an automated process for batch extraction of text data, and a fully automated process design from PDF document extraction to unstructured data generation, which reduces manual intervention and improves processing efficiency and accuracy. In particular, for the processing of large-scale documents, this process significantly improves work efficiency.

[0034] Using advanced regular expressions and natural language processing technology, the text in the PDF document is automatically cleaned and formatted, including correcting line breaks, removing special characters and garbled characters, and removing references. This process improves the accuracy and efficiency of subsequent text analysis. The keyword set is automatically expanded through word form restoration and synonym expansion to capture relevant information in the document. This method can effectively improve the coverage and accuracy of information retrieval, especially when dealing with professional literature and technical documents.

[0035] The GPT model is used to generate unstructured information, such as tabular data, based on the preprocessed text. This not only demonstrates the advanced application of natural language processing technology, but also reflects the innovative application of artificial intelligence in the field of automatic document analysis. At the same time, multiple output formats (such as CSV and text files) are provided to meet different analysis needs. In particular, the CSV output of unstructured data facilitates subsequent data analysis and application and improves the availability of data.

[0036] When extracting data, the prompt set in the present invention can also be flexibly modified, and keywords can be set and keyword extraction processes can be written according to the research content. Other options can also be tried for the selection of API models and the number of tokens, and modifications can be made on the original basis, which is also convenient for later upgrades and maintenance. These innovations demonstrate the advanced technical applications of this method in the fields of document automation processing, information extraction, and data analysis, and have high practical value and broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0038] Figure 1 The present invention is a flowchart of extracting PDF text information.

[0039] Figure 2 This is a flowchart of extracting unstructured information and cleaning data according to the present invention. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] like Figure 1 As shown, a method for processing text unstructured information by using Python to call an artificial intelligence API interface includes the following steps:

[0042] S1: Use Python and related libraries to automatically extract text, tables, images and other information from PDF files.

[0043] Specifically, the related libraries include: PyPDF2, PDFMiner, Pytesseract, Pdfplumber, and tqdm library. First, use PyPDF2 to read the text, then use PDFMiner to analyze the content and layout of the text, and use Pytesseract to perform OCR processing. The specific steps are:

[0044] S11: Use PyPDF2 to read the text and extract each text block from the PDF;

[0045] S12: Combined with PDFMiner library to extract page layout.

[0046] If PDFMiner reads successfully, first, it uses the extract_pages function to traverse each page of the PDF file and extract the page layout information from the specified PDF file; second, during the traversal process, it checks each page layout element, such as a text container, and extracts its text content (LTTextContainer); finally, it organizes the extracted text content into paragraphs for further processing or analysis.

[0047] If PDFMiner fails to read, the name of the PDF file is extracted and saved in TXT format.

[0048] S13: extracting text information on the page according to the page layout, and extracting text from tables and images in a structured format in the PDF file;

[0049] Used to extract text and tables from PDF files, and perform image processing. Use the text_extraction function to parse element objects (text blocks, images, tables and other structural information) to extract text and format information, and convert the given PDF file to text. Use the PyPDF2 library to convert the given PDF file to text. Count the number of tokens in a text string.

[0050] S14: Integrate the complete text content on the page. According to the logical structure of the web page content, convert the text information into JSON format, such as titles, paragraphs, lists, etc.

[0051] S15: Encode the generated JSON data (such as UTF-8), and finally save it as a .json file.

[0052] Convert all files in a given directory into cached JSON files, create a folder for failed reads, and save the failed PDF file path to the specified TXT directory. Use the tqdm library to display a progress bar when processing multiple files to enhance the user interaction experience. Increase fault tolerance through the try-except structure. The try module is the normal execution code that converts the PDF file into JSON format. If an exception occurs in the try module code, the except module automatically jumps to the except block to execute the error handling code, print the error message, and record the PDF file as a failure. Save it in the failed_pdfs list for subsequent centralized processing, and also ensure that the failure of a single file processing will not affect the entire batch processing process.

[0053] S2: Data parsing and preservation technology based on the GPT model, outputting CSV files.

[0054] like Figure 2 As shown in the figure, the JSON file is processed by setting keywords and writing specific prompts, and the ChatGPT API model is used for efficient data extraction. Writing specific prompts is based on the interpretation of keywords, which makes it easier for AI to understand and locate; secondly, the extraction process is written for text information, and the extraction range is determined according to keywords, and finally the corresponding data is extracted. In this process, the text model in GPT is used for extraction based on keywords and specific prompts. After the data extraction is completed, the GPT model is used to perform semantic analysis on the data, automatically identify and organize key information, thereby improving the quality and usability of data extraction. When executing the script, the script CSV file processor is used to organize the content of the DataFrame table, and the CSV table file is saved after the script is executed.

[0055] The functions applied in step S2 include custom functions and third-party libraries, including re (regular expression processing), os (operating system interface), tiktoken (token counting), pandas (data processing), csv (CSV file processing), json (JSON data processing), nltk (natural language processing), openai (OpenAI API call), tqdm (progress bar display) and numpy (mathematical calculation).

[0056] S21: Use the correct_line_breaks and remove_special_characters functions to repair the continuous spaces and incorrectly segmented words in the text and complete the JSON text preprocessing.

[0057] First, when extracting, we use regular expression algorithms to solve line breaks, special symbols, garbled characters, etc. We use the correct_line_breaks and remove_special_characters functions to repair the continuous spaces and incorrectly segmented words in the text, remove the extra spaces and special characters and garbled characters in the text, and only keep the ASCII characters. In addition, the data corresponding to the keywords generally appear in the main text, so the reference part is removed. These operations are aimed at cleaning up the text data so that information can be extracted and analyzed more accurately.

[0058] S22: Set keywords, use the nltk library to obtain synonyms of keywords, and filter out text fragments containing specific keywords from the extracted text through regular expressions and string operations.

[0059] Set keywords and filter based on keywords. Use regular expressions and string operations to filter out text fragments containing specific keywords from the extracted text. Use the Natural Language Toolkit for keyword expansion, including lemmatization and synonym expansion, to enhance the model's ability to detect relevant information.

[0060] The set keywords are determined based on the required data. When writing prompts, you need to mark the meaning of each keyword and the process of searching in the text so that AI can extract the required data more accurately.

[0061] Because the form of keywords in the text is not uniform, when judging whether the text contains keywords, use WordNetLemmatizer and wordnet in the nltk library to obtain synonyms of keywords, restore the keywords and convert them to lowercase. Define contain_keywords and filter_by_keywords functions to check whether the text contains the specified keywords and filter the text list according to the keywords. At the same time, when writing the prompt project, filter the text list according to the keywords.

[0062] S23: Segment the text and call OpenAI’s API model to obtain the GPT model response.

[0063] Use the split_strings and merge_contents functions to split the string into multiple parts with no more than the specified number of tokens, and merge them into segments with no more than the specified number of tokens. The API sets the number of tokens to be read to 3000 (which can be adjusted according to the model and prompt content). Use the get_gpt_response function to call OpenAI's API model to get the GPT response. Extract unstructured information and obtain the corresponding unstructured information based on the keyword matching algorithm and API model response.

[0064] S24: Parse the GPT model response, extract the tabular data that conforms to the preset format, and organize the data.

[0065] The GPT model response is parsed to extract tabular data that conforms to the preset format. Secondly, the long text is split to fit the token limit of the GPT model, and multiple responses are merged into a single data set for further analysis and processing. The parsed tabular data is further organized and formatted using the Pandas library, including operations such as expanding rows and rearranging, and finally a structured DataFrame is generated.

[0066] S25: storing the sorted data.

[0067] Parse the table data from the response text obtained from the OpenAI API model and save the data table as a CSV file for subsequent data analysis and storage. At the same time, save the response of the original GPT model to a text file TXT for review or further processing. It should be noted that the way to output and save the results is that one document corresponds to a CSV table and a TXT text file, and finally merge all the CSV file results together.

[0068] S3: Merge tables and clean data in CSV files:

[0069] Using Python code, merge all CSV tables together. Import Pandas and re libraries to clean the data, including removing NaN values, removing data with garbled characters, removing brackets and their contents, and converting data to specific units.

[0070] Finally the script performs the following steps:

[0071] S31: Data cleaning:

[0072] Define a function called clean_data, and use the data.columns method to clean the specified column definition.

[0073] S32: Unit conversion:

[0074] Check if the cleaned string contains "mV" (millivolts). If only one number is found, try to convert the number to a floating point number and divide it by 1000, then return the converted value with the unit changed from "mV" to "V" (volts). If the conversion fails (for example, the number format is incorrect), Pandas NA value is returned.

[0075] S33: Garbled code processing:

[0076] For data containing garbled characters (mainly non-data type information), it is regarded as invalid data and deleted.

[0077] S34: Brackets removed:

[0078] If the data is followed by brackets and their contents, remove the brackets and their contents and keep the data itself. Use regular expressions to remove brackets and their contents from a string.

[0079] 1) Data cleaning: for the extracted data set;

[0080] 2) Unit conversion, unify the units;

[0081] 3) Garbled code processing: Data containing garbled codes will be considered invalid data and deleted;

[0082] 4) Removal of bracket contents: If the data is followed by brackets and their contents, remove the brackets and their contents and retain the data itself.

[0083] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for processing unstructured text information by using Python to call artificial intelligence API interface. It is characterized in that the steps include: S1: Automatically extract text, table and image information from PDF files using Python and related libraries; S2: Use functions to perform data parsing and preservation technology based on the GPT model and output CSV files; S3: Merge tables and clean data in CSV files.

2. According to claim 1, the method for processing text unstructured information by using Python to call artificial intelligence API interface, It is characterized in that The related libraries described in step S1 include: PyPDF2, PDFMiner, Pytesseract, Pdfplumber and tqdm libraries.

3. The method for processing text unstructured information by using Python to call an artificial intelligence API interface according to claim 2, It is characterized in that The method for extracting text, table and image information from a PDF file in step S1 is: S11: Use PyPDF2 to read the text and extract each text block from the PDF; S12: extracting the page layout in combination with the PDFMiner library to obtain the text information in the text block; S13: extracting text information on the page according to the page layout, and extracting text from tables and images in a structured format in the PDF file; S14: Integrate the complete text content on the page and convert the text information into JSON format according to the logical structure of the web page content; S15: Encode the generated JSON data and save it as a .json file.

4. The method for processing text unstructured information by using Python to call an artificial intelligence API interface according to claim 3, It is characterized in that The method of extracting page layout in combination with the PDFMiner library in step S12 is: If PDFMiner reads successfully, first, use the extract_pages function to traverse each page of the PDF file and extract the page layout information from the specified PDF file; second, during the traversal process, check each page layout element and extract its text content; finally, organize the extracted text content into paragraphs to obtain text information; If PDFMiner fails to read, the name of the PDF file is extracted and saved in TXT format.

5. The method for processing text unstructured information by using Python to call an artificial intelligence API interface according to claim 3 or 4, It is characterized in that The method of extracting text from the table and image of the PDF file in a structured format in step S13 is: Convert the given PDF file to text using the text_extraction function to parse text blocks, images, and tables etc. to extract text and formatting information. Convert the given PDF file to text using the PyPDF2 library.

6. The method for processing text unstructured information by using Python to call an artificial intelligence API interface according to claim 5, It is characterized in that The method of performing data analysis and storage technology based on the GPT model in step S2 is: S21: Use the correct_line_breaks and remove_special_characters functions to fix the continuous spaces and incorrectly segmented words in the text and complete the preprocessing of the .json file; S22: Set keywords, use the nltk library to obtain synonyms of keywords, and filter out text fragments containing specific keywords from the extracted text through regular expressions and string operations; S23: Segment the text and call the OpenAI API model to obtain the GPT model response; S24: parsing the GPT model response, extracting the table data that conforms to the preset format and organizing the data; S25: Store the sorted data to obtain a CSV file.

7. The method for processing text unstructured information by using Python to call an artificial intelligence API interface according to claim 6, It is characterized in that The functions described in step S2 include re function, os function, tiktoken function, pandas function, csv function, json function, nltk function, openai function, tqdm function and numpy function.

8. The method for processing text unstructured information by using Python to call an artificial intelligence API interface according to claim 7, It is characterized in that The method for segmenting the text in step S23 is as follows: Use the split_strings and merge_contents functions to split the string into at least one part with the number of tokens, and merge the split parts into at least one fragment, where the number of tokens is less than the number of tokens set to read by the API. Then, use the get_gpt_response function to call the OpenAI API model to obtain the GPT response, and filter and obtain the corresponding unstructured information based on the keyword matching algorithm and the GPT response.

9. The method for processing text unstructured information by using Python to call an artificial intelligence API interface according to claim 8, It is characterized in that The method for extracting tabular data in a preset format and organizing the data in step S24 is as follows: parsing the GPT model response to extract tabular data in a preset format, and then splitting the long text to adapt to the token restrictions of the GPT model, and merging multiple responses into a single data set; using the Pandas library to further organize and format the parsed single data set, including operations such as expanding rows and rearranging, and finally generating a structured DataFrame.

10. The method for processing text unstructured information by using Python to call an artificial intelligence API interface according to any one of claims 7 to 9, It is characterized in that The method for merging and cleaning the CSV files described in step S3 is: using Python code to merge all the CSV files together, importing the Pandas library and the re function to clean the data, and the data cleaning includes deleting NaN values, removing data containing garbled characters, removing brackets and their contents, and converting data into specific units.

Images