Intelligent document analysis method of knowledge base management system

Abstract

The invention discloses an intelligent document analysis method of a knowledge base management system, and relates to the field of artificial intelligence large model multi-modal analysis. Comprising the following steps: step 1, preliminarily identifying a document format according to a document extension name, and performing corresponding analysis according to the type of the document format: performing text analysis for pdf, docx, doc and txt document formats, and extracting texts and images in the pdf document format for the pdf document format; aiming at the docx document format, analyzing the compressed package in the docx document format according to the natural sequence of the document, and extracting a text and a picture; aiming at the doc document format, identifying and extracting all text information in the doc document format; for a txt document format, character codes are recognized and processed, and various languages and symbols are extracted; the method comprises the following steps of 1, analyzing a text, 2, processing the analyzed text by utilizing a large model, namely segmenting the text, performing vectorization processing, converting the text into a numerical form which can be understood by the large model, and storing the converted text in a knowledge base for information retrieval, data analysis and knowledge mining, and 3, performing visual output on the text at the same time.

Description

Technical Field

[0001] The present invention discloses a document intelligent parsing method for a knowledge base management system, and relates to the field of artificial intelligence large model multimodal analysis. Background Art

[0002] In today's big data era, information data is growing explosively. With the continuous deepening and extensive application of information technology in work, a large amount of documents in different formats have been accumulated in actual work. Quickly and effectively searching and analyzing massive data has become an important requirement for many companies and organizations. Manual analysis methods alone are far from meeting the actual needs of work. Efficient tools must be used to greatly improve work efficiency and reduce work pressure. However, in the existing knowledge base management system, document processing and parsing technology has the following limitations:

[0003] 1. Poor format compatibility: Most systems only support documents in specific formats. There is a lack of effective parsing methods for diverse document formats such as PDF, DOC, DOCX, TXT, XLSX, etc., and it is impossible to adopt corresponding parsing strategies based on the characteristics of different documents, which affects the accuracy and efficiency of parsing.

[0004] 2. Incomplete information extraction: Existing technologies can often only extract text information from documents, while ignoring non-text content such as pictures and tables, resulting in incomplete content in the knowledge base.

[0005] 3. Inconvenient data management: The parsed document data often lacks effective organization and management, making it difficult to achieve systematic storage and rapid retrieval of knowledge.

[0006] 4. Lack of visual support: During the document parsing process, the processing and display of visual elements such as pictures are not intuitive enough, which affects users' understanding and absorption of knowledge. Summary of the invention

[0007] In view of the problems of the prior art, the present invention provides a document intelligent parsing method for a knowledge base management system, which realizes efficient parsing of documents of different formats, comprehensively extracts text, pictures and table information in the documents, and organically integrates them into the knowledge base. This method not only improves the intelligent level of document processing, but also lays a foundation for the visual display and in-depth utilization of the knowledge base.

[0008] The specific scheme proposed by the present invention is:

[0009] A document intelligent parsing method for a knowledge base management system, comprising:

[0010] Step 1: Preliminarily identify the document format according to the document extension, and perform corresponding analysis according to the type of document format: perform text analysis on pdf, docx, doc and txt document formats respectively. For pdf document format, extract text and images in pdf document format; for docx document format, parse according to the natural order of the document according to the compressed package of docx document format, extract text and images; for doc document format, identify and extract all text information in doc document format; for txt document format, identify and process character encoding, and extract various languages ​​and symbols;

[0011] Step 2: Use the big model to process the parsed text: segment the text and perform vectorization, convert the text into a numerical form that the big model can understand, and store the converted text in the knowledge base for information retrieval, data analysis, and knowledge mining.

[0012] Step 3: Visualize the text at the same time.

[0013] Furthermore, in step 1 of the document intelligent parsing method of the knowledge base management system, for the pdf document format, each page in the pdf document format is converted into a tree structure, each node in the page represents an element, and the elements include text, images and tables. The content of the tree structure is traversed to extract the elements. When an image element is encountered, the obtained original image is named with a unique identifier uuid and uploaded to the MinIO object storage service. The storage path of the image is then embedded in the text content of the tree structure in a custom tag format to achieve the combination of image and text.

[0014] Furthermore, in step 1 of the document intelligent parsing method of the knowledge base management system, when parsing images in the docx document format, the storage path of the image is also embedded into the text content in a custom tag format to achieve the combination of image and text.

[0015] Furthermore, in step 1 of the document intelligent parsing method for a knowledge base management system, character encoding is identified and processed line by line for the txt document format, and various languages ​​and symbols are extracted.

[0016] The present invention also provides a document intelligent parsing device for a knowledge base management system, comprising a recognition and parsing module, a large model processing module and a display module.

[0017] The recognition and parsing module preliminarily identifies the document format according to the document extension, and performs corresponding parsing according to the type of document format: it performs text parsing on pdf, docx, doc and txt document formats respectively. For pdf document format, it extracts text and images from pdf document format; for docx document format, it parses the compressed package of docx document format according to the natural order of the document to extract text and images; for doc document format, it recognizes and extracts all text information in doc document format; for txt document format, it recognizes and processes character encoding and extracts various languages ​​and symbols;

[0018] The big model processing module uses the big model to process the parsed text: segment the text and perform vector processing, convert the text into a numerical form that the big model can understand, and store the converted text in the knowledge base for information retrieval, data analysis, and knowledge mining.

[0019] The display output module also outputs the text visually.

[0020] Furthermore, the identification and parsing module of the document intelligent parsing device of the knowledge base management system targets the pdf document format, converts each page in the pdf document format into a tree structure, each node in the page represents an element, and the elements include text, images and tables. The content of the tree structure is traversed to extract the elements. When an image element is encountered, the obtained original image is named with a unique identifier uuid and uploaded to the MinIO object storage service. The storage path of the image is then embedded in the text content of the tree structure in a custom tag format to achieve the combination of image and text.

[0021] Furthermore, when the recognition and analysis module of the document intelligent analysis device of the knowledge base management system performs image analysis on the docx document format, the storage path of the image is also embedded into the text content in a custom tag format to achieve the combination of image and text.

[0022] Furthermore, the recognition and analysis module of the document intelligent analysis device of the knowledge base management system recognizes and processes character codes line by line for the txt document format, and extracts various languages ​​and symbols.

[0023] The benefits of the present invention are:

[0024] Compared with the existing analysis, the method of the present invention can not only parse the contents of various document formats, but also can more accurately and comprehensively feedback the text, images, tables and other contents of the document. In particular, the visualization processing of pictures lays a solid foundation for the graphic retrieval and visualization display of the knowledge base, ensures the richness and accuracy of the retrieval results, increases the readability and intuitive simplicity of knowledge, and has good promotion and use value. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a schematic flow chart of the method of the present invention. DETAILED DESCRIPTION

[0026] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0027] Example 1

[0028] The present invention provides a document intelligent parsing method for a knowledge base management system, comprising:

[0029] Step 1: Preliminarily identify the document format according to the document extension, and perform corresponding analysis according to the type of document format: perform text analysis on pdf, docx, doc and txt document formats respectively. For pdf document format, extract text and images in pdf document format; for docx document format, parse according to the natural order of the document according to the compressed package of docx document format, and extract text and pictures; for doc document format, identify and extract all text information in doc document format; for txt document format, identify and process character encoding, and extract various languages ​​and symbols.

[0030] Step 1 may further include:

[0031] For the pdf document format, each page in the pdf document format is converted into a tree structure. Each node in the page represents an element, including text, images and tables. The content of the tree structure is traversed to extract the elements. When an image element is encountered, the original image is named with a unique identifier uuid and uploaded to the MinIO object storage service. At the same time, the corresponding storage address of the image is generated, and then the storage path of the image is embedded in the sequential position of the original image in the text content of the tree structure according to the custom tag format to achieve the combination of image and text.

[0032] For docx document format, when parsing images, the original image can also be stored in the MinIO object storage service with a unique tag as the name, and the corresponding storage address of the image is generated at the same time. It is then embedded in the text content in the order position of the original image in a custom tag format, thereby achieving an organic combination of text and images.

[0033] For the txt document format, in order to optimize performance and reduce memory usage, character encoding is identified and processed line by line, and various languages ​​and symbols are extracted. Even large text files can be easily processed while maintaining the fluency and efficiency of data processing. In this way, the integrity and accuracy of the text content can be ensured, while the ability to process large amounts of data can be improved.

[0034] Step 2: Use the big model to process the parsed text: segment the text for more detailed analysis and understanding, perform vectorization, convert the text into a numerical form that the big model can understand, and store the converted text in the knowledge base for information retrieval, data analysis, and knowledge mining.

[0035] Step 3: When the knowledge base retrieves relevant content, the text can be visualized, which significantly improves the readability and comprehension of knowledge. This combination of text and graphics not only enhances the intuitiveness of information, but also makes complex concepts easier to grasp, providing users with a richer and more intuitive knowledge experience.

[0036] Example 2

[0037] The present invention also provides a document intelligent parsing device for a knowledge base management system, comprising a recognition and parsing module, a large model processing module and a display module.

[0038] The recognition and parsing module preliminarily identifies the document format according to the document extension, and performs corresponding parsing according to the type of document format: it performs text parsing on pdf, docx, doc and txt document formats respectively. For pdf document format, it extracts text and images from pdf document format; for docx document format, it parses the compressed package of docx document format according to the natural order of the document to extract text and images; for doc document format, it recognizes and extracts all text information in doc document format; for txt document format, it recognizes and processes character encoding and extracts various languages ​​and symbols;

[0039] The big model processing module uses the big model to process the parsed text: segment the text and perform vector processing, convert the text into a numerical form that the big model can understand, and store the converted text in the knowledge base for information retrieval, data analysis, and knowledge mining.

[0040] The display output module also outputs the text visually.

[0041] As the information interaction and execution process between the modules in the above-mentioned device are based on the same concept as the embodiment of the method of the present invention, the specific contents can be found in the description of the embodiment of the method of the present invention and will not be repeated here.

[0042] Similarly, compared with existing analysis, the device of the present invention can not only parse the contents of various document formats, but also can more accurately and comprehensively feedback the text, images, tables and other contents of the document. In particular, the visualization processing of pictures lays a solid foundation for graphic and text retrieval and visualization of the knowledge base, ensures the richness and accuracy of the retrieval results, increases the readability and intuitive simplicity of knowledge, and has good promotion and use value.

[0043] It should be noted that not all steps and modules in the above-mentioned processes and device structures are necessary, and some steps or modules can be ignored according to actual needs. The execution order of each step is not fixed and can be adjusted as needed. The system structure described in the above-mentioned embodiments can be a physical structure or a logical structure, that is, some modules may be implemented by the same physical entity, or some modules may be implemented by multiple physical entities, or some components in multiple independent devices may be implemented together.

[0044] The above-described embodiments are only preferred embodiments for fully illustrating the present invention, and the protection scope of the present invention is not limited thereto. Equivalent substitutions or changes made by those skilled in the art based on the present invention are within the protection scope of the present invention. The protection scope of the present invention shall be subject to the claims.

Claims

1. A document intelligent parsing method for a knowledge base management system, characterized by include: Step 1: Preliminarily identify the document format according to the document extension, and perform corresponding analysis according to the type of document format: perform text analysis on pdf, docx, doc and txt document formats respectively. For pdf document format, extract text and images in pdf document format; for docx document format, parse according to the natural order of the document according to the compressed package of docx document format, extract text and images; for doc document format, identify and extract all text information in doc document format; for txt document format, identify and process character encoding, and extract various languages ​​and symbols; Step 2: Use the big model to process the parsed text: segment the text and perform vectorization, convert the text into a numerical form that the big model can understand, and store the converted text in the knowledge base for information retrieval, data analysis, and knowledge mining. Step 3: Visualize the text at the same time.

2. According to the document intelligent parsing method of the knowledge base management system of claim 1, it is characterized by In step 1, for the pdf document format, each page in the pdf document format is converted into a tree structure. Each node in the page represents an element, including text, images and tables. The content of the tree structure is traversed to extract the elements. When an image element is encountered, the original image is named with a unique identifier uuid and uploaded to the MinIO object storage service. The storage path of the image is then embedded in the text content of the tree structure in a custom tag format to achieve the combination of image and text.

3. The document intelligent parsing method of a knowledge base management system according to claim 1 is characterized by: In step 1, when parsing the image in the docx document format, the storage path of the image is also embedded into the text content in a custom tag format to achieve the combination of image and text.

4. The document intelligent parsing method of a knowledge base management system according to claim 1 is characterized by: In step 1, for the txt document format, character encoding is identified and processed line by line to extract various languages ​​and symbols.

5. A document intelligent parsing device for a knowledge base management system, characterized in that It includes recognition and analysis module, large model processing module and display output module. The recognition and parsing module preliminarily identifies the document format according to the document extension, and performs corresponding parsing according to the type of document format: it performs text parsing on pdf, docx, doc and txt document formats respectively. For pdf document format, it extracts text and images from pdf document format; for docx document format, it parses the compressed package of docx document format according to the natural order of the document to extract text and images; for doc document format, it recognizes and extracts all text information in doc document format; for txt document format, it recognizes and processes character encoding and extracts various languages ​​and symbols; The big model processing module uses the big model to process the parsed text: segment the text and perform vector processing, convert the text into a numerical form that the big model can understand, and store the converted text in the knowledge base for information retrieval, data analysis, and knowledge mining. The display output module also outputs the text visually.

6. The document intelligent parsing device for a knowledge base management system according to claim 5, characterized in that The recognition and parsing module targets the pdf document format and converts each page in the pdf document format into a tree structure. Each node in the page represents an element, including text, images and tables. The content of the tree structure is traversed to extract elements. When an image element is encountered, the original image is named with a unique identifier uuid and uploaded to the MinIO object storage service. The storage path of the image is then embedded in the text content of the tree structure in a custom tag format to achieve the combination of image and text.

7. The document intelligent parsing device for a knowledge base management system according to claim 5, characterized in that When the recognition and parsing module parses images in the docx document format, it also embeds the image storage path into the text content in a custom tag format to achieve image and text integration.

8. The document intelligent parsing device for a knowledge base management system according to claim 5, characterized in that The recognition and parsing module targets the txt document format, recognizes and processes character encodings line by line, and extracts various languages ​​and symbols.

Images