Retrieval-facilitating system file splitting method and system, storage medium and equipment

Abstract

The invention provides a system file splitting method and system beneficial to retrieval, a storage medium and equipment, and the method comprises the steps: importing a system file, and reading the content of the system file; analyzing the content of the system file, and formulating a self-adaptive splitting strategy according to an analysis result; according to the self-adaptive splitting strategy, splitting and structuring the content of the system file to generate structured data of the system file; and storing the structured data of the system file into a database, and retrieving and displaying the structured data through a user interface. By integrating a multi-format reading mode and an intelligent content identification technology, the automation degree and the processing speed of system file splitting are greatly improved, the labor cost is saved, and the working efficiency is improved. The advanced natural language processing and machine learning algorithm is adopted, the file structure and content are accurately recognized, multi-modal data analysis in the file is supported, splitting errors are reduced, and high quality and integrity of the split content are ensured.

Description

Technical Field

[0001] The present invention belongs to the technical field of file splitting, and in particular relates to a method, system, storage medium and device for splitting system files that are convenient for retrieval. Background Art

[0002] As an important carrier for regulating corporate behavior and guiding decision-making, institutional documents cover a wide range of dimensions, including organizational structure, human resources, financial management, and safe production. With the expansion of corporate scale and the increase in business complexity, the number of institutional documents has increased dramatically, and the formats are diverse, including but not limited to PDF, Word, Excel, HTML, and other formats. These documents not only contain structured clauses, but may also embed non-text elements such as charts and attachments, further increasing the complexity of information processing.

[0003] In the prior art, the RAG technology is used to reasonably split the file, and these split blocks are embedded into the vector / graph database respectively, so as to enable accurate recall later.

[0004] However, the existing technology still has the following problems: the styles of institutional documents vary and there is a lack of unified standards, which makes automated processing more difficult; the diversity of institutional document formats requires compatibility with multiple parsing technologies to ensure the accuracy and completeness of information extraction; the demand for personalized adaptation is high, and different companies or organizations have specific requirements for the splitting logic and keyword extraction rules of institutional documents. It is difficult for general solutions to meet all scenarios. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention provides a method for splitting system documents that is conducive to retrieval, the method comprising:

[0006] Import system files and read the contents of system files;

[0007] Analyze the content of institutional documents and develop adaptive splitting strategies based on the analysis results;

[0008] According to the adaptive splitting strategy, the content of the system document is split and structured to generate system document structured data;

[0009] The structured data of institutional documents are stored in the database and retrieved and displayed through the user interface.

[0010] Furthermore, the reading of the system file content includes: detecting the imported system file, and reading the system file content using a corresponding reading method.

[0011] Furthermore, the method of reading the content of the system file by using a corresponding reading method includes:

[0012] When the system file is in text-type PDF format, use PyPDF2 to read the system file content;

[0013] When the system document is in scanned PDF format, use EasyOCR to read the system document content;

[0014] When the system file is in word format, use the python-docx library to read the system file content;

[0015] When the system file is in txt file format, use the OPEN function to read the system file content;

[0016] When the system file is in HTML web page format, use BeautifulSoup to read data from the system file.

[0017] Furthermore, the in-depth analysis of the system document content includes: applying different strategies to analyze the system document content according to the splitting method selected by the user.

[0018] Furthermore, the splitting methods include: personalized splitting and automated splitting;

[0019] The application of different strategies to analyze the content of institutional documents includes:

[0020] When the user selects personalized splitting, the personalized splitting parameters are applied to analyze the system file;

[0021] When the user chooses automated splitting, NLP and machine learning models are applied to analyze the institutional documents.

[0022] Furthermore, the splitting and structuring of the system documents includes:

[0023] According to the developed adaptive splitting strategy, the content of the institutional documents is subdivided into small units;

[0024] Convert the subdivided small units into a structured data format to generate institutional document structured data.

[0025] Furthermore, the sub-units include chapters, sections, articles and clauses;

[0026] The structured data format includes the title, content and associated information of each small unit.

[0027] A system for splitting system documents that facilitates retrieval, the system comprising a system document reading module, an adaptive splitting strategy formulation module, a structured data generation module and a user interface configuration module;

[0028] The system file reading module is responsible for receiving and parsing the content of system files in various formats;

[0029] The adaptive splitting strategy formulation module is used to deeply analyze the content of the system document and formulate an adaptive splitting strategy based on the analysis results;

[0030] The structured data generation module is used to formulate an adaptive splitting strategy to split and structure the content of the system document to generate system document structured data;

[0031] The user interface configuration module is used to store the system document structured data and configure the user interface for retrieval and display.

[0032] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any of the method steps described above is implemented.

[0033] An electronic device, characterized in that it comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0034] Memory, used to store computer programs;

[0035] The processor is used to implement any of the above-mentioned method steps when executing the program stored in the memory.

[0036] Compared with the prior art, the present invention has the following advantages:

[0037] 1. The present invention proposes a system file splitting method that is conducive to retrieval. By integrating multi-format reading methods and intelligent content recognition technology, the automation level and processing speed of system file splitting are greatly improved, labor costs are saved, and work efficiency is improved.

[0038] 2. It uses advanced natural language processing and machine learning algorithms to accurately identify file structure and content, support multimodal data analysis in files, reduce splitting errors, and ensure the high quality and integrity of the split content.

[0039] 3. Provide user-configurable splitting strategies and logic, which can flexibly adjust the splitting rules according to the specific needs of the enterprise or organization, achieve personalized adaptation, and enhance practicality.

[0040] 4. The generated structured data facilitates the construction of an efficient retrieval system, supports fast and accurate information positioning, and improves user retrieval experience and knowledge management efficiency.

[0041] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 A schematic flow chart of a method for splitting institutional documents that is conducive to retrieval is shown.

[0044] Figure 2 A schematic diagram of a system module for splitting institutional documents that facilitates retrieval is shown. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] like Figure 1 As shown, a method for splitting system documents that is conducive to retrieval includes the following steps:

[0047] S1. Import the system file and parse it.

[0048] S1.1. Users upload internal system documents of enterprises or organizations through the user interface.

[0049] S1.2. Detect the uploaded system file type and parse and process the system file using the corresponding parsing method.

[0050] Optionally, for PDF files, if it is a text-type PDF, we use PyPDF2 to read the PDF file, extract the text and metadata, and for the pictures and tables, combine pdfminer.six and tableula-py to extract the content; if it is a scanned PDF, we use EasyOCR to read the PDF file;

[0051] For WORD files, use the python-docx library to read .docx files, extract elements such as paragraphs, tables, and pictures, and keep format information and metadata;

[0052] For ordinary text files (such as txt files, etc.), directly use Python's built-in open function to read;

[0053] For HTML web pages, use Beautiful Soup to extract data (including text, links, image sources, etc.) from the HTML files.

[0054] Optionally, for ordinary text, the original text can be directly extracted;

[0055] For image content, the original image is stored in the file system, and the original image in the file is replaced with a Markdown-formatted image URL so that the original file can be restored during subsequent retrieval; the system then uses EasyOCR to extract the image content as the description of the image URL;

[0056] For table contents, when extracting contents, the merged rows / columns will be repeatedly filled with the contents of the rows / columns before merging, ensuring that the final extracted contents conform to the Markdown format;

[0057] For other attachment contents (such as nested word files, etc.), first save the attachment as a whole to the file system, generate a unique ID with a system identifier and replace it with the original attachment content; and use EasyOCR, open source large models and python-docx parsing tools according to the attachment format to generate an attachment summary, and use the specified connector to associate it with the unique ID so that the original file can be found through the ID and the original content can be restored later.

[0058] S2. Deeply analyze the content of institutional documents and formulate adaptive splitting strategies.

[0059] S2.1. According to the splitting method selected by the user, different strategies are applied to analyze the content of the system document.

[0060] Optionally, when the user selects personalized splitting, the parameters of the personalized splitting are applied to analyze the policy document;

[0061] When users choose automated splitting, NLP and machine learning models are applied to analyze institutional documents.

[0062] S2.2. Based on the results of the institutional document analysis and in combination with the splitting method selected by the user, an adaptive splitting strategy is dynamically generated (for example, the splitting level is determined according to the title level and chapter details, or the splitting points are defined according to specific keywords).

[0063] S3. According to the adaptive splitting strategy formulated in step S2, the system document content is split and structured to generate system document structured data.

[0064] S3.1. According to the adaptive splitting strategy formulated in step S2, subdivide the content of the system document into logically coherent small units (such as chapters, sections, articles, clauses, etc.).

[0065] S3.2. Convert the split small units into a specified structured data format (Markdown).

[0066] Optionally, the structured data format includes metadata such as the title, content, and associated information of each small unit.

[0067] S4. Store the structured data of the system documents in the database, and retrieve and display them through the user interface.

[0068] S4.1. Store the structured data format obtained in step S3 into the database.

[0069] Optionally, the database is a vector or graph database.

[0070] S4.2. Display structured data of institutional documents through the user interface, provide a search box, and support on-demand viewing of detailed content.

[0071] like Figure 2 As shown, a system for splitting system of system documents that is conducive to retrieval includes: a system document reading module, an adaptive splitting strategy formulation module, a structured data generation module and a user interface configuration module.

[0072] The system file reading module is responsible for receiving and reading system files of various formats.

[0073] The adaptive splitting strategy formulation module applies natural language processing (NLP) technology and machine learning algorithms to conduct in-depth analysis of the content of institutional documents, identify different content structures such as chapters, clauses, titles, lists, tables, etc. compared to ordinary documents; and dynamically adjust the adaptive splitting strategy based on the identified content structure.

[0074] The structured data generation module converts the split file information into a standardized, structured data format, such as JSON, XML, Markdown, or plain text, and stores it in a specified database for subsequent indexing and retrieval.

[0075] The user interface configuration module provides an intuitive operation interface, allowing administrators or users to customize splitting parameters, set keyword filtering rules, adjust output formats, etc. according to their own needs, achieving high configurability and personalized services.

[0076] Based on the above disclosed content, the present invention also provides an electronic device accordingly. The electronic device of an embodiment of the present invention includes at least one electrically connected processor and at least one storage medium, the storage medium is electrically connected to the processor, wherein the storage medium stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the method as described above.

[0077] Based on the same inventive concept, the present invention also provides a storage medium, which stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method as described above.

[0078] The above description and the accompanying drawings fully illustrate the embodiments of the present invention so that those skilled in the art can practice them. Other embodiments may include structural and other changes. The embodiments represent only possible variations. Unless explicitly required, individual components and functions are optional, and the order of operations may vary. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. The embodiments of the present invention are not limited to the structures described above and shown in the accompanying drawings, and various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A method for splitting system documents for retrieval, characterized in that: The method comprises, Import system files and read the contents of system files; Analyze the content of institutional documents and develop adaptive splitting strategies based on the analysis results; According to the adaptive splitting strategy, the content of the system document is split and structured to generate system document structured data; The structured data of institutional documents are stored in the database and retrieved and displayed through the user interface.

2. The method for splitting system documents for retrieval-friendly purposes according to claim 1, characterized in that: The reading of the system file content includes: detecting the imported system file, and reading the system file content using a corresponding reading method.

3. The method for splitting system documents for retrieval-friendly purposes according to claim 2, characterized in that: The method of reading the system file content by using a corresponding reading method includes: When the system file is in text-type PDF format, use PyPDF2 to read the system file content; When the system document is in scanned PDF format, use EasyOCR to read the system document content; When the system file is in word format, use the python-docx library to read the system file content; When the system file is in txt file format, use the OPEN function to read the system file content; When the system file is in HTML web page format, use BeautifulSoup to read data from the system file.

4. The method for splitting system documents for retrieval-friendly purposes according to claim 1, characterized in that: The in-depth analysis of the system file content includes: applying different strategies to analyze the system file content according to the splitting method selected by the user.

5. The method for splitting system documents for retrieval-friendly purposes according to claim 4, characterized in that: The splitting methods include: personalized splitting and automated splitting; The application of different strategies to analyze the content of institutional documents includes: When the user selects personalized splitting, the personalized splitting parameters are applied to analyze the system file; When the user chooses automated splitting, NLP and machine learning models are applied to analyze the institutional documents.

6. The method for splitting system documents for retrieval-friendly purposes according to claim 1, characterized in that: The splitting and structuring of the system documents includes: According to the developed adaptive splitting strategy, the content of the institutional documents is subdivided into small units; Convert the subdivided small units into a structured data format to generate institutional document structured data.

7. The method for splitting system documents for retrieval-friendly purposes according to claim 6, characterized in that: The sub-units include chapters, sections, articles and clauses; The structured data format includes the title, content and associated information of each small unit.

8. A system for splitting institutional documents for easy retrieval, characterized in that: The system includes a system document reading module, an adaptive splitting strategy formulation module, a structured data generation module and a user interface configuration module; The system file reading module is responsible for receiving and parsing the content of system files in various formats; The adaptive splitting strategy formulation module is used to deeply analyze the content of the system document and formulate an adaptive splitting strategy based on the analysis results; The structured data generation module is used to formulate an adaptive splitting strategy to split and structure the content of the system document to generate system document structured data; The user interface configuration module is used to store the system document structured data and configure the user interface for retrieval and display.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 7 are implemented.

10. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 7 when executing a program stored in a memory.

Images