System document splitting method, system, storage medium and device facilitating retrieval

Abstract

The application provides a system file splitting method and system, a storage medium and equipment for facilitating retrieval, the method comprising: importing a system file and reading the system file content; analyzing the system file content and formulating an adaptive splitting strategy according to the analysis result; splitting and structuring the system file content according to the adaptive splitting strategy to generate system file structured data; storing the system file structured data into a database and retrieving and displaying through a user interface. The application greatly improves the automation degree and processing speed of system file splitting by integrating multi-format reading mode and intelligent content recognition technology, saves labor cost and improves work efficiency. Advanced natural language processing and machine learning algorithms are adopted to accurately identify file structure and content, support multi-modal data analysis in files, reduce splitting errors and ensure high quality and integrity of the split content.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of file splitting, and particularly relates to a system file splitting method and system beneficial to retrieval, a storage medium and equipment. BACKGROUND

[0002] As an important carrier for regulating enterprise behavior and guiding decision-making, system files cover multiple dimensions such as organizational structure, human resources, financial management, and safety production. With the expansion of enterprise scale and the increase of business complexity, the number of system files has increased dramatically, and the forms are various, including but not limited to PDF, Word, Excel, HTML, and other formats. These files not only contain structured clause content, but also may embed charts, attachments, and other non-text elements, further increasing the complexity of information processing.

[0003] In the prior art, the files are reasonably split by RAG technology, and the split blocks are embedded into a vector / graph database, so that accurate recall can be achieved subsequently.

[0004] However, in the prior art, there are still the following problems: the styles of system files are different, and there is a lack of unified standards, making it difficult to automate processing; the formats of system files are diverse, and multiple parsing technologies need to be compatible to ensure the accuracy and integrity of information extraction; personalized adaptation requirements are high, and different enterprises or organizations have specific requirements for system file splitting logic, keyword extraction rules, etc., and general solutions cannot meet all scenarios. SUMMARY

[0005] The present application provides a system file splitting method beneficial to retrieval, which comprises,

[0006] Importing a system file and reading the content of the system file;

[0007] Analyzing the content of the system file and formulating an adaptive splitting strategy according to the analysis result;

[0008] Splitting and structuring the content of the system file according to the adaptive splitting strategy to generate system file structured data;

[0009] Storing the system file structured data into a database and retrieving and displaying it through a user interface.

[0010] Further, the reading of the content of the system file comprises detecting the imported system file and reading the content of the system file by using a corresponding reading method.

[0011] Further, the reading of the content of the system file by using a corresponding reading method comprises:

[0012] When the system document is in the text type PDF format, use PyPDF2 to read the system document content;

[0013] When the system document is in the scanned PDF format, use EasyOCR to read the system document content;

[0014] When the system document is in the word format, use the python-docx library to read the system document content;

[0015] When the system document is in the txt file format, use the OPEN function to read the system document content;

[0016] When the system document is in the html web page format, use BeautifulSoup to read data from the system document.

[0017] Further, the deep analysis of the system document content includes: according to the user's choice of splitting method, applying different strategies to analyze the system document content.

[0018] Further, the splitting method includes: personalized splitting and automated splitting;

[0019] The application of different strategies to analyze the system document content includes:

[0020] When the user chooses personalized splitting, apply personalized splitting parameters to analyze the system document;

[0021] When the user chooses automated splitting, apply NLP and machine learning models to analyze the system document.

[0022] Further, the splitting and structuring of the system document includes:

[0023] According to the adaptive splitting strategy, the system document content is subdivided into small units;

[0024] Convert the subdivided small units into structured data format to generate structured data of the system document.

[0025] Further, the small unit includes chapter, section, article and clause;

[0026] The structured data format includes the title, content and related information of each small unit.

[0027] A system for splitting and retrieving system documents, the system includes a system document reading module, an adaptive splitting strategy development module, a structured data generation module and a user interface configuration module;

[0028] The system document reading module is responsible for receiving and parsing the content of system documents in various formats;

[0029] The adaptive splitting strategy module is used for deep analysis of the content of the system document, and formulating an adaptive splitting strategy according to the analysis result;

[0030] The structured data generation module is used for splitting and structuring the content of the system document according to the formulated adaptive splitting strategy, and generating structured data of the system document;

[0031] The user interface configuration module is used for storing the structured data of the system document, and configuring the user interface for retrieval and display.

[0032] A computer readable storage medium, the computer readable storage medium stores a computer program, the computer program is executed by a processor to realize the method steps of any of the above.

[0033] An electronic device, characterized in that, comprising processor, communication interface, memory and communication bus, wherein, processor, communication interface and memory complete mutual communication through communication bus;

[0034] Memory, for storing computer programs;

[0035] Processor, for executing the program stored on the memory, realizing the method steps of any of the above.

[0036] Compared with the prior art, the present application has the following advantages:

[0037] 1. The present application proposes a system document splitting method for retrieval, which greatly improves the automation degree and processing speed of system document splitting by integrating multi-format reading mode and intelligent content recognition technology, saves labor cost and improves work efficiency.

[0038] 2. Advanced natural language processing and machine learning algorithms are used to accurately identify file structure and content, support multi-modal data analysis in files, reduce splitting errors, and ensure high quality and integrity of the split content.

[0039] 3. User-configurable splitting strategies and logic are provided, which can flexibly adjust the splitting rules according to the specific needs of enterprises or organizations, realize personalized adaptation, and enhance the practicality.

[0040] 4. The generated structured data facilitates the construction of an efficient retrieval system, supports fast and accurate information positioning, and improves user retrieval experience and knowledge management efficiency.

[0041] Other features and advantages of the present application will be set forth in the description that follows, and in part will be apparent from the description, or can be learned by practice of the application. The purposes and other advantages of the present application will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0043] Figure 1 A system file splitting method for facilitating retrieval is shown.

[0044] Figure 2 A system file splitting system module diagram for facilitating retrieval is shown. DETAILED DESCRIPTION

[0045] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0046] As shown in Figure 1 A system file splitting method for facilitating retrieval, the steps of which include:

[0047] S1, import system files and perform parsing processing.

[0048] S1.1, the user uploads the system files in the enterprise or organization through the user interface.

[0049] S1.2, detect the type of uploaded system files, and use the corresponding parsing method to parse and process the system files.

[0050] Optionally, for PDF files, if it is a text type PDF, we use PyPDF2 to read the PDF file, extract the text and metadata, and for the pictures and tables therein, we realize content extraction in combination with pdfminer.six and tabula-py; if it is a scanned type PDF, we use EasyOCR to read the PDF file;

[0051] For WORD files, use the python-docx library to read .docx files, extract elements such as paragraphs, tables, and pictures, and preserve formatting information and metadata;

[0052] For ordinary text files (such as txt files, etc.), directly use Python's built-in open function to read;

[0053] For HTML web pages, use Beautiful Soup to extract data (including text, links, image sources, etc.) from HTML files.

[0054] Alternatively, for ordinary text, the original text can be directly extracted;

[0055] For image content, the original image is stored in the file system and replaced with a Markdown-formatted image URL so that the original file can be restored during subsequent retrieval. The system then uses EasyOCR to extract the image content as a description of the image URL.

[0056] For table content, when extracting content, the merged rows / columns will be repeatedly filled with the content of the rows / columns before the merging, ensuring that the final extracted content conforms to the Markdown format;

[0057] For other attachment content (such as nested word files), first save the attachment as a whole to the file system, generate a unique ID with a system identifier and replace it with the original attachment content; and use EasyOCR, open source large models and python-docx parsing tools according to the attachment format to generate an attachment summary, and use the specified connector to associate it with the unique ID so that the original file can be found through the ID and the original content can be restored later.

[0058] S2. Deeply analyze the content of institutional documents and formulate adaptive splitting strategies.

[0059] S2.1. Apply different strategies to analyze the content of the system document based on the splitting method selected by the user.

[0060] Optionally, when the user selects personalized splitting, the parameters of the personalized splitting are applied to analyze the policy document;

[0061] When users choose automated splitting, NLP and machine learning models are applied to analyze institutional documents.

[0062] S2.2. Based on the results of the institutional document analysis and the splitting method selected by the user, an adaptive splitting strategy is dynamically generated (for example, the splitting level is determined according to the title level and chapter details, or the splitting points are defined according to specific keywords).

[0063] S3.1, according to the adaptive splitting strategy formulated in step S2, the content of the system document is subdivided into logically coherent small units (such as chapters, sections, articles, clauses, etc.).

[0064] S3.1, according to the adaptive splitting strategy formulated in step S2, the content of the system document is subdivided into logically coherent small units (such as chapters, sections, articles, clauses, etc.).

[0065] S3.2, the small units after splitting are converted into the formulated structured data format (Markdown).

[0066] Optionally, the structured data format contains the title, content, and associated information of each small unit, etc.

[0067] S4, the structured data of the system document is stored in the database, and is retrieved and displayed through the user interface.

[0068] S4.1, the structured data format obtained in step S3 is stored in the database.

[0069] Optionally, the database is a vector or graph database.

[0070] S4.2, the structured data of the system document is displayed through the user interface, a search box is provided, and detailed content can be viewed on demand.

[0071] As shown in Figure 2 A system for splitting a system document for retrieval includes a system document reading module, an adaptive splitting strategy formulation module, a structured data generation module, and a user interface configuration module.

[0072] The system document reading module is responsible for receiving and reading system documents of various formats.

[0073] The adaptive splitting strategy formulation module applies natural language processing (NLP) techniques and machine learning algorithms to deeply analyze the content of the system document, identify different content structures such as chapters, clauses, titles, lists, tables, etc. that are unique in ordinary files, and dynamically adjust the adaptive splitting strategy based on the identified content structures.

[0074] The structured data generation module converts the split file information into a standardized, structured data format such as JSON, XML, Markdown, or plain text, and stores it in a designated database for subsequent indexing and retrieval.

[0075] The user interface configuration module provides an intuitive operation interface, allowing administrators or users to customize splitting parameters, set keyword filtering rules, adjust output formats, etc. according to their own needs, achieving high configurability and personalized services.

[0076] Based on the above disclosure, the present application also provides an electronic device accordingly. The electronic device of the embodiments of the present application comprises at least one processor and at least one storage medium electrically connected with the processor, wherein the storage medium stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method as described above.

[0077] Based on the same inventive concept, the present application also provides a storage medium storing instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method as described above.

[0078] The above description and drawings sufficiently illustrate embodiments of the present application to enable one skilled in the art to practice them. Other embodiments can include structural and other changes. Embodiments are merely representative of possible variations. Individual components and functions are optional unless explicitly required, and the order of operations can be varied. Portions and features of some embodiments can be included in, or substituted for, those of other embodiments. Embodiments of the present application are not limited to the structures described above and shown in the drawings, and can be varied in various ways. The scope of the present application is limited only by the claims appended hereto.

Claims

1. A method for splitting system documents that is convenient for retrieval, characterized in that: The method comprises, Import the system file and read the content of the system file; Analyze the content of institutional documents and develop adaptive splitting strategies based on the analysis results; According to the adaptive splitting strategy, the content of the system document is split and structured to generate system document structured data; Store the structured data of the system documents in the database and retrieve and display them through the user interface; The reading of the system file content includes: detecting the imported system file and reading the system file content using a corresponding reading method; The method of reading the content of the system file using the corresponding reading method includes: When the system file is in text-type PDF format, use PyPDF2 to read the system file content; When the policy document is in scanned PDF format, use EasyOCR to read the content of the policy document; When the system file is in Word format, use the python-docx library to read the system file content; When the system file is in txt format, use the OPEN function to read the system file content; When the system file is in HTML web page format, use BeautifulSoup to read data from the system file; The analyzing the content of the system file includes: applying different strategies to analyze the content of the system file according to the splitting method selected by the user; The splitting methods include: personalized splitting and automated splitting; The application of different strategies to analyze the content of institutional documents includes: When the user selects personalized splitting, the personalized splitting parameters are applied to analyze the system documents; When users choose automated splitting, NLP and machine learning models are applied to analyze the institutional documents.

2. The method for splitting system documents for facilitating retrieval according to claim 1, characterized in that: The splitting and structuring of institutional documents includes: According to the developed adaptive splitting strategy, the content of the system document is subdivided into small units; Convert the subdivided small units into a structured data format to generate institutional document structured data.

3. The method for splitting system documents for facilitating retrieval according to claim 2, characterized in that: The said small units include chapters, sections, articles and clauses; The structured data format includes the title, content and associated information of each small unit.

4. A system for splitting system documents that is convenient for retrieval, characterized in that: Using the method of claim 1, the system includes a policy document reading module, an adaptive splitting strategy formulation module, a structured data generation module, and a user interface configuration module; The system file reading module is responsible for receiving and parsing the content of system files in various formats; The adaptive splitting strategy formulation module is used to deeply analyze the content of the system document and formulate an adaptive splitting strategy based on the analysis results; The structured data generation module is used to formulate an adaptive splitting strategy to split and structure the content of the system document and generate system document structured data; The user interface configuration module is used to store the structured data of the system documents and configure the user interface for retrieval and display.

5. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 3 are implemented.

6. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory for storing computer programs; A processor, configured to implement the method steps described in any one of claims 1 to 3 when executing a program stored in a memory.

Images