A document decoupling and synthesis system based on paragraph small file storage

Abstract

The application discloses a kind of based on paragraph small file storage's document decoupling and synthesis system, comprising: data document analysis and storage module, for complete document storage and to document are parsed;Relevance document acquisition module, according to keyword acquisition relevance document;Paragraph small file storage module, to the file after splitting transmission verification, storage and management;Document batch disassembly and merging module, based on paragraph list tree structure title realizes file batch disassembly and merging;Merging resource pool module, select paragraph and add to resource pool;Resource pool content synthesis and download module, for resource pool content synthesis and download and new file stream generation.The application is through paragraph small file storage, relevance document query based on keyword, combine big data search engine and automatically filter out the relevance paragraph of meeting search content, based on the content of selected paragraph realizes document content disassembly, merging and storage, for user verification screening and corresponding processing.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of office automation, and particularly relates to a document decoupling and synthesis system based on paragraph small file storage. BACKGROUND

[0002] With the rapid development of computer technology, the popularity of Word documents greatly improves office efficiency, and some disadvantages of paper operation are improved with the application of electronic documents, so that office efficiency is greatly improved. In daily work, such as participating in a bidding meeting, a large number of electronic documents are generated. At present, the documents are mainly arranged by manual operation, but manual operation is very troublesome, time-consuming and laborious, and errors are prone to occur during arrangement, so that work efficiency cannot be improved.

[0003] In the daily file arrangement of the bidding book, a large number of documents and the correlation between the document contents are not directly related, which causes great inconvenience for users to find valuable documents, and each user has to download a large number of documents, browse them one by one, and delete the repeated and similar documents.

[0004] Therefore, a system for decoupling and merging new files based on document content is urgently needed. SUMMARY

[0005] The technical problem to be solved by the application is to provide a document decoupling and synthesis system based on paragraph small file storage, which solves the above technical problems of the prior art. The system can automatically filter out relevant paragraphs that meet the search content from a large number of documents by using paragraph small file storage directory tree management and file keyword-based correlation document query, and based on the content of the selected paragraphs, the system can specify the selected paragraphs as the title of the seventh level and synthesize them in the document. The system can also realize the disassembly, merging and storage of document pictures, videos and text content, so that the user can verify and filter the relevant paragraphs and perform corresponding processing.

[0006] To achieve the above technical purpose, the technical solution adopted by the application is as follows:

[0007] A document decoupling and synthesis system based on paragraph small file storage, comprising:

[0008] A data document analysis and storage module is used for storing complete documents and analyzing the documents, and based on Hash calculation, MD5 encoding, paragraph encoding and paragraph content are obtained.

[0009] A correlation document acquisition module is used for acquiring correlation documents according to the input key words of a user.

[0010] A paragraph small file storage module is used for transmitting, verifying, storing and managing the split files.

[0011] The document batch disassembly and merging module is used for implementing batch disassembly and merging of Word files based on a paragraph list tree structure title.

[0012] The merging resource pool module is used for adding paragraphs to a resource pool based on a paragraph tree structure list selection.

[0013] The resource pool content synthesis and download module is used for synthesizing and downloading resource pool content and generating a new Word file stream.

[0014] To optimize the above technical solutions, the following specific measures are taken:

[0015] In the above data document analysis and storage module, a MongoDB database is built in the application server to realize complete document storage; ElasticSearch is used as a search engine, a program is written based on Java, multi-threading is started to upload Word files to the MongoDB database, Hash calculation is performed on the documents, the Word files are parsed to form MD5 codes of file unique identification, and the paragraph decomposition is performed on the file content; the MD5 code of the file upload unique identification and the paragraph code are used as keys, and the paragraph content is used as a value to be stored in the ElasticSearch service for easy searching.

[0016] The above correlation document acquisition module assigns corresponding operation document permissions to different users and submits resource documents to be synthesized, and specifically:

[0017] (1) The document permission system is intervened to assign the operations that the user has on the searched documents, and the read-only and editable permissions are assigned to the user to download and synthesize the file;

[0018] (2) The interface is used to acquire the keyword input by the user, the search matching file name and file content are used in the ElasticSearch service, the matched file name MD5 code set is obtained, and the paragraph information list of the file is returned for easy user searching.

[0019] The above paragraph small file storage module is based on Netty to realize the HTTP protocol, performs file download through the HTTP protocol, adopts data packeting + md5 file verification, and stores the data format of the split file after the protobuf serialization.

[0020] The above paragraph small file storage module is provided with a file deletion + trash function unit, realizes the moving of the deleted file to the trash, and recovers the file from the trash.

[0021] The document batch disassembling and merging module installs the MongoDB document storage service unit; when decoupling the screening file, first, it is judged whether the decoupled data exist in the paragraph small file storage module, if the decoupled data exist, the paragraph list tree structure title of the file decoupling is obtained through the file MD5 coding and the MongoDB service, and then returned to the user end for selection; if the decoupled data do not exist, the decoupling operation is performed.

[0022] The decoupling operation process includes:

[0023] The Word file is obtained, all title lists of the document are obtained, and the paragraph content is obtained by circulating and traversing each level of title;

[0024] The paragraph content is converted into a file stream form and stored in the paragraph small file storage module, and the corresponding parameters of each paragraph are set, and finally a document content resource pool is constructed in a tree structure storage mode;

[0025] The corresponding parameters are added and the tree structure is returned to the front end page for the user to select the paragraph.

[0026] In the merging resource pool module, the MD5 code and the paragraph of the source file are stored in the resource pool based on the paragraph tree structure list selection.

[0027] The resource pool content synthesis and download module searches the resource pool file paragraph based on the MD5 code of the source file, obtains the corresponding paragraph content in the original file Word, then combines the file stream to generate a new Word file stream, and then returns the user end for downloading.

[0028] The present application has the following beneficial effects:

[0029] 1. The user is given the freedom of title extraction, and can freely combine according to the selected keyword content to form a multi-dimensional, multi-angle and multi-level directory document.

[0030] 2. The document processing capacity is large, and the advantage is obvious in large batch processing: under normal memory conditions, it can basically achieve unlimited processing, only the time of searching keywords is consumed, and the processing efficiency is greatly improved; under the same conditions, manual acquisition of document content for processing requires a long time.

[0031] 3. The document processing is infinitely close to zero error. According to the existing test results, no error phenomenon occurs, and the error can be infinitely close to zero when the test sample increases. Under the same conditions, manual processing is affected by many factors such as individual time, energy and operation steps, and multiple errors are likely to occur.

[0032] 4、Can realize that user obtains relevant PPT document through keyword. In document, obtains corresponding keyword, then according to different screening standard, grabs corresponding required title text, forms corresponding word file, satisfies the demand of user required reference multi-word generation new document;Supports in personal resource pool word file drag order adjustment, then synthesizes new document.

[0033] 5、Can batch process multi-word file, simultaneously carries out decoupling, and in resource pool supports preview, reduces the content operation cost of user between multiple documents, improves work efficiency.

[0034] 6、Based on search engine, the document retrieval of search engine mainly uses index service to provide content review to user, realizes the improvement of user document acquisition efficiency;The document uploaded by user is split into a paragraph of MD5 encrypted paragraph, then is saved to a common resource pool, so that for user content retrieval, can quickly find the document wanted by oneself through service terminal, and can match through the MD5 value of search relevance content;Can define whether the paragraph has relevance through the set threshold, so as to retrieve the relevance content input by user. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 The document decoupling and synthesis system based on paragraph small file storage of the application is designed. DETAILED DESCRIPTION

[0036] The embodiments of the application are further described in detail below in combination with the drawings.

[0037] As shown in Figure 1 The purpose of the embodiments of the application is to provide a document decoupling and synthesis system based on paragraph small file storage, the basic idea is to input screening condition, select relevant document, judge the content specification of document, disassemble document paragraph, and carry out synthesis processing. The basic implementation scheme includes the following modules:

[0038] 1. Data document analysis and storage module, is used for complete document storage, and carries out analysis to document, obtains MD5 code, paragraph code and paragraph content based on Hash calculation;

[0039] (1) Build MongoDB database in application server as complete document storage service;

[0040] Use ElasticSearch as search engine

[0041] (2) Based on Java programming, starting multi-threaded Word file upload to MongoDB database, Hash calculation on the document, parsing Word file to form MD5 encoding of file unique identification, while paragraph decomposition on file content, file upload unique identification MD5 encoding and paragraph encoding as key, paragraph content as value stored in ElasticSearch service for easy search;

[0042] 2. Correlation document acquisition module, for acquiring correlation document according to user input keyword;

[0043] By assigning different users corresponding operation document permissions, and submitting the resource document to be synthesized:

[0044] (1) Intervene in the document permission system, allocate user's operation on the searched document, and allocate read-only and editable permissions for user download and synthesis of the file;

[0045] (2) Get the user input keyword through the interface, use search matching file name and file content in ElasticSearch service, get the matched file name MD5 code set, and return the file paragraph information list for easy user search;

[0046] 3. Paragraph small file storage module, for transmission verification, storage and management of the split file;

[0047] (1) Based on Netty, HTTP protocol is realized, and file can be downloaded through HTTP protocol

[0048] (2) File upload and download core function transformation, data packeting + md5 file verification is adopted to ensure file transmission accuracy

[0049] (3) The data format of the split file is stored after protobuf serialization, which can improve the serialization speed and optimize the occupied disk space size

[0050] (4) File deletion + trash can function. The file is moved to the trash can after deletion, and the file can be restored from the trash can

[0051] 4. Document batch splitting and merging module, for implementing Word file batch splitting and merging based on paragraph list tree structure title;

[0052] (1) Install MongoDB document storage service.

[0053] (2) User decouples the click on the screening file, obtains whether there is decoupled data in the small file system, if there is, finds the paragraph list tree structure title of the file decoupling through the file MD5 code to obtain the MongoDB service and returns the user terminal selection.

[0054] (3) If there is no decoupled data, decoupling operation is performed. The Word file is obtained, all title lists of the document are obtained, and the paragraph content is obtained by circulating through each level of title.

[0055] (4) The paragraph content is converted into a file stream form and stored in the self-developed small file system, and the corresponding parameters (source file MD5, upper paragraph ID, own paragraph ID, paragraph name, and level) are set. Finally, a document content resource pool is constructed in a tree structure storage mode.

[0056] (5) The corresponding parameters (source file MD5, paragraph name, child paragraph, and level) are added, and the tree structure is returned to the front-end page for user selection of paragraphs.

[0057] 5. The resource pool module is used for selecting paragraphs to add to the resource pool.

[0058] (1) The user obtains the paragraph tree structure list, checks the corresponding paragraph to add to the resource pool.

[0059] (2) The MD5 code of the source file and the paragraph are obtained and stored in the resource pool table.

[0060] 6. The resource pool content synthesis and download module is used for resource pool content synthesis and download and new Word file stream generation.

[0061] (1) The user can adjust the order of the content of the resource pool and click to synthesize and download.

[0062] (2) The source file MD5 code in the list and the resource pool file paragraph are obtained to find the corresponding paragraph content in the original file Word, then the file stream is combined to generate a new Word file stream, and then the user terminal is returned for download.

[0063] 7. The detailed process of document disassembly and synthesis is as follows:

[0064] (1) The user selects the relevant documents according to the keywords, selects the document for analysis, and can select one or more documents for simultaneous analysis. The user first performs Hash processing on the selected document, such as using the MD5 algorithm. If the calculated Hash value exists in MongoDB, it will not be uploaded again.

[0065] (2) through the API interface to construct a document object model, then read the file input stream, use the document object to extract all paragraph list tags as list items, form a collection of paragraph nodes, loop through each paragraph, through the custom set reading title level parameter, judge the current paragraph style is one of the built-in title styles of Word document, and get the level where the paragraph is located, if less than the custom level parameter, recursively query the child paragraph, and so on, find the last level paragraph.

[0066] After obtaining the paragraph title data of each level, encapsulate a paragraph object, including paragraph ID, document ID, paragraph title name, last paragraph title name, and subscript index of the level (used for paragraph ordering of the synthesized file). Combine the document ID and paragraph ID to form the key value as the paragraph content.

[0067] Read the document input stream, take the paragraph title as the input parameter, through the API, you can get all the content nodes DOM or XmlDocument, traverse all nodes to extract the content of the paragraph, first verify the node validity, that is, whether it exists in the document, if it exists, perform the operation of saving to ElasticSearch and small file system: use the key formed by combining the document ID and paragraph ID as the primary key, and store the content as value in ElasticSearch, construct the output stream and save it as a small file.

[0068] (3) split the keywords input by the user, match the values stored in ES, get the matched object primary key, regular primary key get the document ID, read the paragraph to the page for tree structure display

[0069] (4) the user checks the title, clicks to add to the resource pool, and can select other documents to continue adding paragraphs. Enter the resource pool, the user can sort all paragraphs, click to synthesize the file, which will form the key of the document ID and paragraph ID of all list items as parameters, and go to the small file system to match and find, find all synthesized resources input stream, write to a new file and return the output stream to the browser for download. At this point, the file synthesis process is completed.

[0070] The above is only the preferred embodiment of the present application, the protection scope of the present application is not limited to the above-mentioned embodiments, all technical solutions under the idea of the present application belong to the protection scope of the present application. It should be pointed out that for ordinary technical personnel in this technical field, some improvements and decorations without departing from the principle of the present application should be regarded as the protection scope of the present application.

Claims

1. A document decoupling and synthesis system based on paragraph-based small file storage, characterized in that, include: The data document parsing and storage module is used for storing complete documents and parsing multiple documents, obtaining MD5 encoding, paragraph encoding, and paragraph content based on hash calculation; The relevant document retrieval module is used to segment the user-input keywords, match the values ​​stored in ElasticSearch, obtain the primary key of the matched object, obtain the document ID from the regular expression primary key, and read the paragraphs to the page for tree structure display in order to obtain relevant documents. The segmented small file storage module is used for the transmission verification, storage, and management of the split files; The document batch splitting and merging module is used to batch split and merge Word files based on the paragraph list tree structure headings. The Merge Resource Pool module is used to select titles based on a paragraph tree structure list and add them to the resource pool. The resource pool content synthesis and download module is used to sort all paragraphs in the resource pool content. When a synthesis file is clicked, the document ID and paragraph ID of all list items are used as a key as parameters to match and search in the paragraph small file storage module. After the search is completed, an input stream of all synthesis resources is constructed, which is then written into a new file and returned to the output stream for download in the browser, as well as generating a new Word file stream.

2. The document decoupling and synthesis system based on paragraph-based small file storage according to claim 1, characterized in that, In the data document parsing and storage module, a MongoDB database is built within the application server to achieve complete document storage. ElasticSearch is used as the search engine. A Java program is written to start multiple threads to upload Word files to the MongoDB database. The document is hashed, and the Word file is parsed to form a unique MD5 encoding of the file. At the same time, the file content is decomposed into paragraphs. The unique MD5 encoding of the uploaded file and the paragraph encoding are used as keys, and the paragraph content is used as values ​​to be stored in the ElasticSearch service for easy retrieval.

3. The document decoupling and synthesis system based on paragraph-based small file storage according to claim 1, characterized in that, The relevant document acquisition module assigns corresponding permissions to different users for operating documents and submits the resource documents to be synthesized. Specifically: (1) By intervening in the document permission system, assign users the operations they have on the searched documents, and assign read-only and editable permissions to the files downloaded and synthesized by users; (2) Obtain the keywords entered by the user through the interface, use the search function in the ElasticSearch service to match the file name and file content, obtain the MD5 encoding set of the matched file name, and then iterate through the returned list of paragraph information of the file to facilitate the user's search.

4. The document decoupling and synthesis system based on paragraph small file storage according to claim 1, characterized in that, The segmented small file storage module implements the HTTP protocol based on Netty and downloads files via the HTTP protocol; it adopts data packetization + MD5 file verification; and it transforms the storage format of the split file data into protobuf serialization before it is written to disk.

5. A document decoupling and synthesis system based on paragraph-based small file storage according to claim 4, characterized in that, The paragraph small file storage module is equipped with a file deletion + trash can function unit, which realizes that after a file is deleted, it is moved to the trash can, and the file is recovered from the trash can.

6. A document decoupling and synthesis system based on paragraph-based small file storage according to claim 1, characterized in that, The document batch splitting and merging module installs the MongoDB document storage service unit. When decoupling filtered files, it first determines whether decoupling data exists in the paragraph small file storage module. If decoupling data exists, it retrieves the paragraph list tree structure title of the decoupling file from the MongoDB service through the file MD5 encoding and returns it to the user for selection. If decoupling data does not exist, the decoupling operation is performed.

7. A document decoupling and synthesis system based on paragraph-based small file storage according to claim 6, characterized in that, The decoupling process includes: Get the Word file, get the list of all headings in the document, and loop through the headings at each level to get the paragraph content; The paragraph content is converted into a file stream and stored in the paragraph small file storage module. The parameters corresponding to each paragraph are set, and finally a document content resource pool is built in a tree structure storage method. Add the corresponding parameters and return the tree structure to the front-end page for the user to select paragraphs.

8. A document decoupling and synthesis system based on paragraph-based small file storage according to claim 1, characterized in that, In the merged resource pool module, the MD5 encoding and paragraphs of the source file are selected based on the paragraph tree structure list and stored in the resource pool.

9. A document decoupling and synthesis system based on paragraph-based small file storage according to claim 1, characterized in that, The resource pool content synthesis and download module searches for paragraphs in the existing resource pool file based on the MD5 encoding of the source file, obtains the corresponding paragraph content in the original Word file, then combines the file streams to generate a new Word file stream, and then returns it to the user terminal for download.

Images