Intelligent document management and retrieval method and system

Abstract

Disclosed is an intelligent document management and retrieval method and system. The method comprises: obtaining a tag associated with a question input by a user from a repository; obtaining a first number of large text blocks from the repository according to the tag; performing vector matching calculation on a vector representation of the question input by the user and a vector representation of a plurality of small text blocks in the repository, and obtaining a second number of small text blocks according to the calculation result; obtaining a third number of large text blocks associated with the second number of small text blocks from the repository; providing a collection or intersection of the first number of large text blocks and the third number of large text blocks as supplementary corpus to a semantic retrieval model so that the semantic retrieval model gives a query result, wherein the repository pre-establishes and stores relevant data. The method and system adopt a hierarchical management mode of text blocks, and improves the efficiency and accuracy of document recall through a dual-path recall mechanism.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of computer technology, and in particular to an intelligent document management and retrieval method and system. Background Art

[0002] The existing classification and management of rules and regulations mainly rely on manual operations, including a series of activities such as drafting, compiling, soliciting opinions, modifying, countersigning, publishing, and implementing the rules and regulations. Rules and regulations are usually classified according to their nature and content, such as personnel management system, financial management system, production management system, safety management system, etc. The specific classification needs to be adjusted according to the actual situation of the enterprise.

[0003] At present, these rules and regulations are systematically or labeled through information technology or traditional artificial intelligence (AI), which has improved the efficiency of system management to a certain extent. However, existing processing solutions usually require a lot of manual intervention and preset rules, and are difficult to adapt to the changes and complexity of system content. In addition, traditional AI models usually focus on specific keywords or phrases, while ignoring the context and overall meaning of the system content, which may lead to insufficient accuracy and comprehensiveness of classification. Summary of the invention

[0004] The purpose of the embodiments of the present disclosure is to provide an intelligent document management and retrieval method and system to solve the problems existing in the prior art.

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided an intelligent document management and retrieval method, comprising:

[0006] Based on the user input question, obtain the tags associated with it from the repository;

[0007] According to the tag, obtaining a first number of large text blocks from the repository;

[0008] Performing vector matching calculation on the vector representation of the user input question and the vector representation of the plurality of small text blocks in the repository, and obtaining a second number of small text blocks according to the calculation result;

[0009] According to the second number of small text blocks, obtaining a third number of large text blocks associated therewith from the storage repository;

[0010] providing a collection or an intersection of the first number of large text blocks and the third number of large text blocks as supplementary corpus to a semantic search model so that the semantic search model gives a query result,

[0011] The repository is pre-established and used to store large text blocks of various rules and regulations documents, vector representations of small text blocks, correspondences between large text blocks and small text blocks, and relationships between large text blocks and labels.

[0012] In some embodiments, the vector matching calculation uses one of the following algorithms: cosine similarity, Euclidean distance, and Jaccard similarity.

[0013] In some embodiments, obtaining the second number of small text blocks according to the calculation result is one of the following methods:

[0014] Sorting the plurality of small text blocks in the storage repository from large to small according to the calculation results, and obtaining the second largest number of small text blocks;

[0015] One or more small text blocks among the plurality of small text blocks in the storage repository whose matching degree is greater than a set threshold are recorded as the second number of small text blocks.

[0016] In some embodiments, keywords are extracted from the user input question, and tags associated with the keywords are obtained from a repository based on the keywords.

[0017] In some embodiments, the method further includes: using a model to segment the rules and regulations document to obtain small text blocks; and merging one or more small text blocks into a large text block.

[0018] In some embodiments, the tags in the repository include manually created knowledge base tags and generated tags.

[0019] In some embodiments, a segmentation model for regulatory documents is obtained by fine-tuning a pre-trained model based on a training dataset, and the training dataset is obtained by annotating semantic relationships represented by proper nouns and text formats in the regulatory documents.

[0020] In some embodiments, the preferred granularity sizes of small text blocks and large text blocks are determined through multiple fine-tuning of a pre-trained model.

[0021] According to a second aspect of an embodiment of the present disclosure, there is provided an intelligent document management and retrieval system, comprising:

[0022] Document segmentation module, which is used to segment regulatory documents of various formats into large text blocks and small text blocks, and vectorize small text blocks;

[0023] Tag management module, used to manage tags for large text blocks;

[0024] A knowledge base management module, which is used to store and manage various rules and regulations documents, large text blocks, small text blocks, vector representations of small text blocks, correspondences between large text blocks and small text blocks, and relationships between large text blocks and tags;

[0025] A retrieval recall module is used to obtain the corresponding small text block and the large text block corresponding to the corresponding small text block from the knowledge base management module according to the user input question, and simultaneously obtain the corresponding large text block by the user input, and use the collection or intersection of the two large text blocks obtained as the supplementary corpus of the semantic retrieval model to obtain the corresponding retrieval result.

[0026] According to a third aspect of an embodiment of the present disclosure, there is provided a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned intelligent document management and retrieval method is implemented.

[0027] The intelligent document management and retrieval system and method provided by the embodiment of the present disclosure adopts a hierarchical management mode of text blocks, and improves the efficiency and accuracy of document recall through a dual-path recall mechanism. In addition, the method and system also determine the granularity of the preferred large and small text blocks suitable for regulatory documents through multiple fine-tuning of the pre-trained model. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The above and other objects, features and advantages of the embodiments of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0029] Figure 1 is a block diagram of an intelligent document management and retrieval system provided by an embodiment of the present disclosure;

[0030] Figure 2 The embodiment of the present disclosure provides Figure 1 Flowchart of the document chunking module in ;

[0031] Figure 3 The embodiment of the present disclosure provides Figure 1 Flowchart of the retrieval and recall module in . DETAILED DESCRIPTION

[0032] The embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. In each of the accompanying drawings, the same elements are represented by similar reference numerals. For the sake of clarity, the various parts in the accompanying drawings are not drawn to scale. In addition, some well-known parts may not be shown.

[0033] The following describes the embodiments of the present disclosure based on the embodiments, but the embodiments of the present disclosure are not limited to these embodiments. In the detailed description of the embodiments of the present disclosure below, some specific details are described in detail. For those skilled in the art, the embodiments of the present disclosure can be fully understood without the description of these details. In order to avoid confusing the essence of the embodiments of the present disclosure, well-known methods, processes, processes, components and circuits are not described in detail.

[0034] Unless the context clearly requires otherwise, the words "include", "comprising", and the like throughout the specification and claims should be interpreted as inclusive rather than exclusive or exhaustive; that is, as "including but not limited to". In the description of the embodiments of the present disclosure, it should be understood that the terms "first", "second", etc. are used for descriptive purposes only and are not to be understood as indicating or implying relative importance. In addition, in the description of the embodiments of the present disclosure, unless otherwise specified, "plurality" means two or more.

[0035] The disclosed embodiments relate to the centralized management and retrieval of rules and regulations documents. The processing logic of the centralized management of rules and regulations documents is: first collect rules and regulations documents of various formats, then divide the rules and regulations documents into small text blocks, integrate the small text blocks into large text blocks, generate labels from the large text blocks, generate vector representations from the small text blocks, and then store the original rules and regulations documents, store large text blocks, small text blocks, vector representations of small text blocks, correspondences between large and small text blocks, and labels of large text blocks. The retrieval processing logic of the rules and regulations documents is: obtain the small text block corresponding to the user input through user input, obtain the corresponding large text block from the correspondence between the small text block corresponding to the user input and the large and small text blocks managed centrally, and simultaneously obtain the corresponding large text block from the user input, and combine the large text blocks obtained twice as a supplementary corpus for the semantic retrieval model to obtain the corresponding retrieval results.

[0036] Unless otherwise stated, the various models used in the embodiments of the present application can be implemented by fine-tuning the pre-trained model. Usually, a suitable pre-trained model is selected based on the application scenario and functional requirements, and then the parameters of the pre-trained model are fine-tuned using the training data set.

[0037] Figure 1 1 is a block diagram of an intelligent document management and retrieval system provided by an embodiment of the present disclosure. As shown in the figure, it includes a retrieval and recall module 101, a document segmentation module 102, a tag management module 103 and a knowledge base management module 104.

[0038] The main function of the document segmentation module 102 is to segment regulatory documents of various formats into large text blocks and small text blocks, and to vectorize the small text blocks.

[0039] The main function of the tag management module 103 is to manage the tags of large text blocks. The tag management module 103 adopts two-level management for tags, namely knowledge base tags and large text block tags. The knowledge base tag is manually selected and set when the knowledge base is established. This tag can be combined with the prompt word and its extension word in the upstream system to provide more accurate prompt options for user input, reduce the semantic deviation of the content caused by user input, and increase the accuracy of the system. Large text block tags can be generated by the document segmentation module 102. In this tag management module 103, users can modify, delete and set tags through manual operations, which is convenient for improving the correlation and accuracy between data and tags during subsequent recall work.

[0040] The knowledge base management module 104 is the core storage library of the system, which not only stores the rules and regulations documents of various formats imported by users, as well as the large and small text blocks after the documents are segmented. The knowledge base management module 104 also provides effective management and maintenance of this information, including but not limited to the functions of knowledge base construction, update and query. In order to achieve efficient data storage and retrieval, the knowledge base management module 104 uses a relational database, a vector database and a file library. The relational database is responsible for storing system background data and knowledge base related information, the vector database is responsible for storing the vector data generated after the document is segmented, and the file library is responsible for storing the rules and regulations documents of various formats imported by users.

[0041] The processing process of the document segmentation module 102 mainly includes document parsing, document slicing and text block vectorization.

[0042] Document parsing is an important starting point for data processing. It converts various formats of rules and regulations documents into data that can be used by the system. At present, the document parsing function of the system has the ability to process multiple data types, including but not limited to the file formats of Microsoft Office software (such as .docx, .xlsx and .txt), as well as CSV, HTML, JSON, Markdown, PDF and other common data formats. In this process, regardless of the original format of the file, the system can first parse the file content and convert it into text content of string type. This text content can be used as input for subsequent data processing processes, enabling the system to further process and analyze the data. Optionally, the text content of string type can also be preprocessed, such as removing irrelevant information through data cleaning, filtering noise and standardizing the text content to provide clear and standardized text content, thereby improving the accuracy of analysis and processing.

[0043] Document slicing is the core of document processing. To ensure the quality of slicing, this embodiment adopts a text block hierarchical management mode. In this mode, the system first segments the text content output by document parsing through a segmentation model to obtain small text blocks.

[0044] For example, the segmentation model is obtained by fine-tuning the BERT (Bidirectional Encoder Representations from Transformers) model, which stands out for its deep language understanding capabilities. As a pre-trained model, a key advantage of the BERT model is its bidirectional training feature, which enables the model to understand the context on the left and right sides of a given text at the same time, thereby gaining a comprehensive understanding of the text. Compared with the traditional method of segmenting according to rules, the segmentation model obtained by fine-tuning the BERT model implements semantics-based segmentation. This has the advantage of ensuring that the content of the segmented text block is a complete sentence without being truncated in the middle, and enhancing the relevance and semantic consistency between text blocks.

[0045] The pre-trained BERT model can be fine-tuned in a supervised manner. Specifically, a training data set is prepared for the training model, and the key features in the rules and regulations are marked to obtain the training data set, including but not limited to: marking the proper nouns in the rules and regulations (including the proper nouns for the issuing units and administrative levels, roles, regional information, target audiences, priorities, and importance of the rules and regulations); marking the semantic relationships represented by the text format in the rules and regulations and the semantic relationships represented by other methods, such as marking sentences that represent primary and secondary relationships, causal relationships, parallel relationships, and progressive relationships represented by text formats such as unsigned lists, signed lists, and dashes. Then, the BERT model is trained using the training data set. The training gradually adjusts the parameters according to the set hyperparameters and optimization algorithms to complete the model's recognition of the labeled content. Finally, the fine-tuned BERT model is evaluated on the validation data set to ensure the required accuracy of the fine-tuned BERT model. In this way, the fine-tuned segmentation model can not only recognize the proper nouns in rules and regulations, but also recognize the semantic relationships between sentences or phrases, such as primary and secondary relationships, causal relationships, parallel relationships, and progressive relationships, and adaptively segment small text blocks based on these relationships.

[0046] Furthermore, the system uses an integration algorithm to merge the small text blocks generated by the model segmentation into large text blocks. The system usually first sets a limit on the particle size of the large and small text blocks in the model. For example, if the characters of the small text block are between 200 and 400 characters, based on this limit, the fine-tuned segmentation model will adaptively place sentences with semantic relationships such as primary and secondary relationships, causal relationships, parallel relationships, and progressive relationships in a small text block or in several small text blocks. For example, for a regulations document, the fine-tuned segmentation model will place the main sentence and several subordinate clauses representing the master-slave relationship in the same small text block to achieve semantic completeness, while for another regulations document, the fine-tuned segmentation model will place the main sentence and several subordinate clauses representing the master-slave relationship in the same small text block to meet the limit on the particle size of the small text block. For another example, if the large text block is limited to between 800 and 1200 characters, one or several small text blocks will be merged into one large text block based on this limit and the semantic relationship between the small text blocks. At the same time, the particle size of the limited large text blocks and small text blocks can also be adjusted according to the actual application effect, with the aim of ensuring that the small text blocks can provide the minimum information unit of the rules and regulations document to the greatest extent, while the large text blocks can ensure the integrity of the information without excessive redundant information.

[0047] In the process of segmenting small text blocks and merging them into large text blocks, the system will also establish associations between large and small text blocks, and the association data will be stored in the knowledge base management module 104. In some cases, the system will use models to perform in-depth analysis on these large text blocks, extract key information and form tags to store in the tag management module 103. For example, the Baichuan2-13B-Chat-4bits model is an open source large model, which can set context for the model through prompt words and generate tags that conform to the context semantics for the text according to needs.

[0048] The hierarchical management mode of large and small text blocks in this embodiment can not only ensure the information integrity of the text blocks, but also effectively maintain the contextual relationship between text blocks, thereby greatly improving the efficiency and quality of document slicing.

[0049] In some embodiments, the above text block hierarchical management method is as follows: Figure 2The flowchart is presented in steps, including receiving uploaded documents in step S201, saving to the file library in step S202, parsing document content into strings in step S203, dividing the string content into small text blocks in step S204, generating large text blocks and establishing indexes between large and small text blocks in step S205, vectorizing small text blocks in step S206, generating labels for large text blocks in step S207, saving vector representations of small text blocks in the vector database in step S208, and saving the correspondence between large text blocks and small text blocks, the correspondence between small text blocks and labels, and the correspondence between small text blocks and vectors in the relational database in step S209. In this process, the association relationship table between large text blocks and small text blocks will be established through the primary key and foreign key of the relational database, and then these large text blocks will also be saved in the relational database for use as corpus during recall. Vector storage is performed for small text blocks and labels. This process uses a model (such as the bge-large-zh-v1.5 model) to implement text vectorization. Document slices and labels need to be passed into the model for automatic vectorization. The generated vector data will eventually be stored in the vector database.

[0050] The retrieval recall module 101 is the output module of the system, which is mainly used to recall relevant text blocks from the knowledge base management module 104 according to the user input questions, and submit the relevant text blocks to the semantic retrieval model as corpus supplement, so as to provide more accurate answers to the user's questions.

[0051] The core of the retrieval and recall module 101 is how to accurately recall relevant text blocks based on the user's questions. Figure 3 The embodiment adopts a two-way recall mechanism, including the following steps.

[0052] In step S301, a question input by a user is received.

[0053] This step uses keyword extraction from the user input question and converts the user input question and the extracted keywords into vector representations. Optionally, this step uses a model to extract keywords from the user input question and converts them into vector representations, and the model can also be obtained by fine-tuning a pre-trained model.

[0054] In step S302, according to the question input by the user, the tag associated with it is retrieved from the repository.

[0055] In this step, the keyword extracted in step S301 may be used to obtain a tag matching the keyword from the tags stored in the tag management module 103 .

[0056] In step S303, vector matching is performed between the vector representation of the user input question and the vector representation of multiple small text blocks in the storage repository, and a second number of small text blocks are obtained according to the vector matching result.

[0057] The vector matching in this step is a commonly used method for measuring text similarity. The basic idea of ​​vector matching is to represent the text as a vector, and then calculate the distance or angle between different vectors to measure the similarity between the vectors. The vector matching algorithms that can be used in the vector matching in this step are, for example, cosine similarity, Euclidean distance, and Jaccard similarity, among which cosine similarity is the preferred vector matching algorithm. The vector matching in this step can be executed in batches by the vector database to obtain efficient execution efficiency.

[0058] In some embodiments, this step may use the vector representation of the user input question in step S301 or the vector representation of the extracted keywords to perform vector matching with the vector representation of each small text block in the vector database of the knowledge base management module 104, and use the small text blocks corresponding to the vector representation of the small text blocks with a matching degree greater than a set threshold as the second number of small text blocks.

[0059] In other embodiments, this step may use the vector representation of the user input question in step S301 or the vector representation of the extracted keywords to perform vector matching with the vector representation of each small text block in the vector database of the knowledge base management module 104, and sort the multiple small text blocks from large to small according to the calculation results, and obtain the second number of small text blocks with the highest sorting.

[0060] In step S304, a first number of large text blocks are obtained from a storage repository according to the tags.

[0061] This step can use the label output in step S302 to retrieve the corresponding relationship data between the label and the large text block in the knowledge base management module 104, and obtain the corresponding first number of large text blocks, and put these large text blocks into the rough sorting text block set.

[0062] In step S305, according to the second number of small text blocks, a third number of large text blocks associated therewith are obtained from the storage library.

[0063] This step can use the second number of small text blocks output from step S303 to reversely retrieve the correspondence data between large text blocks and small text blocks in the knowledge base management module 104, and obtain the corresponding third number of large text blocks, and put these large text blocks into the refined text block set.

[0064] In step S306, the union or intersection of the first number of large text blocks and the third number of large text blocks is provided to the semantic retrieval model.

[0065] Step S306 is to provide the collection or intersection of the roughly sorted text block set and the finely sorted text block set to the semantic retrieval model so that the semantic retrieval model gives the question result for the collection.

[0066] The system and method provided by the embodiments of the present disclosure adopt a hierarchical management mode of text blocks and a dual-path recall mechanism. One path of recall is keyword matching (i.e., using the keywords of the user input question to match the pre-stored tags) to obtain a group of large text blocks, and the other path of recall is semantic matching (i.e., using the vector representation of the user input question to match the vector of the pre-stored small text block) to obtain another two groups of large text blocks. The combination of keyword matching and semantic matching can not only make up for the limitations of a single text matching method, but the complementarity of the two also helps to improve the comprehensiveness and accuracy of the recalled materials, thereby using the two groups of text blocks to provide comprehensive and accurate answers to the user input questions.

[0067] In a specific implementation, the above-mentioned scheme can be executed by a computer device, which can be a terminal or a server, including the above-mentioned system, the model execution involved, and the model training can all be executed by the terminal and / or the server. For example, the server can execute the relevant process of model fine-tuning (for example, using the text content of the marked rules and regulations document to improve the BERT model) to obtain the target model, and the target model can be deployed to the server. During the model application process, the terminal can obtain the user input question and send the user input question to the server; the server can call various services to execute together to obtain the problem result.

[0068] In a specific implementation, the above-mentioned solution can be formed into a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the operations in the above embodiments are implemented.

[0069] The embodiments according to the embodiments of the present disclosure are described above, and these embodiments do not describe all the details in detail, nor do they limit the invention to the specific embodiments described. Obviously, many modifications and changes can be made based on the above description. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of the present disclosure, so that those skilled in the art can make good use of the embodiments of the present disclosure and the modified use based on the embodiments of the present disclosure. The embodiments of the present disclosure are limited only by the claims and their full scope and equivalents.

Claims

1. An intelligent document management and retrieval method, comprising: Based on the user input question, obtain the tags associated with it from the repository; According to the tag, obtaining a first number of large text blocks from the repository; Performing vector matching calculation on the vector representation of the user input question and the vector representation of the plurality of small text blocks in the repository, and obtaining a second number of small text blocks according to the calculation result; According to the second number of small text blocks, obtaining a third number of large text blocks associated therewith from the storage repository; providing a collection or an intersection of the first number of large text blocks and the third number of large text blocks as supplementary corpus to a semantic search model so that the semantic search model gives a query result, The repository is pre-established and used to store large text blocks, small text blocks, vector representations of small text blocks, correspondences between large text blocks and small text blocks, and relationships between large text blocks and labels of various rules and regulations documents.

2. The search method according to claim 1, wherein: The vector matching calculation adopts one of the following algorithms: cosine similarity, Euclidean distance, and Jaccard similarity.

3. The search method according to claim 1, wherein: The method of obtaining the second number of small text blocks according to the calculation result is one of the following methods: Sorting the plurality of small text blocks in the storage repository from large to small according to the calculation results, and obtaining the second largest number of small text blocks; One or more small text blocks among the plurality of small text blocks in the storage repository whose matching degree is greater than a set threshold are recorded as the second number of small text blocks.

4. The intelligent document management and retrieval method according to claim 1, wherein: Keywords are extracted from the user input question, and tags associated with the keywords are obtained from a repository based on the keywords.

5. The intelligent document management and retrieval method according to claim 1, further comprising: The model is used to segment the rules and regulations document to obtain small text blocks; and merge one or more small text blocks into a larger text block.

6. The intelligent document management and retrieval method according to claim 1, wherein: The tags in the repository include manually created knowledge base tags and generated tags.

7. The intelligent document management and retrieval method according to claim 1, wherein: A segmentation model for regulatory documents is obtained by fine-tuning a pre-trained model based on a training dataset, and the training dataset is obtained by annotating semantic relationships represented by proper nouns and text formats in the regulatory documents.

8. The intelligent document management and retrieval method according to claim 7, wherein: The preferred granularity sizes of small and large text blocks are determined through multiple fine-tuning of the pre-trained model.

9. An intelligent document management and retrieval system, comprising: Document segmentation module, which is used to segment regulatory documents of various formats into large text blocks and small text blocks, and vectorize small text blocks; Tag management module, used to manage tags for large text blocks; A knowledge base management module, which is used to store and manage various rules and regulations documents, large text blocks, small text blocks, vector representations of small text blocks, correspondences between large text blocks and small text blocks, and relationships between large text blocks and tags; A retrieval recall module is used to obtain the corresponding small text block and the large text block corresponding to the corresponding small text block from the knowledge base management module according to the user input question, and simultaneously obtain the corresponding large text block by the user input, and use the collection or intersection of the two large text blocks obtained as the supplementary corpus of the semantic retrieval model to obtain the corresponding retrieval result.

10. A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the intelligent document management and retrieval method according to any one of claims 1 to 8.

Images