Metadata-based document rapid labeling retrieval method and system

Abstract

The invention relates to the technical field of information processing, and discloses a metadata-based document rapid labeling retrieval method and system.The method comprises the steps that a metadata extraction task, a task execution strategy and a storage strategy are created through a metadata extraction module; extracting the target document based on the task execution strategy, the metadata extraction task and the storage strategy to obtain a first metadata document; performing structured information extraction on the first metadata document by utilizing a metadata labeling module to obtain a second metadata document, and performing document labeling based on association between the second metadata document and the target document on the second metadata document to obtain a labeled document; constructing a query index based on the metadata and the tagged document by utilizing a metadata retrieval module, quickly positioning the query index through a query condition input by a user, and querying information in the target document based on the query index; according to the method and the device, the problems of relatively low information processing speed and relatively low data processing accuracy in the existing automatic document retrieval are solved.

Description

Technical Field

[0001] The present invention relates to the field of information processing technology, and in particular to a metadata-based document rapid annotation and retrieval method and system. Background Art

[0002] With the rapid development of information technology, the number of documents generated by enterprises, scientific research institutions and individuals in work and life has exploded. Faced with the explosive growth of document data, traditional manual annotation and retrieval methods have been unable to meet the needs of quickly and accurately obtaining the required data. At present, although some existing automated document management methods can achieve automatic classification and retrieval of documents to a certain extent, these methods often need to rely on complex natural language processing algorithms. Once the algorithm is too complex, the data processing speed will become slower and the adaptability to specific fields will also become worse. In addition, when performing document retrieval, the existing methods will ignore a large amount of metadata contained in the document (for example: author information, creation time, keywords, subject classification, etc.) in order to increase the speed as much as possible. These metadata play an important role in improving the accuracy of retrieval. It can be seen that the existing automated document retrieval has the problems of slow information processing speed and low data processing accuracy. Summary of the invention

[0003] The present invention provides a metadata-based document rapid annotation retrieval method and system to solve the problems of slow information processing speed and low data processing accuracy in existing automated document retrieval.

[0004] In order to achieve the above object, the present invention is implemented by the following technical solutions: In a first aspect, the present invention provides a metadata-based document rapid annotation and retrieval method, which is applied to a metadata-based document rapid annotation and retrieval system, the system comprising: a metadata extraction module, a metadata annotation module and a metadata retrieval module, the method comprising: Creating a metadata extraction task, a task execution strategy, and a storage strategy through a metadata extraction module, and extracting metadata from a target document based on the task execution strategy, the metadata extraction task, and the storage strategy to obtain a first metadata document; Using a metadata annotation module to extract structured information from the first metadata document to obtain a second metadata document, and annotating the second metadata document based on the association between the second metadata document and the target document to obtain an annotated document; A query index based on the metadata and the annotated document is constructed using a metadata retrieval module, the query index is quickly located through a query condition input by a user, and information in a target document is queried based on the query index.

[0005] Optionally, the metadata extraction task includes: the type of metadata; The task execution strategy includes: task execution time and task cycle time; The warehousing measurement includes full warehousing and incremental warehousing.

[0006] Optionally, extracting structured information from the first metadata document to obtain a second metadata document includes: The first metadata document is subjected to document preprocessing, document segmentation, and document element extraction to obtain a second metadata document, wherein the document preprocessing includes: removing useless formats, spaces, and special characters in the first metadata document by data cleaning, and then processing the document format in the same first metadata document by standardization; Document segmentation includes: segmenting the document data in the first metadata document into small document units, the small document units being: paragraphs or sentences; Document element extraction includes: extracting specific information elements from the first metadata document through regular expressions, natural language processing tools, deep learning and template matching to construct a second metadata document.

[0007] Optionally, the regular expression is: a character string that matches a specific pattern; The natural language processing tool is: using the NLP library to perform part-of-speech tagging and named entity recognition to identify and extract specific entities in the document; The deep learning is: using a deep learning model to perform sequence labeling, identify and extract specific elements in the document; The template matching is: using a specific template to match a document and extracting a matching format or structure in the document.

[0008] Optionally, the step of annotating the second metadata document based on the association between the second metadata document and the target document to obtain an annotated document includes: Reading existing metadata from the second metadata document, and associating the second data document with the read existing metadata in an automatic or manual manner; The associated document metadata is stored in the database to generate the corresponding annotated document.

[0009] In a second aspect, an embodiment of the present application provides a metadata-based document rapid annotation and retrieval system, the system comprising: a metadata extraction module, a metadata annotation module and a metadata retrieval module: A metadata extraction module, used to create a metadata extraction task, a task execution strategy and a storage strategy, and further used to extract metadata from a target document based on the task execution strategy, the metadata extraction task and the storage strategy to obtain a first metadata document; A metadata annotation module, configured to extract structured information from the first metadata document to obtain a second metadata document, and to annotate the second metadata document based on the association between the second metadata document and a target document to obtain an annotated document; The metadata retrieval module is used to construct a query index based on the metadata and the annotated document, quickly locate the query index through the query conditions input by the user, and query the information in the target document based on the query index.

[0010] Optionally, the metadata retrieval module includes: an index construction module, a query processing module, a retrieval algorithm module and a user interaction module; The index building module is used to build a query index; The query processing module is used to parse user input information and optimize query results; The retrieval algorithm module is used to match the retrieval algorithm according to the user input information; The user interaction module is used to obtain user input information and also to display search results.

[0011] Beneficial effects: The metadata-based document rapid annotation and retrieval method provided by the present invention improves retrieval efficiency: by directly using metadata for retrieval, the complex natural language processing process is avoided, and the retrieval speed is significantly improved; the retrieval accuracy is enhanced: the accuracy and consistency of metadata ensure the accuracy of the retrieval results, and the false detection and missed detection are reduced; the technical threshold is lowered: the dependence on complex natural language processing technology is reduced, and the cost and difficulty of system development and maintenance are reduced; the user experience is improved: the friendly user interface and flexible query method enable users to manage and retrieve documents more conveniently. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 A flowchart of a metadata-based document rapid annotation and retrieval method according to a preferred embodiment of the present invention; Figure 2 A flowchart of a method for implementing metadata extraction according to a preferred embodiment of the present invention; Figure 3 A flowchart of a metadata annotation implementation method according to a preferred embodiment of the present invention; Figure 4 The flowchart of the metadata retrieval implementation method of the preferred embodiment of the present invention. DETAILED DESCRIPTION

[0013] The technical solution of the present invention is described clearly and completely below. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0014] Unless otherwise defined, the technical terms or scientific terms used in the present invention shall have the usual meanings understood by persons with ordinary skills in the field to which the present invention belongs. The words "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, words such as "one" or "one" do not indicate quantity restrictions, but indicate the existence of at least one. Words such as "connect" or "connected" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the object being described changes, the relative positional relationship also changes accordingly.

[0015] See also Figure 1 The embodiment of the present application provides a method for rapid annotation and retrieval of documents based on metadata, which is applied to a system for rapid annotation and retrieval of documents based on metadata. The system includes: a metadata extraction module, a metadata annotation module and a metadata retrieval module. The method includes: Creating a metadata extraction task, a task execution strategy, and a storage strategy through a metadata extraction module, and extracting metadata from a target document based on the task execution strategy, the metadata extraction task, and the storage strategy to obtain a first metadata document; Using a metadata annotation module to extract structured information from the first metadata document to obtain a second metadata document, and annotating the second metadata document based on the association between the second metadata document and the target document to obtain an annotated document; A query index based on the metadata and the annotated document is constructed using a metadata retrieval module, the query index is quickly located through a query condition input by a user, and information in a target document is queried based on the query index.

[0016] Optionally, the metadata extraction task includes: the type of metadata; The task execution strategy includes: task execution time and task cycle time; The warehousing measurement includes full warehousing and incremental warehousing.

[0017] In the above embodiment, if Figure 2As shown, through the metadata extraction module, an extraction task is established for the database under the information library, a scheduling plan is configured, and automatic extraction of structured metadata is performed to obtain technical metadata, business metadata, and management metadata.

[0018] 1) Create a new extraction task Metadata extraction provides a new feature for metadata extraction tasks, which enables the visual construction of metadata extraction tasks.

[0019] 2) Set basic information This function allows you to set the task name, execution strategy, and storage strategy. The execution strategy is mainly the execution time strategy, for example: execute at midnight every day; execute at 18:00 on Monday, Wednesday, and Friday, etc.; the storage strategy mainly includes full and incremental.

[0020] 3) Select the data source Set the data source for metadata extraction. The data comes from the data source connected under Data Collection → Data Source Management >.

[0021] 4) Start the extraction task After starting the extraction task, the extraction task will be started according to the set execution strategy rules. For example: set the support strategy to be executed at midnight every day, and the storage strategy is full. When the start button is clicked, the task will be executed at midnight the next day.

[0022] 5) Execute the extraction task After executing the extraction task, the extraction task will be started immediately according to the set execution strategy rules. For example: set the support strategy to be executed at midnight every day, and the storage strategy is full. When the execution button is clicked, the extraction task will be executed immediately according to the full extraction method.

[0023] Optionally, extracting structured information from the first metadata document to obtain a second metadata document includes: The first metadata document is subjected to document preprocessing, document segmentation, and document element extraction to obtain a second metadata document, wherein the document preprocessing includes: removing useless formats, spaces, and special characters in the first metadata document by data cleaning, and then processing the document format in the same first metadata document by standardization; Document segmentation includes: segmenting the document data in the first metadata document into small document units, the small document units being: paragraphs or sentences; Document element extraction includes: extracting specific information elements from the first metadata document through regular expressions, natural language processing tools, deep learning and template matching to construct a second metadata document.

[0024] Optionally, the regular expression is: a character string that matches a specific pattern; The natural language processing tool is: using the NLP library to perform part-of-speech tagging and named entity recognition to identify and extract specific entities in the document; The deep learning is: using a deep learning model to perform sequence labeling, identify and extract specific elements in the document; The template matching is: using a specific template to match a document and extracting a matching format or structure in the document.

[0025] Optionally, the step of annotating the second metadata document based on the association between the second metadata document and the target document to obtain an annotated document includes: Reading existing metadata from the second metadata document, and associating the second data document with the read existing metadata in an automatic or manual manner; The associated document metadata is stored in the database to generate the corresponding annotated document.

[0026] In the above embodiment, if Figure 3 As shown, through structured information extraction, we can obtain information such as the title, author, creation date and keywords of qualitative documents. Then, by using metadata-based document annotation, we can generate technical metadata, salesperson data and management metadata of the stereotyped documents to provide basic data for metadata retrieval.

[0027] 1) Structured information extraction Structured information extraction is the process of converting unstructured document data into structured data, including document preprocessing, document segmentation and document element extraction.

[0028] Document preprocessing: remove useless formats, spaces, special characters, etc. through data cleaning; unify document formats through standardization, such as the representation of dates and currencies.

[0029] Document segmentation: Split a document into paragraphs, sentences, or smaller units for further processing.

[0030] Document element extraction: The process of extracting specific information elements from a document, such as text, numbers, dates, names, places, etc. The methods used include regular expressions, natural language processing tools, deep learning, and template matching.

[0031] a) Regular expressions: Use regular expressions to match strings with specific patterns, such as phone numbers, email addresses, date formats, etc.

[0032] b) Natural language processing tools, which use NLP libraries to perform tasks such as part-of-speech tagging and named entity recognition to identify and extract specific entities in the text.

[0033] c) Deep learning: Use deep learning models to perform sequence labeling tasks to identify and extract specific elements in documents.

[0034] d) Template matching: using template matching methods to extract fixed formats or structures contained in documents.

[0035] 2) Document annotation based on metadata Metadata-based document annotation is a process of attaching structured information to document content in order to describe the key features and content of the document. Its implementation steps include metadata reading, metadata association, manual review, and metadata storage.

[0036] a) Read metadata, read existing technical metadata, business metadata and management metadata from the information database; b) Metadata association: Use automatic or manual methods to associate documents and metadata. The manual method provides a visual operation interface and is implemented through manual selection. The automatic method uses semantic analysis methods to achieve automatic association between documents and metadata.

[0037] c) Manual review, which provides visual operations to view and review metadata associated automatically or manually, and can modify metadata that has been associated with the document.

[0038] d) Metadata storage: storing the associated and reviewed document metadata in the database for subsequent retrieval and management.

[0039] The embodiment of the present application also provides a metadata-based document rapid annotation and retrieval system, the system comprising: a metadata extraction module, a metadata annotation module and a metadata retrieval module: A metadata extraction module, used to create a metadata extraction task, a task execution strategy and a storage strategy, and further used to extract metadata from a target document based on the task execution strategy, the metadata extraction task and the storage strategy to obtain a first metadata document; A metadata annotation module, configured to extract structured information from the first metadata document to obtain a second metadata document, and to annotate the second metadata document based on the association between the second metadata document and a target document to obtain an annotated document; The metadata retrieval module is used to construct a query index based on the metadata and the annotated document, quickly locate the query index through the query conditions input by the user, and query the information in the target document based on the query index.

[0040] Optionally, the metadata retrieval module includes: an index construction module, a query processing module, a retrieval algorithm module and a user interaction module; The index building module is used to build a query index; The query processing module is used to parse user input information and optimize query results; The retrieval algorithm module is used to match the retrieval algorithm according to the user input information; The user interaction module is used to obtain user input information and also to display search results.

[0041] In the above embodiment, if Figure 4 As shown, metadata retrieval mainly provides index construction and query services, builds multiple indexes based on document annotation results, and provides fast query services based on user input and query conditions.

[0042] a) Index building module, inverted index building - creating a mapping from metadata to documents for fast retrieval; forward index building - maintaining the original content and attributes of the document for displaying retrieval results; index update strategy - processing newly added or modified documents to keep the index up to date.

[0043] b) Query processing module, query parsing - understanding user query intent, including query expansion and spelling correction; query optimization - improving queries to obtain more accurate or relevant results.

[0044] c) Retrieval algorithm module, matching algorithm - matching based on keywords, phrases or sentences; similarity calculation - using cosine similarity and Jaccard similarity to evaluate the relevance of documents to queries; sorting algorithm - sorting retrieval results according to relevance or other criteria.

[0045] d) User interaction module, user interface - provides search box, advanced search options; result display - displays search results in the form of lists, summaries, etc.; user feedback - collects user feedback on search results for improving the system.

[0046] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A metadata-based document rapid annotation and retrieval method, applied to a metadata-based document rapid annotation and retrieval system, characterized in that: The system comprises: a metadata extraction module, a metadata annotation module and a metadata retrieval module, and the method comprises: Creating a metadata extraction task, a task execution strategy, and a storage strategy through a metadata extraction module, and extracting metadata from a target document based on the task execution strategy, the metadata extraction task, and the storage strategy to obtain a first metadata document; Using a metadata annotation module to extract structured information from the first metadata document to obtain a second metadata document, and annotating the second metadata document based on the association between the second metadata document and the target document to obtain an annotated document; A query index based on the metadata and the annotated document is constructed using a metadata retrieval module, the query index is quickly located through a query condition input by a user, and information in a target document is queried based on the query index.

2. The metadata-based document rapid annotation and retrieval method according to claim 1, characterized in that: The metadata extraction task includes: the type of metadata; The task execution strategy includes: task execution time and task cycle time; The warehousing measurement includes full warehousing and incremental warehousing.

3. The metadata-based document rapid annotation and retrieval method according to claim 1, characterized in that: The extracting structured information from the first metadata document to obtain a second metadata document includes: The first metadata document is subjected to document preprocessing, document segmentation, and document element extraction to obtain a second metadata document, wherein the document preprocessing includes: removing useless formats, spaces, and special characters in the first metadata document by data cleaning, and then processing the document format in the same first metadata document by standardization; Document segmentation includes: segmenting the document data in the first metadata document into small document units, the small document units being: paragraphs or sentences; Document element extraction includes: extracting specific information elements from the first metadata document through regular expressions, natural language processing tools, deep learning and template matching to construct a second metadata document.

4. The method for rapid document annotation and retrieval based on metadata according to claim 3, characterized in that: The regular expression is: a string that matches a specific pattern; The natural language processing tool is: using the NLP library to perform part-of-speech tagging and named entity recognition to identify and extract specific entities in the document; The deep learning is: using a deep learning model to perform sequence labeling, identify and extract specific elements in the document; The template matching is: using a specific template to match a document and extracting a matching format or structure in the document.

5. The method for rapid document annotation and retrieval based on metadata according to claim 1, characterized in that: The step of annotating the second metadata document based on the association between the second metadata document and the target document to obtain an annotated document includes: Reading existing metadata from the second metadata document, and associating the second data document with the read existing metadata in an automatic or manual manner; The associated document metadata is stored in the database to generate the corresponding annotated document.

6. A metadata-based document rapid annotation and retrieval system, characterized in that: The system includes: a metadata extraction module, a metadata annotation module and a metadata retrieval module: A metadata extraction module, used to create a metadata extraction task, a task execution strategy and a storage strategy, and further used to extract metadata from a target document based on the task execution strategy, the metadata extraction task and the storage strategy to obtain a first metadata document; A metadata annotation module, configured to extract structured information from the first metadata document to obtain a second metadata document, and to annotate the second metadata document based on the association between the second metadata document and a target document to obtain an annotated document; The metadata retrieval module is used to construct a query index based on the metadata and the annotated document, quickly locate the query index through the query conditions input by the user, and query the information in the target document based on the query index.

7. The metadata-based document rapid annotation and retrieval system according to claim 6, characterized in that: The metadata retrieval module includes: an index construction module, a query processing module, a retrieval algorithm module and a user interaction module; The index building module is used to build a query index; The query processing module is used to parse user input information and optimize query results; The retrieval algorithm module is used to match the retrieval algorithm according to the user input information; The user interaction module is used to obtain user input information and also to display search results.

Images