Intelligent digital archive editing and research method based on AI

Abstract

The invention discloses an AI-based intelligent digital archive editing and research method, and relates to the technical field of digital archives, and the method comprises the following steps: S1, source data: archived digital archive attachments in an archive management system; s2, performing attachment analysis: performing accurate identification and extraction on attachment contents in different formats by adopting an OCR (Optical Character Recognition) technology, generating structured data of XML (Extensible Markup Language), JSON (JavaScript Object Notation) and other types, and storing converted text data into a document library; S3, cleaning the text data, and performing text error correction, data verification and format adjustment on the analyzed and output text data; and S4, extracting key information in the text, wherein the key information comprises text abstract extraction and text keyword extraction. Massive archive data are analyzed through deep mining and intelligent analysis, the thematic direction is accurately grasped, the editing and research result is automatically generated, the efficiency, quality and creativity of digital archive editing and research are greatly improved, and along with continuous development of the AI technology, the application prospect in the field of digital archive editing and research is wider.

Description

Technical Field

[0001] The present invention relates to the field of digital archive technology, and in particular to an AI-based intelligent digital archive compilation and research method. Background Art

[0002] Traditional archive compilation and research relies on the compilation and research personnel to determine the compilation and research topics based on the current status of the archives and the actual work of the enterprise, collect a large amount of relevant archival materials based on the selected compilation and research topics, and select the materials with reference value for sorting and compilation. This compilation and research process greatly increases the workload of the compilation and research personnel, and the breadth and depth of the compilation and research results are insufficient. Enterprises are also facing greater pressure in terms of funds and equipment required to support this process.

[0003] With the development of archival information technology, digital archive construction has become an important strategic choice for enterprise development. Editorial and research personnel can use digital archive management systems to quickly and accurately obtain the required archival editing and research materials. This process greatly improves the efficiency of collecting and organizing editing and research materials, and lays a solid foundation for subsequent editing and research work. Compared with the traditional archive editing and research process, editing and research with the help of archive management systems improves the accuracy of data retrieval, and multi-person collaborative online editing and research improves the efficiency and quality of team collaboration. However, the entire editing and research process, including key links such as the determination of topics, the query of related archives, and the compilation of editing and research results, still needs to be completed by editors and researchers themselves, which requires high professional capabilities of editors and researchers. Therefore, the necessity of intelligent digital archive editing and research based on AI is becoming increasingly prominent. For this reason, this application proposes an intelligent digital archive editing and research method based on AI to solve the above problems. Summary of the invention

[0004] 1. Technical issues to be solved

[0005] In view of the shortcomings of the prior art, the present invention provides an AI-based intelligent digital archive compilation and research method to solve the problems of heavy workload, low efficiency and poor quality in the traditional archive compilation and research process.

[0006] (II) Technical solution

[0007] To achieve the above objectives, the technical solution adopted by the present invention is: an AI-based intelligent digital archive compilation and research method, comprising the following steps:

[0008] S1: Source data, digital archive attachments that have been archived in the archive management system;

[0009] S2: Attachment parsing: Use OCR recognition technology to accurately identify and extract attachment contents in different formats, generate structured data such as XML and JSON, and save the converted text data to the document library;

[0010] S3: Text data cleaning, text error correction, data verification, and format adjustment of the parsed text data;

[0011] S4: Extract key information from the text, including text summary extraction and text keyword extraction;

[0012] S5: New word combination, based on the generated one or more keywords, determine the theme of the research topic;

[0013] S6: Search for editing and research topics of new words. To check whether the newly constructed editing and research topics have been created, the topics of the latest editing and research topics are used as query conditions to search for editing and research topics that have been created. After the search is completed, the search results are analyzed;

[0014] S7: Whether the research topic for searching new words has been created;

[0015] S8: The research topic has not been created, that is, the new word has not yet generated a research topic. Retrieve all documents associated with the new word to prepare data for generating research documents. Use Solr's reverse index to retrieve documents related to the new word. Unlike the forward index, the reverse index is document-oriented, mapping each term in the document to a document list. When querying Solr, all documents containing the term are found by searching the reverse list of the term.

[0016] S9: The research topic has been created. All documents associated with the new word are retrieved and compared with the document content in the created research topic.

[0017] S10: If the document contents are the same, the retrieved editing and research topics will be directly returned;

[0018] S11: If the document contents are different, the reference attachments in the topic will be combined with the new content according to the degree of difference and the existing topic documents. If there is new content, the editorial research content generated by the newly added text and summary will be added to the editorial research document. If the newly retrieved document content is completely different from the document content of the topic reference that has been created, it is necessary to update the topic document and regenerate the editorial research topic document.

[0019] Preferably, the text keyword extraction in step S4 is specifically as follows:

[0020] Extract keywords from the cleaned text data in the document library. The intelligent keyword extraction process is implemented by combining NLP (natural language processing) technology with LLM (large language model) technology;

[0021] First, use the TF-IDF algorithm in natural language processing technology to perform preliminary keyword extraction on the text content. The TF-IDF keyword extraction algorithm calculates the weight values ​​of the two statistical features of word frequency and inverse document frequency in the text, and uses the words with larger weight values ​​as the keywords of the text. The calculation formula for word frequency is as follows:

[0022]

[0023] Where n ij Indicates that the term is in the text d j The number of times it appears in k n k，j Indicates that in the text j The sum of all occurrences of the term in .

[0024] The calculation formula for inverse document frequency is as follows:

[0025]

[0026] Where |D| represents the total number of all texts, Indicates that it contains the term t i The number of texts;

[0027] The calculation formula of TF-IDF is as follows:

[0028] TF-IDF=TF i,j ×IDF i

[0029] Then, LLM technology is used to further extract keywords.

[0030] Preferably, the text summary extraction in step S4 is specifically as follows:

[0031] Since the maximum number of tokens that a large language model can receive is limited, the text content must be sliced ​​when processing massive text data. Text summary extraction is divided into three steps: slicing, summary extraction, and integration;

[0032] First, slice each text into appropriate sizes according to the maximum number of tokens that the large language model can accept;

[0033] Then, the prompt engineering is used to construct prompts for each slice to guide LLM to extract summaries, such as generating a summary for the text and setting prompts according to user needs;

[0034] Finally, the summaries are integrated together according to the order of the slices in the text, and then the integrated summaries are input into the LLM combined with the prompt engineering to generate the overall summary of the text.

[0035] Preferably, the attachment contents in different formats in step S2 include formats such as pdf, word, txt, png, tif, etc.

[0036] Preferably, the text data cleaning in step S3 specifically includes cleaning non-text elements, irrelevant characters, etc. in the text to ensure the availability and accuracy of the text content.

[0037] Preferably, the use of LLM technology to further extract keywords is as follows: the keyword candidate list obtained by the TF-IDF keyword extraction algorithm is input into a large pre-trained language model, and its powerful context-awareness ability is used to evaluate the correlation between these keyword pre-texts. This process goes deep into the specific position of the candidate keyword in the text context, the text environment of the context, and its role and influence in the entire text; through the self-attention mechanism, LLM can deeply understand and automatically parse the complex dependencies between words. This mechanism can not only extract high-frequency words, but also capture words that are not obvious in the statistical process of conventional methods but are relatively important in this article, thereby improving the accuracy of keyword extraction.

[0038] (III) Beneficial effects

[0039] The beneficial effects of the present invention are:

[0040] This AI-based intelligent digital archive compilation and research method analyzes massive amounts of archival data through deep mining and intelligent analysis, accurately grasps the subject direction, and automatically generates compilation and research results, greatly improving the efficiency, quality, and creativity of digital archive compilation and research. With the continuous development of AI technology, its application prospects in the field of digital archive compilation and research will be even broader. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is a flow chart of the intelligent analysis method of the present invention. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] like Figure 1 As shown, the present invention provides a technical solution: an AI-based intelligent digital archive compilation and research method, comprising the following steps:

[0044] S1: Source data, digital archive attachments that have been archived in the archive management system;

[0045] S2: Attachment parsing: Use OCR recognition technology to accurately identify and extract attachment contents in different formats, generate structured data such as XML and JSON, and save the converted text data to the document library;

[0046] S3: Text data cleaning, text error correction, data verification, and format adjustment of the parsed text data;

[0047] S4: Extract key information from the text, including text summary extraction and text keyword extraction;

[0048] S5: New word combination, based on the generated one or more keywords, determine the theme of the research topic;

[0049] S6: Search for editing and research topics of new words. To check whether the newly constructed editing and research topics have been created, the topics of the latest editing and research topics are used as query conditions to search for editing and research topics that have been created. After the search is completed, the search results are analyzed;

[0050] S7: Whether the research topic for searching new words has been created;

[0051] S8: The research topic has not been created, that is, the new word has not yet generated a research topic. Retrieve all documents associated with the new word to prepare data for generating research documents. Use Solr's reverse index to retrieve documents related to the new word. Unlike the forward index, the reverse index is document-oriented, mapping each term in the document to a document list. When querying Solr, all documents containing the term are found by searching the reverse list of the term.

[0052] S9: The research topic has been created. All documents associated with the new word are retrieved and compared with the document content in the created research topic.

[0053] S10: If the document contents are the same, the retrieved editing and research topics will be directly returned;

[0054] S11: If the document contents are different, the reference attachments in the topic will be combined with the new content according to the degree of difference and the existing topic documents. If there is new content, the editorial research content generated by the newly added text and summary will be added to the editorial research document. If the newly retrieved document content is completely different from the document content of the topic reference that has been created, it is necessary to update the topic document and regenerate the editorial research topic document.

[0055] The text keyword extraction in step S4 is specifically as follows:

[0056] Extract keywords from the cleaned text data in the document library. The intelligent keyword extraction process is implemented by combining NLP (natural language processing) technology with LLM (large language model) technology;

[0057] First, use the TF-IDF algorithm in natural language processing technology to perform preliminary keyword extraction on the text content. The TF-IDF keyword extraction algorithm calculates the weight values ​​of the two statistical features of word frequency and inverse document frequency in the text, and uses the words with larger weight values ​​as the keywords of the text. The calculation formula for word frequency is as follows:

[0058]

[0059] Where n ij Indicates that the term is in the text d j The number of times it appears in k n k,j Indicates that in the text j The sum of all occurrences of the term in .

[0060] The calculation formula for inverse document frequency is as follows:

[0061]

[0062] Where |D| represents the total number of all texts, Indicates that it contains the term t i The number of texts;

[0063] The calculation formula of TF-IDF is as follows:

[0064] TF-IDF=TF i，j ×IDF i Then, LLM technology is used to further extract keywords, and the keyword candidate list obtained by the TF-IDF keyword extraction algorithm is input into a large pre-trained language model, and its powerful context-awareness ability is used to evaluate the correlation between these keyword pre-texts. This process goes deep into the specific position of the candidate keywords in the text context, the text environment of the context, and its role and influence in the entire text; through the self-attention mechanism, LLM can deeply understand and automatically parse the complex dependencies between words. This mechanism can not only extract high-frequency words, but also capture words that are not obvious in the statistical process of conventional methods but are more important in this article, thereby improving the accuracy of keyword extraction.

[0065] The text summary extraction in step S4 is as follows:

[0066] Since the maximum number of tokens that a large language model can receive is limited, the text content must be sliced ​​when processing massive text data. Text summary extraction is divided into three steps: slicing, summary extraction, and integration;

[0067] First, slice each text into appropriate sizes according to the maximum number of tokens that the large language model can accept;

[0068] Then, the prompt engineering is used to construct prompts for each slice to guide LLM to extract summaries, such as generating a summary for the text and setting prompts according to user needs;

[0069] Finally, the summaries are integrated together according to the order of the slices in the text, and then the integrated summaries are input into the LLM combined with the prompt engineering to generate the overall summary of the text.

[0070] The attachment contents in different formats in step S2 include pdf, word, txt, png, tif, etc.

[0071] The text data cleaning in step S3 specifically involves cleaning non-text elements, irrelevant characters, etc. in the text to ensure the availability and accuracy of the text content.

[0072] In the description of the present invention, it is necessary to understand that the indicated orientation or positional relationship is based on the orientation or positional relationship shown in the accompanying drawings, and is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0073] In the present invention, unless otherwise clearly stipulated and limited, for example, it can be a fixed connection, a detachable connection, or an integrated one; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. Unless otherwise clearly defined, for ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances.

[0074] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An AI-based intelligent digital archive compilation and research method, characterized in that: The following steps are involved: S1: Source data, digital archive attachments that have been archived in the archive management system; S2: Attachment parsing: Use OCR recognition technology to accurately identify and extract attachment contents in different formats, generate structured data such as XML and JSON, and save the converted text data to the document library; S3: Text data cleaning, text error correction, data verification, and format adjustment of the parsed text data; S4: Extract key information from the text, including text summary extraction and text keyword extraction; S5: New word combination, based on the generated one or more keywords, determine the theme of the research topic; S6: Search for editing and research topics of new words. To check whether the newly constructed editing and research topics have been created, the topics of the latest editing and research topics are used as query conditions to search for editing and research topics that have been created. After the search is completed, the search results are analyzed; S7: Whether the research topic for searching new words has been created; S8: The research topic has not been created, that is, the new word has not yet generated a research topic. Retrieve all documents associated with the new word to prepare data for generating research documents. Use Solr's reverse index to retrieve documents related to the new word. Unlike the forward index, the reverse index is document-oriented, mapping each term in the document to a document list. When querying Solr, all documents containing the term are found by searching the reverse list of the term. S9: The research topic has been created. All documents associated with the new word are retrieved and compared with the document content in the created research topic. S10: If the document contents are the same, the retrieved editing and research topics will be directly returned; S11: If the document contents are different, the reference attachments in the topic will be combined with the new content according to the degree of difference and the existing topic documents. If there is new content, the editorial research content generated by the newly added text and summary will be added to the editorial research document. If the newly retrieved document content is completely different from the document content of the topic reference that has been created, it is necessary to update the topic document and regenerate the editorial research topic document.

2. The AI-based intelligent digital archive compilation and research method according to claim 1 is characterized by: The text keyword extraction in step S4 is specifically as follows: Extract keywords from the cleaned text data in the document library. The intelligent keyword extraction process is implemented by combining NLP (natural language processing) technology with LLM (large language model) technology; First, use the TF-IDF algorithm in natural language processing technology to perform preliminary keyword extraction on the text content. The TF-IDF keyword extraction algorithm calculates the weight values ​​of the two statistical features of word frequency and inverse document frequency in the text, and uses the words with larger weight values ​​as the keywords of the text. The calculation formula for word frequency is as follows: Where n ij Indicates that the term is in the text d j The number of times it appears in k n k，j Indicates that in the text j The sum of all occurrences of the term in . The calculation formula for inverse document frequency is as follows: Where |D| represents the total number of all texts, Indicates that it contains the term t i The number of texts; The calculation formula of TF-IDF is as follows: TF-IDF=TF i，i ×IDF i Then, LLM technology is used to further extract keywords.

3. The AI-based intelligent digital archive compilation and research method according to claim 1 is characterized by: The text summary extraction in step S4 is specifically as follows: Since the maximum number of tokens that a large language model can receive is limited, the text content must be sliced ​​when processing massive text data. Text summary extraction is divided into three steps: slicing, summary extraction, and integration; First, slice each text into appropriate sizes according to the maximum number of tokens that the large language model can accept; Then, the prompt engineering is used to construct prompts for each slice to guide LLM to extract summaries, such as generating a summary for the text and setting prompts according to user needs; Finally, the summaries are integrated together according to the order of the slices in the text, and then the integrated summaries are input into the LLM combined with the prompt engineering to generate the overall summary of the text.

4. The AI-based intelligent digital archive compilation and research method according to claim 1 is characterized by: The attachment contents in different formats in step S2 include pdf, word, txt, png, tif, etc.

5. The AI-based intelligent digital archive compilation and research method according to claim 1 is characterized by: The text data cleaning in step S3 specifically involves cleaning non-text elements, irrelevant characters, etc. in the text to ensure the availability and accuracy of the text content.

6. The AI-based intelligent digital archive compilation and research method according to claim 2 is characterized by: The use of LLM technology to further extract keywords is specifically as follows: the keyword candidate list obtained by the TF-IDF keyword extraction algorithm is input into a large pre-trained language model, and its powerful context-awareness ability is used to evaluate the correlation between these keyword pre-texts. This process goes deep into the specific position of the candidate keyword in the text context, the text environment of the context, and its role and influence in the entire text; through the self-attention mechanism, LLM can deeply understand and automatically parse the complex dependencies between words. This mechanism can not only extract high-frequency words, but also capture words that are not obvious in the statistical process of conventional methods but are more important in this article, thereby improving the accuracy of keyword extraction.

Images