Data analysis-based literature core content extraction method

Abstract

The invention discloses a literature core content extraction method based on data analysis, and relates to the technical field of information retrieval, the method comprises the following steps: introducing a core evaluation function to construct a model, and determining a core word bag library and a weight; traversing the word bag library to retrieve and mine the semantic relationship; an architecture construction model is determined and initialized; extracting, integrating and outputting a literature core system through a recognition word bag library; and carrying out visual display after checking compensation. The technical problems that in the prior art, extraction of the core content of the literature is not accurate, and the result reliability is low are solved, and the technical effects of improving the accuracy and reliability of the result and improving the searching and reading efficiency are achieved through accurate, comprehensive and structured extraction of the core content of the literature.

Description

Technical Field

[0001] The present application relates to the field of information retrieval technology, and in particular to a method for extracting core content of a document based on data analysis. Background Art

[0002] In the context of scientific and technological document processing, the demand for scientific and technological novelty search and data reading is growing, and the contradiction between the demand for efficient extraction of the core content of documents is becoming more prominent. Accurately extracting the core content of documents and improving the efficiency of scientific and technological novelty search and data reading have become crucial links in the current scientific and technological document processing. Traditional methods for extracting the core content of documents are often relatively simple and one-sided, relying only on keyword matching or manual screening, with insufficient semantic understanding of documents, lack of in-depth analysis and utilization of the documents as a whole, and difficulty in comprehensively and accurately extracting the core content of documents. There are unclear situations in the output structure, resulting in disorganized extraction results. There is also a lack of effective verification and compensation mechanism for the reliability of the extraction results, which cannot well meet the actual needs of scientific and technological novelty search and data reading.

[0003] At the current stage, relevant technologies have technical problems such as inaccurate extraction of core content of documents and low reliability of results. Summary of the invention

[0004] The present application provides a method for extracting the core content of documents based on data analysis, introduces a core evaluation function to build a model, determines the core word bag library and weights; traverses the word bag library to search and mine semantic relationships; determines the architecture to build a model and initializes it; identifies the word bag library to extract, integrate and output the core system of the document; and visualizes the document after verification and compensation. By accurately, comprehensively and structuredly extracting the core content of the document, the application achieves the technical effect of improving the accuracy and reliability of the results and enhancing the efficiency of searching and reading.

[0005] The present application provides a method for extracting core content of a document based on data analysis, comprising:

[0006] A core evaluation function is introduced, and a core evaluation model is constructed. The target document is scanned and the core degree is evaluated to determine a core word bag library, wherein the core word bag library is marked with a core weight; the core word bag library is traversed to perform big data retrieval and mine the semantic relationship of terms; the core output architecture is determined and a large language extraction model is constructed, and a self-attention mechanism is configured based on the core weight, and the large language extraction model is initialized with the semantic relationship of the terms as a constraint; the core word bag library is identified, and core content extraction and architecture distribution integration are performed based on the large language extraction model, and content hierarchical expansion integration based on multi-level granularity is performed to output a document core system; the document core system is confidence-checked and compensated, and the interface is visualized on a display terminal.

[0007] The present application proposes a method for extracting the core content of documents based on data analysis. First, a core evaluation function is introduced to construct a model to determine the core word bag library and its weight; the word bag library is traversed to search and mine semantic relationships; the architecture is determined to build a model and initialize it; the word bag library is identified to extract, integrate and output the core system of the document; and the document is visualized after verification and compensation. Through the precise, comprehensive and structured extraction of the core content of the document, the technical effect of improving the accuracy and reliability of the results and enhancing the efficiency of searching and reading is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to more clearly illustrate the technical solution of the embodiment of the present invention, the accompanying drawings of the embodiment of the present invention will be briefly introduced below. A flow chart is used in the present application to illustrate the operations performed by the system according to the embodiment of the present application. It should be understood that the preceding or following operations are not necessarily performed accurately in order. On the contrary, various steps can be processed in reverse order or simultaneously as needed. At the same time, other operations can also be added to these processes, or one or more operations can be removed from these processes.

[0009] Figure 1 A schematic diagram of a process for extracting core content of a document based on data analysis provided in an embodiment of the present application;

[0010] Figure 2 A schematic diagram of the process of determining a core word bag library for a method for extracting core content of a document based on data analysis provided in an embodiment of the present application. DETAILED DESCRIPTION

[0011] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below.

[0012] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0013] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict, and the terms "first\second" involved are merely to distinguish similar objects and do not represent a specific ordering of objects. The terms "including" and "having" and any variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products, or devices. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those generally understood by technicians in the technical field of this application. The terms used herein are for the purpose of describing the embodiments of the present application only.

[0014] The present application embodiment provides a method for extracting the core content of a document based on data analysis, such as Figure 1 As shown, the method includes:

[0015] Step S100, introduce the core evaluation function, and build a core evaluation model, scan and evaluate the core degree of the target document, determine the core word bag library, and the core word bag library is marked with a core weight. Specifically, in the document core content extraction scheme, first introduce the core evaluation function, which comprehensively considers factors such as the frequency of keyword occurrence, the importance of a specific position, and the degree of association with specific field knowledge. It can be designed using a mathematical formula or algorithm, such as based on a machine learning evaluation model or a variant of a word frequency-inverse document frequency algorithm, etc., to build a core evaluation model, select a suitable machine learning or deep learning model, collect a large number of document samples with annotated core content for training, adjust the parameters to make the model learn to accurately apply the core evaluation function, and use an independent test data set to evaluate the optimization model, input the target document into the model for scanning, analyze the content, structure and language features, apply the core evaluation function to evaluate the core degree of each entry in the document, consider the context information and the relationship with other entries, and finally select high-core degree entries according to the evaluation results to form a core word bag library, and mark the core weight for each entry to reflect its importance in the document, laying the foundation for subsequent semantic relationship mining, content extraction and integration.

[0016] In one possible implementation, Figure 2As shown, a core evaluation function is introduced, and a core evaluation model is constructed to scan the target document and evaluate its core degree, and a core word bag library is determined. The core word bag library is marked with core weights. Step S100 further includes step S110, which segments the target document based on a preset segmentation criterion to determine a text segment set. Specifically, the preset segmentation criterion can be set according to the structural characteristics of the document, such as by chapter. The purpose is to segment the target document into relatively independent and relatively complete text segments. Each text segment should not be too small to avoid low efficiency caused by excessive subsequent calculations, and at the same time, it can ensure that the text segment has sufficient representativeness. If the text segment is too small, on the one hand, it will increase the computational workload because more small segments need to be processed; on the other hand, the small segment may not fully reflect the overall theme and core content of the document, thus reducing the accuracy and reliability of the analysis.

[0017] Step S120, traverse the non-significant word library to perform stop filtering identification on the text segment set to determine an identified text segment set. The non-significant word library is a set of non-significant entries. Specifically, the non-significant word library is composed of entries without specific meanings such as "le", "shi", "de", etc. The frequency of these words in the text is often high, but they do not help in analyzing the core content of the document and may interfere with the analysis of meaningful words, reducing the bias of the analysis of meaningful words. By traversing the non-significant word library, perform stop filtering identification on each text segment in the text segment set, mark or remove the meaningless words, so as to obtain a processed identified text segment set, reduce unnecessary interference factors, and make the subsequent analysis more focused on valuable entries.

[0018] Step S130, combine the core evaluation function to evaluate the core degree of the entries in the identified text segment set to determine the entry core coefficient. Specifically, the core evaluation function plays a key role in the entire solution. It comprehensively considers multiple factors to evaluate the core degree of entries. Applying it to the identified text segment set, evaluate the core degree of each entry in it. The evaluation process may consider the frequency of the entry in the text segment, the importance of the specific position (such as the title, beginning, end, etc.) and the degree of association with the document theme, etc. Through evaluation, a core coefficient is determined for each entry, and the coefficient reflects the core degree of the entry in the document. For example, an entry that appears in multiple key positions and is highly relevant to the document theme will have a relatively high core coefficient.

[0019] Step S140, based on the core coefficient threshold, traverse the core coefficients of the terms to select core terms, and perform weight distribution based on the core coefficients of the terms, and integrate to generate the core word bag library. Specifically, a core coefficient threshold is set, and by traversing the core coefficients of the terms, those terms with core coefficients higher than the threshold are selected as core terms. Core terms are considered to be the parts that make important contributions to the core content of the document. Then, weight distribution is performed based on the core coefficients of the terms, that is, the higher the core coefficient of the terms, the greater the weight assigned to the terms in the core word bag library. The selected core terms and their corresponding weights are integrated to generate a core word bag library, which will serve as the basis for subsequent big data retrieval, semantic relationship mining, and core content extraction and integration.

[0020] In a possible implementation, in combination with the core evaluation function, the term core degree of the identified text segment set is evaluated to determine the term core coefficient, and step S130 further includes step S131, obtaining the core evaluation function: Where E is the term coreness, T(i,d) is the normalized term frequency of term i in text segment d, I(t) is the inverse text frequency of term i, and t d is the total number of terms in text segment d, is the number of times term i appears in text segment d, e is the total number of text segments, and e i is the total number of text segments containing term i. Specifically, in the document core content extraction scheme, the core evaluation function expression is Among them, E is the term coreness, T(i,d) is the normalized word frequency of term i in text segment d, and is given by The calculation results reflect the frequency of occurrence of a term in a specific text segment. I(t) is the inverse text frequency of term i. Calculate, where e is the total number of text segments, e i is the total number of text segments containing term i, reflecting the prevalence and particularity of the term in the entire document collection. In the scheme, the function comprehensively considers the frequency of occurrence of the term in a specific text segment and the prevalence and particularity of the term in the document collection, and evaluates the coreness of the term in each text segment after the target document is segmented, quantitatively determines the importance of the term in the document, and provides a basis for subsequent screening of core terms, determining the core word bag library, and identifying core weights, thereby playing a key role in big data retrieval, mining semantic relationships of terms, building large language extraction models, and extracting and integrating core content.

[0021] Step S200, traverse the core word bag library, perform big data search, and mine the semantic relationship of terms. Specifically, in the core content extraction scheme of the document, first traverse the core word bag library that has been determined and marked with core weights, and use the terms therein as the basis to search using big data resources including academic databases, etc., and use strategies such as exact matching, fuzzy matching, and associated retrieval to process a large number of document results related to the core terms obtained by the retrieval. When mining the semantic relationship of terms, the subject of the target document is constrained, the terms in the core word bag library are used as indexes, and other related terms are identified in the search results, and semantic relationships such as pre-association, post-association, and same-level association are determined, which provides important constraints and guidance for the subsequent construction of a large language extraction model and the extraction and integration of core content.

[0022] In a possible implementation, the core word bag library is traversed to perform big data retrieval and to mine semantic relationships of terms. Step S200 further includes step S210, in which the subject of the target document is constrained and the core word bag library is used as an index to perform big data retrieval and determine retrieval statement samples. Specifically, the subject of the target document provides a clear direction and scope for the entire scheme. Using the subject as a constraint can ensure that the results of the big data retrieval are closely related to the target document and avoid retrieving irrelevant information. Performing big data retrieval with the core word bag library as an index means using the terms in the core word bag library as search keywords or clues to more specifically find documents and information related to the core content of the target document. Through big data retrieval, retrieval statement samples are determined. The samples are representative sentences selected from a large number of search results and will serve as the basis for subsequent processing.

[0023] Step S220, traverse the search sentence sample, take the core word bag library as the scope, identify the first core term and perform relevance mining, and determine the first term relationship, wherein the relevance relationship at least includes a preceding relationship, a succeeding relationship and a same-level relationship. Specifically, the determined search sentence samples are analyzed and processed one by one, and natural language processing techniques such as lexical analysis and syntactic analysis are used to better understand the structure and meaning of the sentences. When traversing the search sentence samples, the core word bag library is used as the scope for operation, and only the content related to the entries in the core word bag library is focused on. If a sentence does not contain an entry in the core word bag library, then this sentence may not be closely related to the core content of the target document and can be temporarily ignored. In each search sentence sample, the core terms are identified. The core terms refer to the terms that are most closely related to the core content of the target document. The core terms are matched with the terms in the core word bag library, or determined according to certain algorithms and rules. The identified first core terms are mined for relevance. The purpose of relevance mining is to find other terms that have a certain relationship with the first core term. The relationship can be a pre-association, a post-association, or a peer association. Among them, the pre-association is to analyze which terms usually appear in the search sentence samples. Now it is before the first core term and has a certain logical relationship with the first core term. For example, in the sentence "Disease prevention requires strengthening health education", "disease prevention" is the first core term, and "health education" has a front-end association with it; the post-association is to find the terms that usually appear after the first core term in the search sentence sample, and analyze the relationship between them. For example, in the sentence "The development of artificial intelligence technology has promoted the transformation of the medical industry", "artificial intelligence technology" is the first core term, and "transformation of the medical industry" has a post-association with it; the same-level association is to identify terms that are semantically at the same level as the first core term or have similar functions. For example, in the sentence "Machine learning and deep learning are both important branches of artificial intelligence", "machine learning" and "deep learning" have a same-level association with "artificial intelligence". Through correlation mining, the relationship between the first core term and other related terms, that is, the first term relationship, is determined. The relationship will serve as an important basis for subsequent processing.

[0024] Step S230, traverse the core word bag library, and mine the Nth term relationship of the Nth core term. Specifically, traverse the core word bag library again, the purpose of the traversal is to process each term in the core word bag library one by one, and mine their term relationships. For the Nth core term in the core word bag library, repeat the previous relevance mining process to determine the relationship between it and other related terms, that is, the Nth term relationship, N can be any positive integer, representing different terms in the core word bag library. By performing such processing on each core term, the relationship of all terms in the core word bag library can be fully mined.

[0025] Step S240, integrate the first term relationship to the Nth term relationship, and determine the term semantic relationship. Specifically, the first term relationship, the second term relationship to the Nth term relationship determined previously are integrated. The integration process can use methods such as data fusion and relationship network construction to integrate the various term relationships together to form an overall relationship network. By integrating the various term relationships, the semantic relationship of the terms in the entire core word bag library is finally determined. The semantic relationship reflects the logical connection and semantic association between the various terms in the core word bag library, which provides an important basis for the subsequent extraction and integration of the core content of the document. By taking the subject of the target document as a constraint and the core word bag library as an index to perform big data retrieval, and then traversing the search sentence samples and the core word bag library, the term relationship of each core term is mined, and finally these relationships are integrated to determine the term semantic relationship, which provides strong support for the accurate extraction of the core content of the document.

[0026] Step S300, determine the core output architecture and build a large language extraction model, configure the self-attention mechanism based on the core weight, and initialize the large language extraction model with the semantic relationship of the term as a constraint. Specifically, in the document core content extraction solution, first determine the core output architecture, comprehensively consider user needs, document type characteristics and application scenarios, clarify the presentation form and structure of the document core content finally extracted, then build a large language extraction model, select a suitable language model architecture, perform data preparation and preprocessing, obtain a model with good performance through training and optimization, configure the self-attention mechanism based on the core weights reflecting the importance of terms in the core word bag library, so that the model pays more attention to high-weight terms when processing document content, and enhances the accuracy and pertinence of core content extraction. Finally, initialize the large language extraction model with the semantic relationship of terms determined by big data retrieval and relevance mining as a constraint, integrate the semantic relationship into the model parameters and structure, improve the model performance, make it better follow the semantic logic when extracting core content, and extract more coherent and meaningful content.

[0027] Step S400, identifying the core word bag library, extracting the core content and integrating the architecture distribution based on the large language extraction model, and performing content hierarchical expansion and integration based on multi-level granularity, and outputting the document core system. Specifically, in the document core content extraction scheme, firstly, the determined core word bag library is identified, which is obtained through careful screening and evaluation, and contains entries that have important contributions to the core content of the target document, and the target document is processed using the large language extraction model, and key information fragments are extracted based on the core word bag library and the self-attention mechanism, and distributed and integrated according to the determined core output architecture, while keeping the core output architecture framework system unchanged, the extracted core content is gradually refined and expanded, and a multi-level system is constructed, which is divided into different levels according to the importance and detail of the content, forming a complete document core system, and finally the document core system is output, which can be presented in a variety of forms to facilitate users to conduct scientific and technological novelty searches and data reading.

[0028] In a possible implementation, the core word bag library is identified, core content extraction and architecture distribution integration are performed based on the large language extraction model, and content classification expansion integration based on multi-level granularity is performed to output the document core system. Step S400 further includes step S410, based on the core output architecture, the core content is laid out and integrated to determine the document content system. Specifically, the core output architecture provides a clear framework and direction for the integration of core content. In this step, the extracted core content is laid out and integrated according to the requirements of the core output architecture. First, the characteristics and requirements of the core output architecture are deeply analyzed. For example, if the core output architecture is theme-oriented, then the scope and interrelationships of each theme need to be determined. If it is organized in a hierarchical structure, then the logical relationship between different levels needs to be clarified. The extracted core content is reasonably laid out according to the core output architecture, which involves allocating relevant core content to specific subject areas or levels. For example, if the core output architecture is in the form of a knowledge graph, then the core content needs to be laid out as nodes and the connection relationship between nodes needs to be determined. Through the layout and integration of the core content, a preliminary literature content system is formed. The system reflects the organizational form of the core content under a specific architecture, providing a basis for subsequent further refinement and grading.

[0029] Step S420, receiving the content granularity classification standard uploaded by the user. Specifically, when using the document core content extraction system, the user can upload the content granularity classification standard according to his or her own needs. The standard will guide the subsequent multi-level granularity regularization and inter-level association process. The user may have different requirements for the granularity of the document core content according to different application scenarios and purposes. For example, when conducting in-depth research, a more detailed content classification may be required; while when browsing quickly, only a more general level may be required. The system receives the content granularity classification standard uploaded by the user. The standard can be a specific level division requirement or some parameter settings, such as the number of granularity levels, the amount of information contained in each level, etc. The received content granularity classification standard is interpreted to understand the specific needs of the user, and the standard is applied to the subsequent processing steps to ensure that the final document core system meets the user's expectations.

[0030] Step S430, based on the content fineness classification standard, traverse the document content system, perform multi-level fineness regularization and inter-level association, and determine the document core system. Specifically, based on the determined document content system, start multi-level fineness regularization, traverse each part of the document content system, ensure that all content is properly processed, and perform multi-level fineness regularization on the document content according to the content fineness classification standard uploaded by the user, and divide and organize the content according to different fineness levels. For example, the core content can be divided into multiple levels such as high-level overview, intermediate explanation and detailed details. While performing multi-level fineness regularization, it is also necessary to establish associations between different levels to ensure that users can easily understand the relationship and logical connection between each level when viewing content at different fineness levels. For example, jumps and associations between levels can be achieved by setting links, references, etc. Through multi-level fineness regularization and inter-level associations, the document core system is finally determined. The system not only meets the user's requirements for content fineness, but also maintains the overall logic and coherence.

[0031] Step S440, establish a mutual mapping between the document core system and the target document. Specifically, establishing a mutual mapping between the document core system and the target document can help users better understand the relationship between the core content and the original document. When users view a certain part in the document core system, they can quickly locate the corresponding position in the original document for more in-depth reading and research. A variety of methods can be used to establish mutual mapping. For example, a unique identifier can be assigned to each content unit in the document core system, and the corresponding position can be marked in the target document, or an index table can be established to correspond the content in the document core system to the paragraphs, chapters, etc. in the target document. By establishing mutual mapping, users can switch freely between the document core system and the target document, which improves the convenience and efficiency of use, and also provides strong support for users to conduct further document analysis and research. Through the steps of layout integration based on the core output architecture, receiving the content grading standard uploaded by users, performing multi-level grading and inter-level association, and establishing mutual mapping, a document core system that meets user needs and has good organization and readability is finally determined.

[0032] Step S500, the document core system is verified and compensated with confidence, and the interface is visualized on the display terminal. Specifically, in the document core content extraction scheme, the confidence threshold is first set based on the feature completeness to determine the verification rule, and the confidence threshold can be positively correlated with the core weight. The document core system is verified based on this rule. If the confidence threshold is not met, a secondary extraction instruction and extraction condition are generated, and a secondary extraction is performed in combination with a large language extraction model to compensate for the shortcomings of the document core system. For the convenience of user viewing and use, the interface is visualized on the display terminal, and an intuitive and clear visualization interface is designed. The document core system is presented in various forms, and interactive functions such as search, link jump, annotation, etc. are provided. Focus on optimizing the user experience, ensure that the interface is simple, beautiful and easy to use, meet the needs and habits of different users, and provide personalized setting options.

[0033] In a possible implementation, the document core system is subjected to confidence verification and compensation, and the interface is visualized on the display terminal. Step S500 further includes step S510, setting a confidence threshold based on feature completeness and determining a confidence verification rule, wherein the confidence threshold is positively correlated with the core weight. Specifically, feature completeness is an important indicator for measuring the quality of the document core system, and takes into account the performance of the document core system in various aspects, including the coverage of core content, the accuracy of key information, and the logical coherence between content at various levels. By evaluating the feature completeness, the overall quality level of the document core system can be determined. The confidence threshold is set to establish a standard for judging the reliability of the document core system. The setting of the confidence threshold is based on the feature completeness, that is, a suitable confidence level is determined according to the feature completeness of the document core system. At the same time, the confidence threshold is positively correlated with the core weight. The more core the content, the higher its confidence threshold, because the core content is more important to the document and requires higher accuracy and reliability. For example, if an entry has a high core weight in the document, then when performing confidence verification on it, a higher confidence threshold requirement needs to be met.

[0034] Step S520, based on the confidence verification rules, the document core system is verified. Specifically, after the confidence verification rules are determined, the document core system needs to be verified. The verification process is to compare the document core system with the confidence verification rules to determine whether it meets the requirements of the confidence threshold. First, various aspects of the document core system are analyzed and evaluated in detail, including checking the accuracy, completeness, and logic of the core content. For example, for a core point of view, it is necessary to verify whether it is consistent with the original document, whether there are sufficient supporting arguments, and whether the logical coherence is maintained between different levels of content. According to the confidence verification rules, the characteristic integrity of the document core system is quantitatively evaluated, using certain evaluation indicators and methods, such as calculating the coverage ratio of the core content, the accuracy of key information, etc. Through quantitative evaluation, it can be determined whether the characteristic integrity of the document core system has reached the requirements of the confidence threshold.

[0035] Step S530, if the verification satisfies the confidence threshold of the mapping, the document core system is visualized on the display terminal. Specifically, if the document core system is verified to meet the confidence threshold of the mapping, it means that it has high quality and reliability, and the display terminal visualization can be performed. The display terminal visualization is the process of presenting the document core system to the user in an intuitive and clear manner. Through visualization, the user can more conveniently understand and use the document core system. When performing visualization, a suitable visualization method can be selected according to the characteristics of the document core system and user needs. For example, the relationship between the core content and the content of each level can be displayed in the form of charts, lists, texts, etc. At the same time, visual elements such as color and font size can be used to highlight the core content and important information. The visualization process can also include providing interactive functions, such as search, link jump, annotation, etc. The interactive function can make it easier for users to browse and explore the document core system and improve the user experience. By setting the confidence threshold based on the feature completeness to determine the confidence verification rule, the document core system is verified, and the display terminal visualization is performed when the confidence threshold is met, the quality and reliability of the document core system can be ensured, and a better user experience can be provided for the user.

[0036] In a possible implementation, if the confidence threshold of the mapping is satisfied after verification, the document core system is visualized on a display terminal, and step S530 further includes step S531, if the confidence threshold of the mapping is not satisfied, a secondary extraction instruction is generated, and an extraction condition is generated, and a secondary extraction is performed in combination with the large language extraction model to compensate the document core system. Specifically, in the document core content extraction scheme, if the document core system does not satisfy the confidence threshold of the mapping after verification, the system will automatically generate a secondary extraction instruction, and generate extraction conditions according to the specific insufficient situation, and the large language extraction model will analyze and process the target document again according to the extraction conditions, and the results of the secondary extraction will be integrated into the existing document core system for compensation after deduplication and fusion, and the document core system needs to be verified again after compensation. If the confidence threshold is still not satisfied, it may be necessary to further adjust the extraction conditions, perform multiple extractions and compensations, so as to improve the quality and reliability of the document core system.

[0037] In a possible implementation, the document core system is subjected to confidence verification and compensation, and the interface is visualized on the display terminal. Step S500 further includes step S540, which obtains periodic verification compensation features and reading feedback information based on a preset period. Specifically, the preset period is a time interval set to ensure continuous optimization and improvement of the document core content extraction scheme. During the period, the system will actively obtain periodic verification compensation features and reading feedback information. The features are generated in the process of confidence verification and compensation of the document core system, for example, which parts of the content often need secondary extraction and compensation, which types of errors or deficiencies occur more frequently, etc. By analyzing the verification compensation features, the problems and weak links in the current document core content extraction scheme can be understood. The reading feedback information comes from the user's use and evaluation of the document core system. In the process of viewing and using the document core system, the user may put forward some opinions and suggestions, such as the accuracy, completeness, readability and other aspects of the content. The feedback information is very valuable for improving the document core content extraction scheme.

[0038] Step S550: updating and learning the large language extraction model based on the verification compensation feature and the reading feedback information. Specifically, the obtained verification compensation features and reading feedback information are deeply analyzed to determine which aspects need to be improved, as well as the focus and direction of the improvement. For example, if the verification compensation features show that the literature in a specific field often has the problem of inaccurate extraction, then more in-depth research and optimization can be carried out in this field. According to the analysis results, the large language extraction model is updated and learned. A variety of methods can be used to update the model, such as adjusting the parameters of the model, adding new training data, improving the algorithm of the model, etc. If the reading feedback information shows that the user is not satisfied with the extraction results of certain types of content, the performance of the model in these aspects can be improved by adding relevant training data or adjusting the structure of the model. By periodically obtaining verification compensation features and reading feedback information and updating and learning the large language extraction model, continuous optimization and improvement of the core content extraction solution of the document can be achieved. Over time, the model will become more and more accurate and reliable, and can better meet the needs of users. By obtaining verification compensation features and reading feedback information based on a preset period, and updating and learning the large language extraction model based on this, the quality and performance of the core content extraction solution of the document can be continuously improved.

[0039] The embodiment of the present application adopts the introduction of a core evaluation function to construct a model, determine the core word bag library and weights; traverse the word bag library to search and mine semantic relationships; determine the architecture to build a model and initialize it; identify the word bag library to extract, integrate and output the core system of the document; and visualize the display after verification and compensation. Through the precise, comprehensive and structured extraction of the core content of the document, the technical effect of improving the accuracy and reliability of the results and enhancing the efficiency of searching and reading is achieved.

[0040] The above specific implementation manner does not constitute a limitation to the protection scope of the present application. It should be understood by those skilled in the art that various modifications, combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the protection scope of the present application. In some cases, the actions or steps recorded in the present application can be performed in an order different from that in the embodiment and can still achieve the desired results. In addition, the process depicted in the accompanying drawings does not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

Claims

1. A method for extracting core content of documents based on data analysis, characterized in that: The method comprises: Introducing a core evaluation function and building a core evaluation model, scanning and evaluating the coreness of target documents, and determining a core word bag library, wherein the core word bag library is marked with a core weight; Traversing the core word bag library, performing big data retrieval, and mining semantic relationships of terms; Determine the core output architecture and construct a large language extraction model, configure a self-attention mechanism based on the core weights, and initialize the large language extraction model with the semantic relationship of the terms as a constraint; Identify the core word bag library, extract the core content and integrate the architecture distribution based on the large language extraction model, and perform content classification expansion integration based on multi-level granularity to output the core system of the document; The document core system is subjected to confidence verification and compensation, and the interface is visualized on a display terminal.

2. A method for extracting core content of a document based on data analysis as claimed in claim 1, characterized in that: Determining the core word bag library includes: Based on a preset segmentation standard, the target document is segmented to determine a text segment set; Traversing the non-meaning word library, deactivating and filtering the text segment set, and determining the identified text segment set, wherein the non-meaning word library is a collection of meaningless entries; In combination with the core evaluation function, the term coreness evaluation is performed on the identified text segment set to determine the term core coefficient; Based on the core coefficient threshold, the core terms are traversed through the term core coefficients to filter the core terms, and weight distribution is performed based on the term core coefficients to integrate and generate the core word bag library.

3. A method for extracting core content of a document based on data analysis as claimed in claim 2, characterized in that: The method further comprises: Get the core evaluation function: Where E is the term coreness, T(i,d) is the normalized term frequency of term i in text segment d, I(t) is the inverse text frequency of term i, and t d is the total number of terms in text segment d, t i d is the number of times term i appears in text segment d, e is the total number of text segments, and e i is the total number of text segments containing term i.

4. A method for extracting core content of a document based on data analysis as claimed in claim 1, characterized in that: The mining of semantic relations of terms includes: Taking the subject of the target document as a constraint and the core word bag library as an index, a big data search is performed to determine a search statement sample; Traversing the search sentence sample, taking the core word bag library as the scope, identifying the first core term and performing relevance mining to determine the first term relationship, wherein the relevance relationship at least includes a preceding relationship, a succeeding relationship and a same-level relationship; Traversing the core word bag library to mine the Nth term relationship of the Nth core term; The first term relationship up to the Nth term relationship are integrated to determine the term semantic relationship.

5. A method for extracting core content of a document based on data analysis as claimed in claim 1, characterized in that: The content classification expansion integration based on multi-level granularity includes: Based on the core output framework, the core content is laid out and integrated to determine the document content system; Receive the content grading standards uploaded by the user; Based on the content granularity classification standard, the document content system is traversed, multi-level granularity regularization and inter-level association are performed to determine the document core system; Establish a mutual mapping between the document core system and the target document.

6. A method for extracting core content of a document based on data analysis as claimed in claim 1, characterized in that: Conduct confidence verification and compensation on the document core system, including: Based on the feature completeness, the confidence threshold is set and the confidence verification rule is determined, where the confidence threshold is positively correlated with the core weight; Based on the confidence verification rules, verifying the document core system; If the verification satisfies the confidence threshold of the mapping, the document core system is visualized on a display terminal.

7. A method for extracting core content of a document based on data analysis as claimed in claim 6, characterized in that: If the confidence threshold of the mapping is not met, a secondary extraction instruction is generated, and extraction conditions are generated, and secondary extraction is performed in combination with the large language extraction model to compensate for the core system of the document.

8. A method for extracting core content of a document based on data analysis as claimed in claim 1, characterized in that: The method further comprises: Based on a preset period, obtain periodic calibration compensation characteristics and reading feedback information; Based on the verification compensation feature and the reading feedback information, the large language extraction model is updated and learned.

Images