Source document evaluation processing method and device for retrieval generation
By conducting multi-dimensional evaluation and weighted processing of Word documents, the problem of inconsistent Word document quality in enterprise AI question-answering systems was solved. Quantitative evaluation and optimization suggestions on structure, content, and format were provided, improving the accuracy of AI question-answering systems and the efficiency of document optimization.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING QDING INTERCONNECTION TECHNOLOGY CO LTD
- Filing Date
- 2025-12-24
- Publication Date
- 2026-05-05
AI Technical Summary
When enterprises use AI question-answering systems based on RAG technology, they encounter problems such as poor knowledge retrieval performance, a lack of effective document diagnosis and optimization tools, and an inability to assess the AI-friendly nature of the structure, content, and format of Word documents, resulting in low AI answer accuracy and unclear optimization directions.
This paper provides a method for evaluating and processing source documents generated through retrieval. By parsing Word documents, it obtains text, structure, and format information, performs multi-dimensional evaluation, including evaluation of structural integrity, content clarity, context independence, and format standardization, and performs weighted processing to generate comprehensive evaluation results and optimization suggestions.
It enables intelligent parsing and quantitative evaluation of Word documents, improving document quality, providing specific optimization suggestions, and enhancing the knowledge retrieval effect and document optimization efficiency of the AI question-answering system.
Smart Images
Figure CN121981107A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to a method and apparatus for evaluating and processing source documents generated through retrieval. Background Technology
[0002] With the widespread adoption of Large Language Models (LLM), enterprise-level intelligent question answering and knowledge assistants based on Retrieval Augmentation (RAG) technology have become core application scenarios. RAG technology enhances the accuracy and reliability of LLM responses by retrieving relevant information from the enterprise's internal knowledge base. Word documents, as one of the most common and mainstream knowledge carriers in enterprises, form the core foundation of the RAG knowledge base; therefore, the quality of Word documents directly impacts the performance of the RAG system.
[0003] Currently, when enterprises use AI question-answering systems based on RAG technology, they encounter problems such as poor knowledge retrieval results and a lack of effective document diagnosis and optimization tools due to the inconsistent quality of Word documents in the underlying knowledge base. Summary of the Invention
[0004] The embodiments of this application aim to at least partially address one of the technical problems in the related art. To this end, embodiments of this application propose a method and apparatus for evaluating and processing generated source documents.
[0005] The embodiments of this application provide a method for evaluating and processing source documents generated through retrieval. This method includes: parsing the source document to obtain its text information, structured information, and format information; performing multi-dimensional evaluation processing based on the text information, structured information, and format information to obtain at least two of the following: a structural integrity evaluation result, a content clarity evaluation result, a context independence evaluation result, and a format conformity evaluation result; and weighting at least two of the structural integrity evaluation result, content clarity evaluation result, context independence evaluation result, and format conformity evaluation result to obtain a comprehensive evaluation result for the source document.
[0006] In some implementations, a multi-dimensional evaluation process is performed based on at least one of textual information, structured information, and formatting information to obtain a structural integrity evaluation result, including: determining the title style of the source document based at least on structured information to obtain a title standardization evaluation result; determining the list usage in the source document based at least on structured information to obtain a list usage evaluation result; determining the paragraph length in the source document based at least on textual information to obtain a paragraph length evaluation result; and obtaining a structural integrity evaluation result based on the title standardization evaluation result, the list usage evaluation result, and the paragraph length evaluation result.
[0007] In some implementations, a multi-dimensional evaluation process is performed based on at least one of textual information, structured information, and formatted information to obtain a content clarity evaluation result, including: at least based on textual information, performing natural speech processing to identify the frequency of use of ambiguous words to obtain a language clarity evaluation result; at least based on textual information, performing technical terminology identification to obtain a technical terminology evaluation result; at least based on textual information, performing misspelling and grammar identification to obtain misspelling and grammar evaluation results; and based on the language clarity evaluation result, the technical terminology evaluation result, and the misspelling and grammar evaluation result, obtaining a content clarity evaluation result.
[0008] In some implementations, a multi-dimensional evaluation process is performed based on at least one of textual information, structured information, and formatted information to obtain a context independence evaluation result, including: identifying ambiguous pronouns based at least on textual information to obtain a referential clarity evaluation result; determining the theme of the content corresponding to the title based at least on textual information and structured information to obtain an information atomization evaluation result; and obtaining a context independence evaluation result based on the referential clarity evaluation result and the information atomization evaluation result.
[0009] In some implementations, a multi-dimensional evaluation process is performed based on at least one of text information, structured information, and format information to obtain a format standardization evaluation result, including: determining the ratio between text information and non-text information based at least on text information and format information to obtain a text extractability evaluation result; determining whether an image has a corresponding text description based at least on text information and format information to obtain an image-text separation evaluation result; and obtaining a format standardization evaluation result based on the text extractability evaluation result and the image-text separation evaluation result.
[0010] In some implementations, the method further includes: determining a first content to be optimized in the source document based on the comprehensive evaluation results, and generating document optimization suggestions for the first content to be optimized; generating a document evaluation report based on the comprehensive evaluation results and the document optimization suggestions, wherein the document evaluation report includes the comprehensive evaluation results and document rating information, a multi-dimensional radar chart of the source document, document optimization suggestions, and location information of the first content to be optimized in the source document.
[0011] In some implementations, the method further includes: determining a second content to be optimized in the source document based on the comprehensive evaluation results; outputting optimization instruction information based on the second content to be optimized; and optimizing the source document in response to receiving an optimization instruction for the optimization instruction information to generate an optimized document.
[0012] In some implementations, the document evaluation report includes a retrieval generation simulator; the method further includes: in response to receiving a question input by a user, retrieving a first answer from the source document based on the question using the retrieval generation simulator, retrieving a second answer from the optimized document based on the question using the retrieval generation simulator, and generating answer comparison information based on the first and second answers.
[0013] In some implementations, the method further includes integrating the source document evaluation method as a plugin into the document editor to facilitate the evaluation of the source document during the writing of the source document through the document editor.
[0014] The embodiments of this application provide an evaluation processing apparatus for retrieved and generated source documents. The apparatus includes: a parsing module for parsing the source document to obtain text information, structured information, and format information; an evaluation module for performing multi-dimensional evaluation processing based on the text information, structured information, and format information to obtain at least two of the following: a structural integrity evaluation result, a content clarity evaluation result, a context independence evaluation result, and a format conformity evaluation result; and a processing module for weighting at least two of the following: the structural integrity evaluation result, the content clarity evaluation result, the context independence evaluation result, and the format conformity evaluation result, to obtain a comprehensive evaluation result for the source document.
[0015] The embodiments of this application provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method of any of the above embodiments.
[0016] Embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method of any of the above embodiments. Attached Figure Description
[0017] Figure 1 A flowchart illustrating the source document evaluation processing method for retrieving generated documents, provided as an embodiment of this application; Figure 2 A schematic diagram of Word document evaluation processing in the source document multi-dimensional evaluation system provided for the embodiments of this application; Figure 3 A schematic diagram of an apparatus for evaluating and processing source documents retrieved according to an embodiment of this application; Figure 4 A block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation
[0018] The embodiments of the technical solution of this application will now be described in detail with reference to the accompanying drawings. These embodiments are only used to more clearly illustrate the technical solution of this application and are therefore merely examples, and should not be used to limit the scope of protection of this application.
[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.
[0020] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.
[0021] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.
[0022] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.
[0023] In the description of the embodiments of this application, the term "multiple" refers to two or more (including two), similarly, "multiple sets" refers to two or more (including two sets), and "multiple pieces" refers to two or more (including two pieces).
[0024] In the description of the embodiments of this application, unless otherwise expressly specified and limited, technical terms such as "installation," "connection," "joining," and "fixing" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. For those skilled in the art, the specific meaning of the above terms in the embodiments of this application can be understood according to the specific circumstances.
[0025] With the widespread adoption of Large Language Models (LLM), enterprise-level intelligent question answering and knowledge assistants based on Retrieval Augmentation (RAG) technology have become core application scenarios. RAG technology enhances the accuracy and reliability of LLM responses by retrieving relevant information from the enterprise's internal knowledge base. Word documents, as one of the most common and mainstream knowledge carriers in enterprises, form the core foundation of the RAG knowledge base; therefore, the quality of Word documents directly impacts the performance of the RAG system.
[0026] Currently, when enterprises use AI question-answering systems based on RAG technology, they encounter problems such as poor knowledge retrieval results and a lack of effective document diagnosis and optimization tools due to the inconsistent quality of Word documents in the underlying knowledge base.
[0027] Some methods identify typos and basic grammatical errors in documents using traditional document spelling and grammar checking tools (such as the built-in review function in Microsoft Word, and other third-party plugins), but their technical solutions have fundamental limitations: 1. Lack of structured assessment from an "AI perspective": Traditional tools have very limited inspection dimensions and cannot assess whether a document's structure is AI-friendly. For example, it will not judge a document's "structural integrity" as poor simply because a user uses "bold and centered" plain text to disguise a title. Nor can it determine whether paragraphs are too long, hindering AI's information chunking.
[0028] 2. Inability to assess the “clarity” and “context independence” of content: Existing technologies cannot determine from a semantic level whether a sentence is ambiguous or whether it uses too many vague pronouns such as “it” or “the former,” which are key factors that cause AI to misunderstand in RAG retrieval.
[0029] 3. "Inspection" rather than "scoring," and no optimization suggestions: Traditional tools can only passively identify errors, but cannot provide a quantitative, comprehensive score for the overall "AI friendliness" of a document from multiple dimensions. More importantly, it cannot provide users with specific, actionable optimization suggestions aimed at improving RAG performance based on the evaluation results.
[0030] Specifically, enterprises face the following technical challenges when using AI question-answering systems based on RAG technology: 1. Poor knowledge retrieval performance and low AI response accuracy: Many Word documents are written with human reading habits in mind, and generally suffer from problems such as disorganized structure, lack of standard heading levels, excessively long paragraphs, and unclear references. This makes it difficult for AI to accurately locate and understand information chunks when performing semantic retrieval, ultimately generating incorrect or irrelevant answers.
[0031] 2. The "black box" nature of knowledge base quality assessment: Knowledge base administrators cannot quantitatively assess how "AI-friendly" a Word document is. When the RAG system performs poorly, they are unsure whether the problem lies with the model itself or the quality of the source document, lacking effective diagnostic tools.
[0032] 3. Unclear document optimization direction and low return on investment: Even if administrators know that the document quality is problematic, they are unclear on which specific aspects (structure, content, or format) to optimize in order to most effectively improve the AI's search and response performance. Blindly making manual modifications is time-consuming, labor-intensive, and inefficient.
[0033] Therefore, this application provides a method for evaluating and processing source documents generated through retrieval, which can intelligently parse Word documents and quantitatively evaluate them from multiple dimensions such as structure, content, and format, so as to improve document quality.
[0034] In the technical solution of this application embodiment, the source document is first parsed to obtain its text information, structured information, and format information. Then, based on the text information, structured information, and format information, a multi-dimensional evaluation process is performed to obtain at least two of the following: structural integrity evaluation result, content clarity evaluation result, context independence evaluation result, and format standardization evaluation result. Finally, at least two of the following evaluation results are weighted to obtain a comprehensive evaluation result of the source document. This method can intelligently parse Word documents and quantitatively evaluate them from multiple dimensions, including structure, content, and format, to improve document quality.
[0035] Figure 1 This is a flowchart illustrating the method for evaluating and processing source documents generated during retrieval, as provided in an embodiment of this application.
[0036] like Figure 1 As shown, the source document retrieval evaluation processing method 100 provided in this application includes, for example, steps S110-S130.
[0037] Step S110: Parse the source document to obtain its text information, structured information, and formatting information.
[0038] For example, the source document can be a document of the underlying knowledge base used for retrieving question-answering systems (such as AI question-answering systems based on RAG technology). The source document format can be, for example, a Word document (.docx). The Word document related to the knowledge base can be deeply parsed to obtain the text information, structure information and format information of the Word document. The text information can include, for example, all plain text content in the document. The structure information can include, for example, the table structure (such as structured tables and headers), heading levels (such as which text is correctly marked as "Heading 1", "Heading 2" and other standard styles), and list styles (such as ordered lists and unordered lists) in the document obtained by parsing the underlying XML structure (a document structure, such as OOXML) of the document. The format information can include, for example, other formatting such as bold, italics, and highlighting in the document.
[0039] Step S120: Based on text information, structured information, and format information, perform multi-dimensional evaluation processing to obtain at least two of the following: source document structural integrity evaluation result, content clarity evaluation result, context independence evaluation result, and format standardization evaluation result.
[0040] For example, multi-dimensional parallel evaluation and scoring can be performed based on the text information, structured information, and formatting information obtained from parsing the source document, resulting in evaluation results in multiple dimensions. For instance, at least two of the following can be obtained: source document structural integrity evaluation result, content clarity evaluation result, context independence evaluation result, and formatting standardization evaluation result. Specifically, the source document structural integrity evaluation result can be obtained based on aspects such as the standardization of headings, the use of lists, and paragraph length in the Word document; the content clarity evaluation result can be obtained based on aspects such as the clarity of language descriptions, the description of professional terms, and the absence of typos and grammar in the Word document; the context independence evaluation result can be obtained based on aspects such as the use of pronouns and information atomization in the Word document; and the formatting standardization evaluation result can be obtained based on aspects such as the text ratio (text extractability rate) and the separation of text and images in the Word document.
[0041] Step S130: Weight at least two of the structural integrity assessment results, content clarity assessment results, context independence assessment results, and format standardization assessment results to obtain the comprehensive assessment result of the source document.
[0042] For example, the evaluation results of each dimension can be weighted based on the degree of influence on the evaluation results of the source document. For instance, the structural integrity evaluation result can be preset with a weight of 30%. The evaluation results can be weighted based on the preset weight of each dimension. For example, at least two of the structural integrity evaluation result, content clarity evaluation result, context independence evaluation result, and format standardization evaluation result can be weighted to obtain the comprehensive evaluation result of the source document. The comprehensive evaluation result can be the final score obtained by weighting the multi-dimensional evaluation results. For instance, the scores of each dimension can be weighted and averaged according to the preset weights to obtain a comprehensive score of 0-100 as the comprehensive evaluation result. This score can represent "AI friendliness".
[0043] In the technical solution of this application embodiment, the source document is first parsed to obtain its text information, structured information, and format information. Then, based on the text information, structured information, and format information, a multi-dimensional evaluation process is performed to obtain at least two of the following: structural integrity evaluation result, content clarity evaluation result, context independence evaluation result, and format standardization evaluation result. Finally, at least two of the following evaluation results are weighted to obtain a comprehensive evaluation result of the source document. This method can intelligently parse Word documents and quantitatively evaluate them from multiple dimensions, including structure, content, and format, to improve document quality.
[0044] In one example, the source document evaluation processing method for retrieval generated in this application is based on a multi-dimensional source document evaluation system. This system can perform in-depth analysis of user-uploaded Word documents, quantitatively evaluate the Word documents from multiple dimensions such as structure, content, and format, and automatically generate optimization suggestions. It simulates the "reading and understanding" process of an AI model and intelligently scores the source documents to systematically improve the quality of the knowledge base. The following is combined with... Figure 2 Detailed description.
[0045] Figure 2 A schematic diagram of Word document evaluation processing for the source document multi-dimensional evaluation system provided in this application embodiment.
[0046] like Figure 2 As shown, the process of the source document multi-dimensional evaluation system for evaluating Word documents includes the following steps: S201, the user uploads a Word document (.docx), and the document deep analysis module extracts the text content and parses the structure and format information (heading level / list / table).
[0047] S202 uses an AI-powered friendliness scoring engine to assess structural integrity, content clarity, context independence, and format conformity.
[0048] S203, a weighted comprehensive score is calculated, and at the same time, targeted optimization suggestions are generated through the intelligent optimization suggestion generation module.
[0049] S204 generates visual reports on the front end, displaying analysis results in the form of radar charts, scorecards, and question highlights. Users can view the reports and optimize the documents.
[0050] The following describes in detail the process of retrieving and evaluating the source documents and generating an evaluation report.
[0051] First, after a user uploads a Word document (source document) to the system, the system performs deep analysis, extracting text information, structured information, and formatting information in parallel. For example: Text content (text information): Extracts all plain text from the document.
[0052] Structure and Formatting Information: For key steps in Word document processing, the system parses the document's underlying XML structure (such as OOXML) to obtain metadata (structured and formatting information), including: heading levels, such as identifying which text is correctly marked as "Heading 1", "Heading 2", etc.; list styles, such as identifying ordered and unordered lists; table structures, such as parsing structured tables and identifying table headers; and other formatting, such as bold, italics, and highlighting.
[0053] Then, based on the AI friendliness scoring engine, the parsed information is evaluated and scored in parallel from multiple dimensions to obtain the source document's structural integrity evaluation results, content clarity evaluation results, context independence evaluation results, and format standardization evaluation results. The following is a detailed explanation of each evaluation dimension.
[0054] For example, a multi-dimensional evaluation process is performed based on at least one of textual information, structured information, and formatting information to obtain a structural integrity evaluation result. For instance, firstly, based at least on structured information, the title style of the source document is determined to obtain a title standardization evaluation result; then, based at least on structured information, the list usage in the source document is determined to obtain a list usage evaluation result; next, based at least on textual information, the paragraph length in the source document is determined to obtain a paragraph length evaluation result; finally, based on the title standardization evaluation result, the list usage evaluation result, and the paragraph length evaluation result, a structural integrity evaluation result is obtained.
[0055] Specifically, based on the heading hierarchy in the structured information, we can check whether the document contains headings and whether the H1 / H2 / H3 hierarchy (text heading hierarchy) is continuous and reasonable. For headings that do not use standard heading styles, such as those disguised by simply "bold and enlarge," points are deducted to obtain the heading standardization assessment result. Based on the list styles in the structured information, we can check whether there are cases in the document where lists should be used but are not (e.g., using manual numbering "1.", "2."), to obtain the list usage assessment result. Based on the text information, we can calculate the average and maximum paragraph lengths in the document and deduct points for excessively long paragraphs to obtain the paragraph length assessment result. Then, based on the scores of the heading standardization assessment result, the list usage assessment result, and the paragraph length assessment result, we obtain the structural integrity assessment result. The structural integrity result can be, for example, the assessment score for this dimension.
[0056] In the technical solution of this application embodiment, the title style of the source document is first determined based on at least structured information to obtain the title standardization evaluation result. Then, the list usage in the source document is determined based on at least structured information to obtain the list usage evaluation result. Next, the paragraph length in the source document is determined based on at least text information to obtain the paragraph length evaluation result. Finally, the structural integrity evaluation result is obtained based on the title standardization evaluation result, the list usage evaluation result, and the paragraph length evaluation result. By deeply analyzing the underlying structured information of the Word document as the key basis for evaluating "structural integrity", it helps AI to accurately understand the document hierarchy.
[0057] For example, a multi-dimensional evaluation process is performed based on at least one of textual information, structured information, and formatted information to obtain a content clarity evaluation result. For instance, firstly, at least based on textual information, natural language processing is used to identify the frequency of use of ambiguous words to obtain a language clarity evaluation result; then, at least based on textual information, technical terminology is identified to obtain a technical terminology evaluation result; next, at least based on textual information, misspellings and grammar are identified to obtain misspellings and grammar evaluation results; finally, based on the language clarity evaluation result, the technical terminology evaluation result, and the misspellings and grammar evaluation result, a content clarity evaluation result is obtained.
[0058] Specifically, based on textual information, an NLP (Natural Language Processing) model can be used to detect and statistically analyze the frequency of use of ambiguous words (such as "maybe" or "probably") in the document text to obtain a language clarity assessment result; based on textual information, technical terms or abbreviations that are not explained in context can be identified to obtain a technical terminology assessment result; based on textual information, a mature grammar checking model can be invoked to identify and score typos and grammar to obtain typo and grammar assessment results; then, based on the language clarity assessment results, technical terminology assessment results, and typo and grammar assessment results, a content clarity assessment result can be obtained, and the structural integrity result can be, for example, the score for this dimension.
[0059] In the technical solution of this application embodiment, firstly, at least based on text information, natural speech processing is performed to identify the frequency of use of ambiguous words to obtain a language clarity assessment result. Then, at least based on text information, technical terminology is identified to obtain a technical terminology assessment result. Next, at least based on text information, misspellings and grammar are identified to obtain misspellings and grammar assessment results. Finally, based on the language clarity assessment result, technical terminology assessment result, and misspellings and grammar assessment result, a content clarity assessment result is obtained. This quantitative assessment of the source document is performed from the content parsing dimension of the document, improving the accuracy of content assessment.
[0060] For example, a multi-dimensional evaluation process is performed based on at least one of textual information, structured information, and formatted information to obtain a context independence evaluation result. For instance, firstly, based at least on textual information, vague pronouns are identified to obtain a referential clarity evaluation result; then, based at least on textual information and structured information, the topic of the content corresponding to the title is determined to obtain an information atomization evaluation result; finally, based on the referential clarity evaluation result and the information atomization evaluation result, a context independence evaluation result is obtained.
[0061] Specifically, based on textual information, the number of vague pronouns such as "it," "this," and "the former" in the document, as well as instances with unclear referents, can be identified and counted to obtain the clarity of reference assessment results. Based on the heading hierarchy in the textual and structured information, it is possible to identify and assess whether the content under each heading focuses on an independent and complete topic to obtain the information atomization assessment results. Then, based on the clarity of reference assessment results and the information atomization assessment results, the context independence assessment results can be obtained, for example, as a score for this dimension.
[0062] In the technical solution of this application embodiment, firstly, based at least on textual information, vague pronouns are identified to obtain a referential clarity assessment result. Then, based at least on textual information and structured information, the topic of the content corresponding to the title is determined to obtain an information atomization assessment result. Finally, based on the referential clarity assessment result and the information atomization assessment result, a context independence assessment result is obtained. This surpasses pure text analysis and quantitatively assesses Word documents from multiple dimensions such as structure and content, thereby improving the accuracy of the assessment.
[0063] For example, a multi-dimensional evaluation process is performed based on at least one of text information, structured information, and format information to obtain a format standardization evaluation result. For instance, firstly, based on at least text information and format information, the ratio between text information and non-text information is determined to obtain a text extractability evaluation result; then, based on at least text information and format information, it is determined whether the image has a corresponding text description to obtain an image-text separation evaluation result; finally, based on the text extractability evaluation result and the image-text separation evaluation result, a format standardization evaluation result is obtained.
[0064] Specifically, based on text and format information, the proportion of space occupied by images and embedded objects (non-text information) in the document can be calculated. If the proportion of text (text information) is too low, a deduction will be made from the evaluation result of this dimension. Based on text and format information, it can be checked and confirmed whether the image (e.g., below) has a corresponding text description (i.e., caption) to obtain the image-text separation evaluation result. Then, based on the text extractability evaluation result and the image-text separation evaluation result, the format standardization evaluation result can be obtained. The format standardization evaluation result can be, for example, the evaluation score for this dimension.
[0065] In the technical solution of this application embodiment, firstly, based on at least text information and format information, the ratio information between text information and non-text information is determined to obtain the text extractability evaluation result. Then, based on at least text information and format information, it is determined whether the image has a corresponding text description to obtain the image-text separation evaluation result. Finally, based on the text extractability evaluation result and the image-text separation evaluation result, the format standardization evaluation result is obtained. The Word document is quantitatively evaluated from multiple dimensions such as structure, content, and format, which improves the accuracy and comprehensiveness of the evaluation.
[0066] Then, at least two of the structural integrity assessment results, content clarity assessment results, context independence assessment results, and format conformity assessment results are weighted to obtain the comprehensive assessment result of the source document. For example, the system calculates a weighted average of the scores of each dimension according to preset weights, where the weight of the structural integrity assessment result is, for example, 30%, the weight of the content clarity assessment result is, for example, 25%, the weight of the context independence assessment result is, for example, 20%, and the weight of the format conformity assessment result is, for example, 15%. After weighting, a comprehensive "AI friendliness" score of 0-100 is obtained, which is the comprehensive assessment result.
[0067] For example, optimization suggestions and document evaluation reports can also be generated based on the comprehensive evaluation results and the evaluation deductions for each dimension. For instance, firstly, based on the comprehensive evaluation results, the first content to be optimized in the source document is identified, and document optimization suggestions are generated for the first content to be optimized; then, based on the comprehensive evaluation results and document optimization suggestions, a document evaluation report is generated, wherein the document evaluation report includes the comprehensive evaluation results and document rating information, a multi-dimensional radar chart of the source document, document optimization suggestions, and location information of the first content to be optimized in the source document.
[0068] Specifically, the system's optimization suggestion generation module can automatically generate a clear and actionable optimization suggestion for each specific dimension's deductions during the evaluation process. For example: Regarding the deducted points (first item to be optimized): "'2.1 Market Status' uses bold plain text instead of the 'Heading 2' style." The generated suggestion (document optimization suggestion) is: "Please set the text '2.1 Market Status' to the standard 'Heading 2' style to help AI understand the document structure." Then, based on the comprehensive evaluation results and document optimization suggestions, a visual report (document evaluation report) can be generated and presented to the user. The report can include an overall score (comprehensive evaluation result), document rating information (such as "Excellent" or "Needs Improvement"), a multi-dimensional radar chart (used to visually display the document's strengths and weaknesses), a detailed list of deductions, and specific optimization suggestions for each item (document optimization suggestions). Furthermore, issues are highlighted in the document preview, indicating the specific sentences or paragraphs with problems (location hints). Users can modify the original Word document according to the suggestions in the report, thereby improving its AI friendliness and laying the foundation for higher-quality RAG applications.
[0069] In the technical solution of this application embodiment, firstly, based on the comprehensive evaluation results, the first content to be optimized in the source document is determined, and document optimization suggestions are generated for the first content to be optimized. Then, based on the comprehensive evaluation results and document optimization suggestions, a document evaluation report is generated. A brand-new, multi-dimensional document quality evaluation system specifically for RAG applications is proposed, which transforms the previously vague and invisible "document friendliness to AI" into a quantifiable, diagnosable, and optimizable specific indicator. At the same time, by intelligently generating specific and actionable optimization suggestions, a complete "diagnosis-optimization" automated solution is provided for knowledge base administrators, which greatly improves the maintenance efficiency of the knowledge base.
[0070] For example, one-click intelligent optimization of documents can also be performed based on the comprehensive evaluation results. For instance, firstly, based on the comprehensive evaluation results, a second content to be optimized in the source document is determined; then, based on the second content to be optimized, optimization instruction information is output; finally, in response to receiving the optimization instruction information for the optimization, the source document is optimized to generate an optimized document.
[0071] Specifically, based on the comprehensive evaluation results, specific structural problems in the source document (the second area to be optimized, such as the lack of use of standard heading styles) can be identified. The system can then provide a "one-click optimization" function (optimization instruction information) to automatically call the Word API or scripts to directly modify and generate an optimized version of the document.
[0072] In the technical solution of this application embodiment, firstly, based on the comprehensive evaluation results, the second content to be optimized in the source document is determined, then based on the second content to be optimized, optimization instruction information is output, and finally, in response to receiving the optimization instruction for the optimization instruction information, the source document is optimized to generate an optimized document, which effectively improves the optimization efficiency of the source document.
[0073] For example, the document evaluation report includes a retrieval generation simulator; it can also generate comparison information of the source document before and after optimization based on the retrieval generation simulator. For example, in response to receiving a question input by the user, the retrieval generation simulator retrieves the source document based on the question to obtain a first answer, and then retrieves the optimized document based on the question to obtain a second answer; then, based on the first answer and the second answer, answer comparison information is generated.
[0074] Specifically, a mini RAG question-and-answer simulator (retrieval generation simulator) can be integrated into the document evaluation report. Users can input several questions related to the document's topic, and the system will display a comparison of the answers that the RAG system might provide when using the "before optimization" and "after optimization" documents as knowledge sources (i.e., generating answer comparison information based on the first and second answers), thus more intuitively demonstrating the value of optimizing the source document.
[0075] In the technical solution of this application embodiment, in response to receiving a question input by the user, a first answer is obtained by searching the source document based on the question through a retrieval generation simulator. A second answer is obtained by searching the optimized document based on the question through the retrieval generation simulator. Then, based on the first answer and the second answer, answer comparison information is generated, thereby realizing the intelligent optimization of the source document and more intuitively demonstrating the value of source document optimization.
[0076] For example, source document evaluation methods can also be integrated online, such as by integrating them as plugins into a document editor to facilitate the evaluation of source documents while they are being written using the document editor.
[0077] Specifically, the system's scoring engine (the aforementioned AI-friendly scoring engine) can be directly integrated into online or local Word editors (such as Microsoft Office plugins or WPS plugins) as a plugin. Content creators can see the AI-friendly score and optimization suggestions of their documents in real time while writing them, achieving "optimization as you write".
[0078] In the technical solution of this application embodiment, the source document evaluation method is integrated into the document editor as a plugin, so as to evaluate the source document during the process of writing the source document through the document editor, which effectively improves the optimization efficiency and quality of the source document.
[0079] Figure 3 A schematic diagram of an apparatus for evaluating and retrieving generated source documents provided in an embodiment of this application.
[0080] like Figure 3 As shown, the source document evaluation processing apparatus 300 for retrieving generated documents includes: The parsing module 310 is used to parse the source document to obtain the text information, structured information and format information of the source document.
[0081] Evaluation module 320 is used to perform multi-dimensional evaluation processing based on text information, structured information, and format information to obtain at least two of the following: source document structural integrity evaluation result, content clarity evaluation result, context independence evaluation result, and format standardization evaluation result.
[0082] Processing module 330 is used to weight at least two of the structural integrity assessment results, content clarity assessment results, context independence assessment results, and format standardization assessment results to obtain a comprehensive assessment result of the source document.
[0083] For example, a multi-dimensional evaluation process is performed based on at least one of textual information, structured information, and formatting information to obtain a structural integrity evaluation result, including: determining the title style of the source document based at least on structured information to obtain a title standardization evaluation result; determining the list usage in the source document based at least on structured information to obtain a list usage evaluation result; determining the paragraph length in the source document based at least on textual information to obtain a paragraph length evaluation result; and obtaining a structural integrity evaluation result based on the title standardization evaluation result, the list usage evaluation result, and the paragraph length evaluation result.
[0084] For example, a multi-dimensional evaluation process is performed based on at least one of textual information, structured information, and formatted information to obtain a content clarity evaluation result, including: at least based on textual information, performing natural speech processing to identify the frequency of use of ambiguous words to obtain a language clarity evaluation result; at least based on textual information, performing technical terminology identification to obtain a technical terminology evaluation result; at least based on textual information, performing misspelling and grammar identification to obtain misspelling and grammar evaluation results; and based on the language clarity evaluation result, the technical terminology evaluation result, and the misspelling and grammar evaluation result, a content clarity evaluation result is obtained.
[0085] For example, a multi-dimensional evaluation process is performed based on at least one of textual information, structured information, and formatted information to obtain a context independence evaluation result, including: identifying ambiguous pronouns based at least on textual information to obtain a referential clarity evaluation result; determining the theme of the content corresponding to the title based at least on textual information and structured information to obtain an information atomization evaluation result; and obtaining a context independence evaluation result based on the referential clarity evaluation result and the information atomization evaluation result.
[0086] For example, a multi-dimensional evaluation process is performed based on at least one of text information, structured information, and format information to obtain a format standardization evaluation result, including: determining the ratio between text information and non-text information based on at least text information and format information to obtain a text extractability evaluation result; determining whether an image has a corresponding text description based on at least text information and format information to obtain an image-text separation evaluation result; and obtaining a format standardization evaluation result based on the text extractability evaluation result and the image-text separation evaluation result.
[0087] For example, the device 300 further includes a first generation module, configured to: determine a first content to be optimized in the source document based on the comprehensive evaluation results, and generate document optimization suggestions for the first content to be optimized; and generate a document evaluation report based on the comprehensive evaluation results and the document optimization suggestions, wherein the document evaluation report includes the comprehensive evaluation results and document rating information, a multi-dimensional radar chart of the source document, document optimization suggestions, and location information of the first content to be optimized in the source document.
[0088] For example, the device 300 further includes an optimization module, configured to: determine a second content to be optimized in the source document based on a comprehensive evaluation result; output optimization instruction information based on the second content to be optimized; and optimize the source document in response to receiving an optimization instruction for the optimization instruction information to generate an optimized document.
[0089] For example, the document evaluation report includes a retrieval generation simulator; the device 300 also includes a second generation module, configured to: in response to receiving a question input by a user, retrieve a first answer from the source document based on the question using the retrieval generation simulator, retrieve a second answer from the optimized document based on the question using the retrieval generation simulator, and generate answer comparison information based on the first and second answers.
[0090] For example, the device 300 also includes an integration module for: integrating the source document evaluation method as a plug-in into the document editor to facilitate the evaluation of the source document during the process of writing the source document through the document editor.
[0091] This application provides an electronic device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the method described above.
[0092] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method in any of the above embodiments.
[0093] Figure 4 A block diagram of an electronic device provided in an embodiment of this application.
[0094] This application provides an electronic device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method in any of the above embodiments.
[0095] like Figure 4 As shown, for ease of understanding, embodiments of this application illustrate a specific electronic device 400.
[0096] Electronic device 400 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic device 400 may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.
[0097] like Figure 4 As shown, device 400 includes a computing unit 401, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 402 or a computer program loaded from storage unit 408 into random access memory (RAM) 403. RAM 403 may also store various programs and data required for the operation of electronic device 400. The computing unit 401, ROM 402, and RAM 403 are interconnected via bus 404. Input / output (I / O) interface 404 is also connected to bus 404.
[0098] Multiple components in electronic device 400 are connected to I / O interface 404. These components include: input unit 406, such as a keyboard or mouse; output unit 407, such as various types of displays or speakers; storage unit 408, such as a disk or optical disk; and communication unit 409, such as a network interface card (NIC), modem, or wireless transceiver. Communication unit 409 allows electronic device 400 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.
[0099] The computing unit 401 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 401 performs the various methods described above. For example, in some embodiments, any one or more of the methods described above can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 408. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 400 via ROM 402 and / or communication unit 409. When the computer program is loaded into RAM 403 and executed by the computing unit 401, one or more steps of any one or more of the methods described above can be performed. Alternatively, in other embodiments, the computing unit 401 can be configured to perform any one or more of the methods described above by any other suitable means (e.g., by means of firmware).
[0100] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this application, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.
[0101] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.
[0102] In the description of this application, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this application, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.
[0103] In the description of this application, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", "axial", "radial", "circumferential", etc., indicating the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application.
[0104] Furthermore, the terms "first," "second," etc., used in the embodiments of this application are for descriptive purposes only and should not be construed as indicating or implying relative importance, or implicitly specifying the number of technical features indicated in this embodiment. Therefore, features defined with terms such as "first" and "second" in the embodiments of this application can explicitly or implicitly indicate that the embodiment includes at least one of those features. In the description of this application, the word "multiple" means at least two or more, such as two, three, four, etc., unless otherwise explicitly and specifically defined in the embodiments.
[0105] In this application, unless otherwise explicitly specified or limited in the embodiments, the terms "installation," "connection," "joining," and "fixing" appearing in the embodiments should be interpreted broadly. For example, a connection can be a fixed connection, a detachable connection, or an integral part; it can also be a mechanical connection, an electrical connection, etc. Of course, it can also be a direct connection, or an indirect connection through an intermediate medium, or it can be the internal communication between two components, or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific implementation.
[0106] In this application, unless otherwise expressly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "on top of," and "over" the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.
Claims
1. A method for evaluating and processing source documents generated through retrieval, characterized in that, The method includes: The source document is parsed to obtain its text information, structured information, and formatting information. Based on the text information, the structured information, and the format information, a multi-dimensional evaluation process is performed to obtain at least two of the following: the source document's structural integrity evaluation result, content clarity evaluation result, context independence evaluation result, and format standardization evaluation result. At least two of the structural integrity assessment results, content clarity assessment results, context independence assessment results, and format conformity assessment results are weighted to obtain a comprehensive assessment result for the source document.
2. The method according to claim 1, characterized in that, Based on at least one of the text information, the structured information, and the format information, a multi-dimensional evaluation process is performed to obtain the structural integrity evaluation result, including: Based at least on the structured information, the title style of the source document is determined, and the title standardization evaluation result is obtained; Based at least on the structured information, determine the list usage in the source document and obtain the list usage evaluation result; Based at least on the text information, determine the paragraph length in the source document to obtain a paragraph length evaluation result; Based on the evaluation results of the title standardization, the evaluation results of the list usage, and the evaluation results of the paragraph length, the evaluation results of the structural integrity are obtained.
3. The method according to claim 1, characterized in that, Based on at least one of the text information, the structured information, and the format information, a multi-dimensional evaluation process is performed to obtain the content clarity evaluation result, including: Based at least on the text information, natural speech processing is used to identify the frequency of use of ambiguous words and obtain language clarity assessment results. Based at least on the text information, technical terminology identification is performed to obtain technical terminology evaluation results; Based at least on the text information, typos and grammar are identified to obtain typo and grammar evaluation results; Based on the language clarity assessment results, the terminology assessment results, and the misspelling and grammar assessment results, the content clarity assessment results are obtained.
4. The method according to claim 1, characterized in that, Based on at least one of the text information, the structured information, and the format information, a multi-dimensional evaluation process is performed to obtain the context independence evaluation result, including: Based at least on the text information, ambiguous pronouns are identified, and the clarity of reference is evaluated. Based at least on the text information and the structured information, the topic of the content corresponding to the title is determined, and the information atomization evaluation result is obtained; Based on the reference clarity assessment result and the information atomization assessment result, the context independence assessment result is obtained.
5. The method according to claim 1, characterized in that, Based on at least one of the text information, the structured information, and the format information, a multi-dimensional evaluation process is performed to obtain the format standardization evaluation result, including: Based at least on the text information and the format information, the ratio between the text information and the non-text information is determined to obtain the text extractability evaluation result; Based at least on the text information and the format information, determine whether the image has a corresponding text description, and obtain the image-text separation evaluation result; Based on the text extractability evaluation results and the image-text separation evaluation results, the format standardization evaluation results are obtained.
6. The method according to any one of claims 1-5, characterized in that, The method further includes: Based on the comprehensive evaluation results, the first content to be optimized in the source document is determined, and document optimization suggestions are generated for the first content to be optimized. Based on the comprehensive evaluation results and the document optimization suggestions, a document evaluation report is generated. The document evaluation report includes the comprehensive evaluation results and document rating information, a multi-dimensional radar chart of the source document, the document optimization suggestions, and location information of the first content to be optimized in the source document.
7. The method according to claim 6, characterized in that, The method further includes: Based on the comprehensive evaluation results, the second content to be optimized in the source document was determined; Based on the second content to be optimized, output optimization instruction information; In response to receiving an optimization instruction for the optimization indication information, the source document is optimized to generate an optimized document.
8. The method according to claim 7, characterized in that, The document evaluation report includes a retrieval generation simulator; the method also includes: In response to receiving a question input by the user, the retrieval generation simulator retrieves a first answer from the source document based on the question, and then retrieves a second answer from the optimized document based on the question. Based on the first answer and the second answer, generate answer comparison information.
9. The method according to any one of claims 1-5 and 7-8, characterized in that, The method further includes: The source document evaluation method is integrated into the document editor as a plugin to facilitate the evaluation of the source document during the writing process using the document editor.
10. An apparatus for evaluating and processing source documents generated through retrieval, characterized in that, The device includes: The parsing module is used to parse the source document to obtain the text information, structured information and format information of the source document; The evaluation module is used to perform multi-dimensional evaluation processing based on the text information, the structured information, and the format information to obtain at least two of the following: the structural integrity evaluation result, the content clarity evaluation result, the context independence evaluation result, and the format standardization evaluation result of the source document. The processing module is used to perform weighted processing on at least two of the structural integrity assessment results, the content clarity assessment results, the context independence assessment results, and the format standardization assessment results to obtain a comprehensive assessment result of the source document.