A document quality evaluation method and computing device
By employing multi-dimensional evaluation methods and information extraction models, the accuracy of document quality evaluation was addressed, enabling fine-grained scoring and improvement guidance for document content, thereby enhancing the accuracy of document quality evaluation and improving user experience.
Patent Information
- Application Number
- CN202310450781.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-04-24
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2043-04-24
AI Technical Summary
In existing technologies, the accuracy of document quality assessment is low, and different users have different understandings of document quality, resulting in insufficient objectivity of the scoring model.
A multi-dimensional evaluation method is adopted. By acquiring the evaluation data of the document to be tested, the information extraction model is used to perform lexical, syntactic and semantic analysis to extract key text fragments, which are then matched with the evaluation dictionary to calculate the quality scores of multiple evaluation dimensions, including content accessibility, content completeness, content accuracy and content comprehensibility.
It improves the accuracy and reliability of document quality assessment, provides improvement instructions and quality assessment tags, helps document authors optimize document content, and enhances the user experience.
Smart Images

Figure CN116629241B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of server technology, and in particular to a document quality evaluation method and computing device. Background Technology
[0002] Product documentation is a document that introduces information such as product features, how to use it, and how to troubleshoot problems. There are numerous product documents published on the internet, and document authors or users need to understand the quality of these documents, such as which parts are highly regarded by users and which are not. Therefore, document quality evaluation is necessary.
[0003] Currently, document quality is scored using evaluation models, but these scores reflect the overall quality of the document, resulting in a coarse-grained assessment and low accuracy. Furthermore, differing user interpretations of document quality detract from the objectivity of the trained scoring models. Therefore, improving the accuracy of document quality evaluation is a pressing issue. Summary of the Invention
[0004] This application provides a document quality evaluation method and computing device, which can improve the accuracy of document quality evaluation.
[0005] In a first aspect, embodiments of this application provide a document quality evaluation method, the method comprising:
[0006] Obtain evaluation data for the document to be tested;
[0007] At least one target text fragment is obtained based on the evaluation data;
[0008] At least one target text fragment is matched with an evaluation dictionary from multiple evaluation dimensions to obtain matching results. The matching results include matching information corresponding to multiple evaluation dimensions. The matching information corresponding to each evaluation dimension includes the matched text fragment and the evaluation parameters corresponding to the matched text fragment. The evaluation dictionary includes at least one reference text fragment. The evaluation parameters are used to indicate the scores of the advantageous or disadvantageous features of the document content described by each matched text fragment.
[0009] The quality evaluation data of the document to be detected is determined based on the matching results; the quality evaluation data includes quality scores corresponding to multiple evaluation dimensions.
[0010] In one possible implementation, the evaluation dictionary includes a sub-dictionary corresponding to each evaluation dimension;
[0011] At least one target text fragment is matched against an evaluation dictionary across multiple evaluation dimensions to obtain matching results, including:
[0012] Match at least one target text fragment with reference text fragments in the target sub-dictionary corresponding to the target evaluation dimension to obtain a matching text fragment that matches at least one target text fragment; the matching text fragment includes positive text fragments and negative text fragments; wherein, the target evaluation dimension is any one of multiple evaluation dimensions, the positive text fragment is used to describe the advantageous features of the content of the document to be detected, and the negative text fragment is used to describe the disadvantageous features of the content of the document to be detected;
[0013] Determine the evaluation parameters corresponding to the matching text fragments from the evaluation parameter comparison data;
[0014] The matching information corresponding to the target evaluation dimension is determined based on the matching text fragment and the corresponding evaluation parameters of the matching text fragment.
[0015] The matching result is determined based on the matching information corresponding to multiple evaluation dimensions.
[0016] This implementation method allows the management server to score the document quality for each evaluation dimension based on both the strengths and weaknesses of the document's content, thereby improving the accuracy of the document's quality score.
[0017] In one feasible approach, quality assessment data for the document to be detected is determined based on the matching results, including:
[0018] The evaluation parameters for each positive text segment and the evaluation parameters for each negative text segment are determined from the matching information corresponding to the target evaluation dimension.
[0019] The positive evaluation parameters are determined based on each positive text segment and the evaluation parameters corresponding to each positive text segment;
[0020] The negative evaluation parameters are determined based on each negative text segment and the evaluation parameters corresponding to each negative text segment;
[0021] The quality score corresponding to the target evaluation dimension is determined based on positive and negative evaluation parameters;
[0022] The quality evaluation data of the document to be tested is determined based on the quality scores corresponding to multiple evaluation dimensions.
[0023] In one feasible approach, at least one target text fragment is obtained based on evaluation data, including:
[0024] Based on the information extraction model, lexical analysis, syntactic analysis and semantic analysis are performed on the evaluation data to obtain at least one target text fragment;
[0025] The information extraction model is obtained by strengthening the training of the natural language processing model through syntactic analysis based on the evaluation data and labeled data of the sample documents. The labeled data includes at least one labeled text segment of the evaluation data of the sample documents.
[0026] This implementation method allows the management server to use the evaluation data and labeled data of sample documents to perform targeted reinforcement training on the natural language processing model, ensuring the accuracy of the model's output results. Based on the model's output results, the quality evaluation data of the documents can be determined, thereby improving the accuracy of document quality evaluation.
[0027] In one feasible approach, before performing lexical, syntactic, and semantic analysis on the evaluation data based on an information extraction model to obtain at least one target text fragment, the method further includes:
[0028] Obtain the evaluation data and annotation data of the sample document. The annotation data includes at least one annotated text fragment of the evaluation data of the sample document.
[0029] Natural language processing models are used to extract information from the evaluation data of sample documents to obtain predicted text fragments.
[0030] The sentence components of the labeled text fragments and the sentence components of the predicted text fragments are determined. The sentence components of the labeled text fragments include the combination relationships between the parts of speech of each text element in the labeled text fragments, and the sentence components of the predicted text fragments include the combination relationships between the parts of speech of each text element in the predicted text fragments.
[0031] The parameters of a natural language processing model are adjusted based on the difference data between the sentence components of annotated text fragments and the sentence components of predicted text fragments, resulting in an information extraction model.
[0032] This implementation allows the management server to train a natural language processing model based on the difference data between the sentence components of the labeled text fragments and the sentence components of the predicted text fragments, thereby obtaining an information extraction model and improving the prediction accuracy of the information extraction model.
[0033] In one possible implementation, the method further includes:
[0034] The document's positive score is determined based on each positive text fragment and the corresponding evaluation parameters.
[0035] The document's negative score is determined based on each negative text segment and the corresponding evaluation parameters.
[0036] Improvement instructions for the document to be tested are determined based on the document's positive and negative scores. These instructions are used to guide the first target user to edit the document.
[0037] This implementation allows the management server to output improvement instructions to the primary target user (such as the document author), providing them with document scores and a list of additions and subtractions to facilitate document improvement.
[0038] In one possible implementation, the method further includes:
[0039] The frequency of occurrence of each positive text segment and the frequency of occurrence of each negative text segment in the matching text segment are determined based on the content of the text segment.
[0040] The weight parameters of each positive text segment are determined based on the frequency of occurrence and evaluation parameters of each positive text segment in the matched text segment, and the weight parameters of each negative text segment are determined based on the frequency of occurrence and evaluation parameters of each negative text segment in the matched text segment.
[0041] The quality evaluation label of the document to be detected is determined based on the weight parameters of each positive text segment and each negative text segment. The quality evaluation label is used to instruct the second target user to use the document to be detected based on the quality evaluation label.
[0042] This implementation allows the management server to output quality evaluation labels to a second target user (such as a document user), providing feedback on the strengths and weaknesses of the document under test. This enables the second target user to use the document in a targeted manner, ensuring a better user experience.
[0043] In one possible implementation, the quality evaluation labels of the document to be detected are determined based on the weight parameters of each positive text segment and each negative text segment, including:
[0044] Sort the weight parameters of each positive text segment in descending order to obtain the K positive text segments corresponding to the top K weight parameters, where K is a positive integer;
[0045] Sort the weight parameters of each negative text segment in descending order to obtain the L negative text segments corresponding to the weight parameters in the first L positions, where L is a positive integer;
[0046] Based on K positive text fragments and L negative text fragments, determine the quality evaluation labels for the document to be tested.
[0047] This implementation allows the management server to customize the number of positive and negative text segments corresponding to quality evaluation labels, thereby increasing the diversity of quality evaluation label formats.
[0048] In one feasible approach, multiple evaluation dimensions include content accessibility, content completeness, content accuracy, and content comprehensibility. Content accessibility indicates the ease of accessing the content in the document under test, content completeness indicates the completeness of the content in the document under test, content accuracy indicates the accuracy of the content in the document under test, and content comprehensibility indicates the ease of understanding the content in the document under test.
[0049] This implementation method enables the management server to evaluate document quality through multiple evaluation dimensions, improving the accuracy of document quality evaluation and making it more reliable and credible.
[0050] Secondly, embodiments of this application provide a document quality evaluation device, the device comprising:
[0051] The acquisition module is used to acquire evaluation data for the document to be tested;
[0052] The processing module is used to obtain at least one target text fragment based on the evaluation data;
[0053] The processing module is also used to match at least one target text fragment with an evaluation dictionary from multiple evaluation dimensions to obtain matching results. The matching results include matching information corresponding to multiple evaluation dimensions. The matching information corresponding to each evaluation dimension includes the matched text fragment and the evaluation parameters corresponding to the matched text fragment. The evaluation dictionary includes at least one reference text fragment. The evaluation parameters are used to indicate the scores of the advantageous or disadvantageous features of the document content described by each matched text fragment.
[0054] The evaluation module is used to determine the quality evaluation data of the document to be detected based on the matching results; the quality evaluation data includes quality scores corresponding to multiple evaluation dimensions.
[0055] Thirdly, embodiments of this application provide a computing device, which includes a processor, a communication interface, and a memory. The processor, communication interface, and memory are electrically connected. The communication interface is controlled by the processor to send and receive data. The memory is used to store a computer program, which includes program instructions. The processor is configured to call the program instructions to execute the document quality evaluation method of the first aspect.
[0056] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, the computer program including program instructions, which are executed by a processor to perform the document quality evaluation method of the first aspect.
[0057] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a computer processor, implements the document quality evaluation method of the first aspect.
[0058] In this embodiment, information extraction processing is performed on the evaluation data of the document to be tested to obtain target text fragments with key semantic information. Then, the target text fragments and the evaluation dictionary are matched to obtain matching information corresponding to multiple evaluation dimensions, thereby obtaining quality evaluation data including quality scores corresponding to multiple evaluation dimensions of the document to be tested. The matching information for each evaluation dimension includes a matched text fragment and its corresponding evaluation parameters. By calculating the quality scores of multiple evaluation dimensions using the matched text fragments and their respective evaluation parameters, multi-dimensional quality analysis can be performed, thereby improving the accuracy of document quality evaluation. Attached Figure Description
[0059] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0060] Figure 1 This is a schematic diagram of the architecture of a document quality evaluation system provided in an embodiment of this application;
[0061] Figure 2 This is a flowchart illustrating a document quality evaluation method provided in an embodiment of this application;
[0062] Figure 3 This is a schematic diagram illustrating an evaluation dimension division provided in an embodiment of this application;
[0063] Figure 4 This is a flowchart illustrating another document quality evaluation method provided in an embodiment of this application;
[0064] Figure 5a This is a schematic diagram illustrating an example of reinforcement training for an NLP model provided in this application embodiment;
[0065] Figure 5bThis is a schematic diagram of a process for generating improved instruction information and quality evaluation labels for documents, provided in an embodiment of this application.
[0066] Figure 5c This is a schematic diagram of a quality evaluation label provided in an embodiment of this application;
[0067] Figure 5d This is a schematic diagram of a document quality evaluation process provided in an embodiment of this application;
[0068] Figure 6 This is a flowchart illustrating another document quality evaluation method provided in an embodiment of this application;
[0069] Figure 7 This is a schematic diagram of another document quality evaluation process provided in an embodiment of this application;
[0070] Figure 8 This is a schematic diagram of the structure of a document quality evaluation device provided in an embodiment of this application;
[0071] Figure 9 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Detailed Implementation
[0072] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.
[0073] It should be noted that the terms "first," "second," etc., used in the embodiments of this application are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a technical feature specified with "first" or "second" may explicitly or implicitly include at least one of those features.
[0074] To better understand the solutions of the embodiments of this application, the relevant terms and concepts that may be involved in the embodiments of this application will be introduced below.
[0075] Product documentation: Documentation that introduces the product's features, installation and debugging methods, usage instructions, and troubleshooting procedures.
[0076] Natural Language Processing (NLP) refers to the technology of using natural language, the language humans use for communication, to interact and communicate with machines. Through human processing of natural language, computers can read and understand it.
[0077] Deep learning: Deep learning learns the inherent patterns and hierarchical representations of sample data to obtain information such as text, images, and sounds, which can explain the patterns in the data. Its ultimate goal is to enable machines to have analytical and learning capabilities like humans, and to recognize data such as text, images, and sounds.
[0078] Document quality: A series of metrics that measure the quality of a document from different dimensions.
[0079] Semantics: The meaning contained in language.
[0080] Document author: Generally, this refers to the document editor or writer who is responsible for the quality and content of the document.
[0081] Document managers typically include staff operating technical support websites and product managers. Technical support website operators need to know the quality of each product document, while product managers consider documentation an integral part of the product and also want to understand its quality.
[0082] There are numerous product documents published online, and for cloud-related products, these documents are even more complex in terms of pages and content. Document authors and users need to understand document quality, such as which pages and content are highly praised and which are frequently criticized, thus requiring quality evaluation. However, user evaluations are scattered and lack automated analysis dimensions and methods. Current user ratings only provide a single score, lacking categorization dimensions and detailed information. While evaluation models can score documents, they also only provide a single score, lacking scoring dimensions and detailed information, significantly impacting the accuracy of document quality assessment. Furthermore, evaluation models are essentially black boxes; dimensional division is uncontrollable, making it difficult to propose improvement directions and strategies from the perspective of product document enhancement. Moreover, different users have inconsistent understandings of document quality, resulting in insufficient objectivity in the trained scoring models.
[0083] Based on this, this application proposes a document quality evaluation method that assesses document quality from four dimensions: content accessibility, content completeness, content accuracy, and content comprehensibility, thereby improving the accuracy of document quality evaluation. Furthermore, by providing improvement instructions, it facilitates document authors in optimizing their documents, and by providing document quality evaluation tags (a type of feature tag) based on document evaluation characteristics, it facilitates document use for users. In addition, this application's embodiments utilize an information extraction model to extract key text fragments from the evaluation data. This information extraction model is derived by specifically improving a natural language processing model based on user evaluation characteristics and a commonly used vocabulary for technical document evaluation. Using this information extraction model to extract key text fragments from the evaluation data ensures the accuracy of these key text fragments, thereby improving the accuracy of document quality evaluation.
[0084] To better understand the technical solutions provided in the embodiments of this application, the relevant embodiments of this application are described below.
[0085] Please see Figure 1 , Figure 1 This is a schematic diagram of the architecture of a document quality assessment system provided in an embodiment of this application. The document quality assessment system includes a management server 101, a first terminal device 102, and a second terminal device 103, wherein:
[0086] Management server 101 provides services such as information extraction, dictionary matching, and document quality evaluation of evaluation data for documents to be tested. For example, users of each document to be tested can store evaluation data (e.g., evaluation text) in a database after using the document. Management server 101 can retrieve the evaluation data of the document to be tested from the database, extract information from the evaluation data to obtain at least one target text fragment, and then perform matching processing on the at least one target text fragment and the evaluation dictionary to obtain matching results. The matching results include matching information corresponding to multiple evaluation dimensions. The matching information corresponding to each evaluation dimension includes the matching text fragment and the evaluation parameters corresponding to the matching text fragment. Among them, the evaluation parameters are used to indicate the score of the advantageous or disadvantageous features of the document content described by each matching text fragment. Finally, management server 101 determines the quality evaluation data of the document to be tested based on the matching results. The quality evaluation data includes quality scores corresponding to multiple evaluation dimensions, thereby characterizing the document quality of the document to be tested through quality evaluation data.
[0087] In addition, the management server 101 can determine a document's positive score item based on the positive and negative text fragments included in the matched text fragments, according to the evaluation parameters corresponding to each positive text fragment; and determine a document's negative score item based on the evaluation parameters corresponding to each negative text fragment. Finally, improvement instruction information for the document to be tested is determined based on the document's positive and negative score items. The management server 101 can also determine the frequency of occurrence of each positive and negative text fragment in the matched text fragments based on the content of the text fragments. Then, a weight parameter for each positive text fragment is determined based on the frequency of occurrence of each positive text fragment in the matched text fragments, and a weight parameter for each negative text fragment is determined based on the frequency of occurrence of each negative text fragment in the matched text fragments. Finally, a quality evaluation label for the document to be tested is determined based on the weight parameters of each positive and negative text fragment. Using the above method, while obtaining the quality evaluation data of the document to be tested, improvement instructions and quality evaluation labels can be obtained, providing document users or document authors with reference information related to document quality, so that document users can edit and use the document, and document authors can make subsequent document improvements.
[0088] The first terminal device 102 may specifically be the terminal device corresponding to the first target object (e.g., the author of the document to be tested). The first target object can obtain the quality evaluation data of the document to be tested and the improvement instruction information of the document to be tested from the management server 101 through the user interface (such as a web interface) provided by the first terminal device 102, and then edit the document to be tested based on the improvement instruction information. For example, the improvement instruction information in the form of a form can provide feedback on each of the plus or minus points of the document to be tested, so that the first target object can understand how to modify the document and provide a reference for document modification and optimization strategies.
[0089] The second terminal device 103 may specifically be the terminal device corresponding to the second target object (e.g., the user of the document to be tested). The second target object can obtain the quality evaluation data and quality evaluation tags of the document to be tested from the management server 101 through the user interface (such as a web interface) provided by the second terminal device 103, and then use the document to be tested based on the quality evaluation tags.
[0090] In one implementation, the management server 101 can, based on two categories—positive and negative scores—according to a rule of score * frequency (the evaluation parameter corresponding to the matched text fragment * the number of times the matched text fragment appears, resulting in a weight parameter), add TopK quality evaluation tags with the largest weight parameters for both positive and negative scores to the end of the document. Taking positive scores as an example, the management server can multiply the frequency of each positive text fragment by its evaluation parameter to obtain the weight parameter for each positive text fragment. The management server then selects the top K positive text fragments with the largest corresponding weight parameters from all positive text fragments, using these as the TopK quality evaluation tags with the largest weight parameters for the positive scores. After the management server 101 adds quality evaluation tags to the end of the document using the method described in the above embodiment (the quality evaluation tags can include: TopK quality evaluation tags with the largest weight parameters for positive scores, and / or TopK quality evaluation tags with the largest weight parameters for negative scores), the second target object can quickly and intuitively understand the document's characteristics based on the quality evaluation tags, facilitating targeted document use and ensuring a better user experience.
[0091] Specifically, the first terminal device 102 and the second terminal device 103 can be smartphones, tablets, laptops, desktop computers, vehicle terminals, wearable devices, etc., and this application embodiment does not limit them. It should be noted that the first terminal device 102 and the second terminal device 103 can refer to the same terminal device or different terminal devices. When the first terminal device 102 and the second terminal device 103 refer to the same terminal device, the first target object and the second target object can be the same object (e.g., the author of the document to be detected is also the user of the document to be detected).
[0092] It should be noted that, Figure 1The number of management server 101, first terminal device 102, and second terminal device 103 shown is merely illustrative. Any number of management servers 101, first terminal devices 102, and second terminal devices 103 can be deployed according to actual needs. Management server 101 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data. In subsequent embodiments, "management server" will refer to management server 101, "first terminal device" will refer to first terminal device 102, and "second terminal device" will refer to second terminal device 103. Further details will not be repeated in subsequent embodiments.
[0093] The implementation details of the technical solutions in the embodiments of this application are described in detail below:
[0094] Please see Figure 2 This application's embodiments are based on Figure 1 The document quality assessment system shown presents a flowchart illustrating a document quality assessment method that can be applied to... Figure 1 The management server 101 in the system may specifically include the following steps:
[0095] 201. Obtain the evaluation data of the document to be tested.
[0096] The document to be tested can refer to a user manual for a product, used to inform users how to use it, how to handle malfunctions, etc. The evaluation data of the document to be tested can refer to the text data generated by users evaluating the content of the document after reading or browsing it, such as identifying which parts of the document are well-written and which are poorly written. In this embodiment, the evaluation data can be considered as a set of evaluation data, which can contain one or more evaluation data points. Furthermore, the evaluation data set can include one or more evaluation data points generated by a single user after using the document to be tested, or it can include one or more evaluation data points generated by multiple users after using the document to be tested.
[0097] It should be noted that the evaluation data of the document to be detected may be evaluation data of a text type, and may also be evaluation data of various data types such as video, audio, and image. The embodiments of the present application do not limit the data type of the evaluation data. When processing the evaluation data, the management server may first convert the evaluation data of an original data type (such as an audio type) into evaluation data of a target data type (such as a text type), and then perform subsequent processing such as information extraction and evaluation dictionary matching, so as to ensure the accuracy of the processing result.
[0098] For example, the document to be detected may be a usage document of a certain application product, and the evaluation data may be text data such as "the path in the first chapter does not match the path of the latest version", "whether module B still points to the motherboard is not introduced in the document", "the interface corresponding to function A is completely different from the actual interface", and "many parameters are written ambiguously and their meanings are unclear".
[0099] 202. Obtain at least one target text segment based on the evaluation data.
[0100] The obtaining at least one target text segment based on the evaluation data may mean that the management server performs information extraction processing on the evaluation data to obtain at least one target text segment. The target text segment may be regarded as a key text segment with key semantic information in the evaluation data. The management server may perform information extraction processing on each piece of evaluation data in the evaluation data set separately to obtain a corresponding target text segment. One or more target text segments may be obtained from each piece of evaluation data through information extraction. In addition, for some evaluation data with unreasonable content (such as "uhhh"), a target text segment may not be extracted, which is not limited in the embodiments of the present application. By acquiring the target text segment in the evaluation data for subsequent matching processing with the evaluation dictionary, the management server can perform corresponding scoring processing based on the key text segment in the document to be detected, so as to obtain the quality evaluation data of the document to be detected, and improve the accuracy of document quality evaluation.
[0101] For example, the target text segment extracted from the evaluation data "the path in the first chapter does not match the path of the latest version" may be "path mismatch". The target text segment extracted from the evaluation data "whether module B still points to the motherboard is not introduced in the document" may be "no introduction in the document". The target text segment extracted from "the interface corresponding to function A is completely different from the actual interface" may be "different from the actual interface". The target text segment extracted from "many parameters are written ambiguously and their meanings are unclear" may be "meaning unclear".
[0102] 203. Match at least one target text fragment with an evaluation dictionary from multiple evaluation dimensions to obtain matching results. The matching results include matching information corresponding to multiple evaluation dimensions. The matching information corresponding to each evaluation dimension includes the matched text fragment and the evaluation parameters corresponding to the matched text fragment.
[0103] The evaluation dictionary includes at least one reference text fragment. A match is established when the text content of a reference text fragment in the evaluation dictionary matches that of at least one target text fragment. The management server can perform matching processing on at least one target text fragment obtained through information extraction and the evaluation dictionary based on multiple evaluation dimensions of document evaluation, obtaining matching information corresponding to each evaluation dimension. The matching information for each evaluation dimension can include the matched text fragment and its corresponding evaluation parameters. The matched text fragment is one of the at least one reference text fragments included in the evaluation dictionary, and its text content matches the text content of the target text fragment (e.g., completely identical content). Since different target text fragments have different semantic information, by obtaining the evaluation parameters corresponding to each matched text fragment for subsequent document quality score calculation, targeted scoring can be performed based on different evaluation content, thereby improving the accuracy and reliability of document quality evaluation.
[0104] The evaluation parameters indicate the score of the strengths or weaknesses of the document content described by each matching text fragment. Strengths refer to the advantages of the content in the document being tested, while weaknesses refer to the disadvantages. For example, the weakness of the matching text fragment "incorrect description" could mean that the content in the document being tested is described incorrectly, and the weakness of the matching text fragment "unclear description" could mean that the content in the document being tested is not clearly described. Therefore, the evaluation parameter for "incorrect description" could be 1.0, and the evaluation parameter for "unclear description" could be 0.6.
[0105] For example, the evaluation dictionary includes at least one reference text fragment, and each reference text fragment corresponds to an evaluation parameter, which can be the score of the reference text fragment. For example, at least one reference text fragment includes "path mismatch", "no description in the document", "different from the actual interface", and "unclear meaning", and their corresponding evaluation parameters (scores) are 0.8, 0.8, 0.6 and 1, respectively.
[0106] In one possible implementation, at least one target text fragment may be included in the reference text fragment of the evaluation dictionary. This indicates that the keyword text fragments extracted from the evaluation data can match the evaluation dictionary and are used for subsequent document score calculation. However, at least one target text fragment may also exist that is not included in the reference text fragment of the evaluation dictionary. This indicates that the keyword text fragments extracted from these one or more target text fragments do not match the evaluation dictionary, and therefore, these one or more target text fragments can be ignored.
[0107] In one feasible approach, multiple evaluation dimensions include content accessibility, content completeness, content accuracy, and content comprehensibility. Content accessibility indicates the ease with which the content in the document under test can be retrieved, such as whether the content a user needs is easily obtained from the document. Content completeness indicates the completeness of the content in the document under test, such as whether the content description is complete and without omissions. Content accuracy indicates the accuracy of the content in the document under test, such as whether the content description is correct. Content comprehensibility indicates the ease with which the content in the document under test can be understood, such as whether the content description is easy to understand, concise, and clear. Evaluating document quality through multiple dimensions improves the accuracy of document quality assessment, making it more evidence-based and credible. This method allows document users and authors to understand the strengths, weaknesses, and characteristics of a document from multiple dimensions, improving the user experience.
[0108] In one possible implementation, multiple evaluation dimensions can refer to at least two of the following: content accessibility, content completeness, content accuracy, and content comprehensibility. The method for matching at least two of these evaluation dimensions with an evaluation dictionary to obtain a matching result, and then determining the quality evaluation data of the document to be tested based on the matching result, can be found in the relevant descriptions in other embodiments of this application, and will not be repeated in this embodiment.
[0109] Please see Figure 3 , Figure 3This is a schematic diagram of an evaluation dimension division provided in an embodiment of this application. The evaluation data includes four text data items: "The path in the first chapter does not match the path in the latest version," "Does module B still point to the motherboard? There is no description in the document," "The interface corresponding to function A is completely different from the actual interface," and "Many parameters are written unclearly and their meaning is unclear." The corresponding reference text fragments are "path mismatch," "no description in the document," "different from the actual interface," and "unclear meaning," respectively. Among them, "path mismatch" is obtained by matching based on the content accessibility evaluation dimension, "no description in the document" is obtained by matching based on the content completeness evaluation dimension, "different from the actual interface" is obtained by matching based on the content accuracy evaluation dimension, and "unclear meaning" is obtained by matching based on the content comprehensibility evaluation dimension. Generally speaking, the reference text fragments corresponding to content accessibility usually include keywords such as "found," "query," and "send to me"; the reference text fragments corresponding to content completeness usually include keywords such as "improve and supplement" and "add"; the reference text fragments corresponding to content accuracy usually include keywords such as "error" and "incorrect"; and the reference text fragments corresponding to content comprehensibility usually include keywords such as "blurry," "cannot be zoomed in," and "not clear enough."
[0110] 204. Determine the quality evaluation data of the document to be detected based on the matching results. The quality evaluation data includes quality scores corresponding to multiple evaluation dimensions.
[0111] The management server can determine the quality score (matching result) for each evaluation dimension based on the matching information, and then determine the quality evaluation data of the document to be tested based on the quality score of each evaluation dimension. The quality evaluation data of the document to be tested provides a clear and intuitive understanding of the document's quality from four dimensions: accessibility, completeness, accuracy, and comprehensibility. This allows users and authors to better understand the document's strengths and weaknesses and improves the user experience.
[0112] In one possible implementation, the management server can determine the quality score for each evaluation dimension based on the matching text fragments, the evaluation parameters corresponding to the matching text fragments, and their frequency of occurrence. Then, the quality scores across multiple evaluation dimensions are used as the quality evaluation data for the document under test. Alternatively, the management server can calculate the average of the quality scores across multiple evaluation dimensions and combine the average value with the individual quality scores for each evaluation dimension as the quality evaluation data for the document under test. These methods improve the flexibility of the format and content of the quality evaluation data.
[0113] For example, multiple evaluation dimensions include: content accessibility, content completeness, content accuracy, and content comprehensibility. Matching text fragments include "path mismatch," "no description in document," "different from the actual interface," and "unclear meaning," with corresponding evaluation parameters of 0.8, 0.8, 0.6, and 1, and corresponding frequencies of occurrence of 30, 20, 10, and 50, respectively. Therefore, by calculating the evaluation parameter multiplied by the frequency of occurrence, the quality scores for the four evaluation dimensions—content accessibility, content completeness, content accuracy, and content comprehensibility—are 24, 16, 6, and 50, respectively. It should be noted that, for ease of description, the number of matching text fragments corresponding to each evaluation dimension and the evaluation parameters corresponding to each matching text fragment are merely exemplary. In the application stage of this application embodiment, the number of matching text fragments corresponding to each evaluation dimension and the evaluation parameters corresponding to each matching text fragment should be adjusted according to the actual business situation. This application embodiment will not elaborate further.
[0114] In this embodiment, information extraction processing is performed on the evaluation data of the document to be tested to obtain target text fragments with key semantic information. Then, the target text fragments are matched with the evaluation dictionary to obtain matching information corresponding to multiple evaluation dimensions, thereby obtaining quality evaluation data including quality scores corresponding to multiple evaluation dimensions of the document to be tested. Each evaluation dimension's matching information includes a matched text fragment and its corresponding evaluation parameters. By calculating the quality scores of multiple evaluation dimensions using the matched text fragments and their respective evaluation parameters, multi-dimensional quality analysis can be performed, improving the accuracy of document quality evaluation. This embodiment also proposes that evaluation dimensions may include content accessibility, content completeness, content correctness, and content comprehensibility. Using multiple evaluation dimensions for document quality evaluation improves the accuracy of document quality evaluation, making it more evidence-based and credible. This method allows document users and authors to understand the strengths and weaknesses and characteristics of a document from multiple dimensions, improving the document's user experience.
[0115] Please see Figure 4 This application's embodiments are based on Figure 1 The diagram shown illustrates another document quality assessment method provided by the document quality assessment system. This method can be applied to... Figure 1 The management server 101 in the system may specifically include the following steps:
[0116] 401. Obtain the evaluation data of the document to be tested.
[0117] For the specific implementation process of step 401, please refer to the relevant description of step 201, which will not be repeated in this embodiment.
[0118] 402. Based on the information extraction model, extract information from the evaluation data to obtain at least one target text fragment.
[0119] The management server can extract information from the evaluation data using an information extraction model to obtain at least one target text fragment, such as a key text fragment. This information extraction model can be trained using evaluation data and labeled data (key text fragments from the evaluation data of the sample documents) on a model such as a Natural Language Processing (NLP) model. This process of extracting information from the evaluation data using an information extraction model can be considered a function provided by a Natural Language Processing unit.
[0120] In one feasible approach, the aforementioned information extraction based on the information extraction model for evaluation data can refer to using the information extraction model to perform lexical analysis, syntactic analysis, and semantic analysis on the evaluation data to obtain at least one target text fragment. By performing multi-dimensional analysis including lexical analysis, syntactic analysis, and semantic analysis, the accuracy of the predicted target text fragment can be improved.
[0121] Information extraction mainly includes three parts: lexical analysis, syntactic analysis, and semantic analysis. Lexical analysis primarily involves segmenting words from a sentence, identifying each morpheme, obtaining linguistic information about the words, and determining their meanings. Syntactic analysis mainly analyzes the sentence's phrase structure to determine the relationships between each word and phrase that constitute the sentence, as well as their respective roles within the sentence. These relationships are expressed using a hierarchical structure, and the syntactic structure is standardized. Semantic analysis primarily associates the analyzed syntactic components with the goals in the application domain, such as interpreting the meaning of different parts (words, phrases, sentences, paragraphs, and texts) of natural language sentences or discourses.
[0122] Because related technologies use Natural Language Processing (NLP) models that provide a two-dimensional linear classification (positive and negative emotions) from a sentiment dimension, sentiment-related words constitute a large proportion, while the documents to be tested are generally user-generated after-sales documents. In the evaluation data of the documents to be tested, we need sentences and phrases of irregular behavioral descriptions, not sentiment words. Therefore, this embodiment uses evaluation data and labeled data (evaluation samples) from sample documents to enhance the training of the syntactic analysis stage, such as enhancing the training of sentence components like nouns, verbs, adjectives, adverbs, and particles, to obtain an information extraction model. This improves the accuracy of the target text fragments output by the model and expands the evaluation dimension to four dimensions. The specific training process of the information extraction model will be described in detail in subsequent embodiments and will not be repeated in this embodiment.
[0123] In this embodiment, the evaluation dictionary may include a sub-dictionary corresponding to each evaluation dimension, such as one sub-dictionary for each evaluation dimension. Based on this, the following will explain the steps 403-405 for matching at least one target text fragment with the evaluation dictionary from multiple evaluation dimensions to obtain the matching result:
[0124] 403. Match at least one target text fragment with the reference text fragment in the target sub-dictionary corresponding to the target evaluation dimension to obtain a matching text fragment that matches at least one target text fragment.
[0125] The target evaluation dimension can be any one of multiple evaluation dimensions. Taking the processing of the target evaluation dimension as an example, the management server matches at least one target text fragment with a reference text fragment in the target sub-dictionary corresponding to the target evaluation dimension, obtaining at least one matching text fragment from the target text fragments that matches the aforementioned reference text fragment (matching can mean content consistency). The matching text fragment is one of the multiple reference text fragments included in the target sub-dictionary. Matching text fragments include positive and negative text fragments. Positive text fragments describe the advantageous features of the content of the document to be tested. Advantageous features refer to the strengths of the content in the document to be tested, such as: clear structure, comprehensive content, complete content, detailed introduction, accurate description, straightforward content, and vivid and specific content description. Negative text fragments describe the disadvantageous features of the content of the document to be tested. Disadvantageous features refer to the weaknesses of the content in the document to be tested, such as: disorganized structure, incorrect content, missing content, unclear introduction, inaccurate description, and difficult-to-understand content. Positive text fragments can be used to calculate positive scores (such as bonus points for the document under test), while negative text fragments can be used to calculate negative scores (such as deduction points for the document under test).
[0126] Please refer to Tables 1-4. Tables 1-4 list, for example, the sub-dictionaries corresponding to the four evaluation dimensions: content accessibility, content completeness, content accuracy, and content comprehensibility (using negative scores as an example; these are counted as bonuses when calculating the document score of the reference text fragment corresponding to a positive score, and as deductions when calculating the document score of the reference text fragment corresponding to a negative score). Each sub-dictionary for each evaluation dimension includes at least one reference text fragment and the corresponding evaluation parameters:
[0127] Table 1
[0128]
[0129] Table 2
[0130]
[0131] Table 3
[0132]
[0133] Table 4
[0134]
[0135] Taking content accessibility as an example, the sub-dictionary corresponding to content accessibility includes multiple reference text fragments, such as "no link", "how to obtain", "send to my email", "how to download", and "not found", with corresponding evaluation parameters of 0.8, 0.6, 0.8, 0.7, and 1, respectively. It should be noted that the number and content of the reference text fragments corresponding to the above four evaluation dimensions, as well as the values of the corresponding evaluation parameters, are merely exemplary, and this application embodiment does not impose any limitations on them.
[0136] 404. Determine the evaluation parameters corresponding to the matching text fragments from the evaluation parameter comparison data.
[0137] The management server determines the evaluation parameters corresponding to each matching text segment from the evaluation parameter comparison data, which are used to calculate the score for each matching text segment. The evaluation parameter comparison data can exist independently of the evaluation dictionary, or it can be a part of the evaluation dictionary (in which case, the evaluation dictionary includes multiple reference text segments and their respective evaluation parameters).
[0138] In one possible implementation, the evaluation parameters for matching text fragments can be determined manually or automatically generated based on artificial intelligence or other methods; this application does not impose any limitations on this.
[0139] In one possible implementation, the management server can pre-set evaluation parameters against data using sentiment analysis methods, such as performing sentiment analysis on each reference text segment in dimensions such as sentiment words, degree words, and punctuation marks, thereby setting the evaluation parameters corresponding to each reference text segment in the evaluation dictionary.
[0140] The following explains the methods of sentiment analysis: Taking the reference text fragment "Different from the actual interface" as an example, sentiment analysis can be used to set the evaluation parameter corresponding to "Different from the actual interface," such as 0.6. Taking the reference text fragment "Completely different from the actual interface" as an example, sentiment analysis can be used to set the evaluation parameter corresponding to "Completely different from the actual interface," such as 0.7. Taking the reference text fragment "Completely different from the actual interface!", sentiment analysis can be used to set the evaluation parameter corresponding to "Completely different from the actual interface!", such as 0.8.
[0141] The management server can determine the reference text fragments (including positive and negative text fragments) included in the four evaluation dimensions of the evaluation parameter comparison data based on the aforementioned sentiment analysis method, as well as the evaluation parameters for each reference text fragment (the sentiment score can be mapped to a value between 0 and 1 through normalization, thereby normalizing the evaluation parameters of the reference text fragments). This method allows the management server to closely integrate the evaluation parameters for the reference text fragments with actual business needs, improving the flexibility of evaluation parameter settings.
[0142] 405. Determine the matching information corresponding to the target evaluation dimension based on the matching text fragment and the evaluation parameters corresponding to the matching text fragment.
[0143] 406. Determine the matching result based on the matching information corresponding to multiple evaluation dimensions.
[0144] In steps 405-406 above, taking the target evaluation dimension as an example, the management server combines the matching text fragment and the corresponding evaluation parameters of the matching text fragment as the matching information corresponding to the target evaluation dimension. The management server performs the above processing on each of the multiple evaluation dimensions separately to obtain the matching information corresponding to each evaluation dimension. The management server then combines the matching information corresponding to each of the multiple evaluation dimensions, and uses the combined result as the overall matching result of the evaluation data.
[0145] The following steps, 407-410, will explain how to determine the quality assessment data of the document to be detected based on the matching results:
[0146] 407. Determine the evaluation parameters corresponding to each positive text segment and each negative text segment from the matching information corresponding to the target evaluation dimension.
[0147] 408. Determine positive evaluation parameters based on each positive text segment and the evaluation parameters corresponding to each positive text segment, and determine negative evaluation parameters based on each negative text segment and the evaluation parameters corresponding to each negative text segment.
[0148] In steps 407-408 above, taking the target evaluation dimension as an example, the management server calculates positive evaluation parameters for each positive text segment and its corresponding evaluation parameters, and calculates negative evaluation parameters for each negative text segment and its corresponding evaluation parameters. The positive evaluation parameters for the target evaluation dimension are used to evaluate the overall score increase of the document under test for that target evaluation dimension, while the negative evaluation parameters are used to evaluate the overall score decrease of the document under test for that target evaluation dimension. A higher positive evaluation parameter indicates a better performance of the document under test in that target evaluation dimension, and a lower positive evaluation parameter indicates a worse performance in that dimension; the same applies to negative evaluation parameters. For example, if the target evaluation dimension is ease of understanding, a higher positive evaluation parameter (or a lower negative evaluation parameter) indicates that the document under test is easier for readers to understand; a lower positive evaluation parameter (or a higher negative evaluation parameter) indicates that the description of the document under test is difficult to understand, resulting in a poor user experience.
[0149] In one possible implementation, taking a positive text fragment as an example, the management server can determine the positive evaluation parameters based on the content of the text fragment, the corresponding positive text fragments, the evaluation parameters of the positive text fragments, and their frequency of occurrence. For example, if a positive text fragment includes phrases like "clear structure," "easy to find," and "easy to understand," and the frequencies of occurrence of these positive text fragments are 10, 8, and 4 respectively, and the corresponding evaluation parameters are 0.8, 0.6, and 0.9 respectively, then, based on the text content, the positive evaluation parameter of the positive text fragment is calculated to be 16.4 (the calculation process is: 10*0.8+8*0.6+4*0.9=16.4).
[0150] 409. Determine the quality score corresponding to the target evaluation dimension based on positive and negative evaluation parameters.
[0151] 410. Determine the quality evaluation data of the document to be tested based on the quality scores corresponding to multiple evaluation dimensions.
[0152] In steps 409-410 above, taking the target evaluation dimension as an example, the management server can add the positive and negative evaluation parameters of the target evaluation dimension to obtain the quality score corresponding to the target evaluation dimension. This quality score is used to characterize the overall quality of the document for the target evaluation dimension. The higher the quality score, the better the overall quality of the document; the lower the quality score, the worse the overall quality of the document. For example, when the target evaluation dimension is content accuracy, a higher quality score indicates higher accuracy of the document to be tested; a lower quality score indicates lower accuracy of the document to be tested. Finally, the management server can combine the quality scores corresponding to multiple evaluation dimensions as the quality evaluation data of the document to be tested, thereby judging the document quality from multiple evaluation dimensions, improving the granularity of document evaluation, and thus improving the accuracy of document evaluation.
[0153] In one possible implementation, the information extraction model is obtained by reinforcing the syntactic analysis of a natural language processing model based on evaluation data and labeled data of sample documents. The labeled data includes at least one labeled text segment from the evaluation data of the sample documents. Since current natural language processing (NLP) models provide a two-dimensional linear classification (positive and negative emotions) from a sentiment dimension, sentiment-related words constitute a large proportion, while the documents to be detected are generally user-related after-sales documents. In the evaluation data of the documents to be detected, we need sentences and phrases of irregular behavioral descriptions, not sentiment words. Therefore, this embodiment uses evaluation data and labeled data (evaluation samples) of sample documents to reinforce the above-mentioned syntactic analysis stage, such as reinforcing sentence components like nouns, verbs, adjectives, adverbs, and particles, to obtain the information extraction model, thereby improving the accuracy of the output target text segment.
[0154] Before performing lexical, syntactic, and semantic analysis on the evaluation data based on the information extraction model to obtain at least one target text fragment, the management server can also train the information extraction model. The training process of the information extraction model will be explained below:
[0155] (1) Obtain the evaluation data and annotation data of the sample document. The annotation data includes at least one annotated text fragment of the evaluation data of the sample document.
[0156] The management server can obtain evaluation data from sample documents and annotation data including at least one annotated text fragment. The evaluation data of a sample document can be viewed as a set of evaluation data, and the number of evaluation data pieces in the set can be multiple.
[0157] For example, the sample document could be the user manual for an application product. The evaluation data in the sample document could be text data such as "The path in the first chapter does not match the path in the latest version", "Whether module B still points to the motherboard is not mentioned in the document", "The interface corresponding to function A is completely different from the actual interface", and "Many parameters are written unclearly and their meaning is unclear". The text fragments corresponding to each of the above annotation data could be "path mismatch", "no description in the document", "different from the actual interface", and "unclear meaning".
[0158] (2) Use natural language processing models to extract information from the evaluation data of sample documents to obtain predicted text fragments.
[0159] In this model, the Natural Language Processing (NLP) model is the model prior to reinforcement training. This NLP model can be obtained through a first-stage training process using other training methods, followed by a second-stage training using evaluation and labeled data from sample documents, ultimately resulting in an information extraction model. This method further improves the accuracy of model predictions. The management server uses the NLP model to extract information from the evaluation data of the sample documents, obtaining predicted text fragments. These predicted text fragments can be considered as key text segments with crucial semantic information within the evaluation data of the sample documents. The management server can perform information extraction processing on each evaluation data point in the sample document's evaluation data set separately, obtaining its corresponding predicted text fragment.
[0160] For example, the predicted text fragment extracted from the evaluation data "The path in the first chapter does not match the path in the latest version" could be "Path mismatch". The predicted text fragment extracted from the evaluation data "Does module B still point to the motherboard? The document does not describe this" could be "The document does not describe this". The predicted text fragment extracted from "The interface corresponding to function A is completely different from the actual interface" could be "Different from the actual interface". The predicted text fragment extracted from "Many parameters are written unclearly; I don't understand their meaning" could be "I don't understand their meaning".
[0161] (3) Determine the sentence components of the labeled text fragment and the sentence components of the predicted text fragment. The sentence components of the labeled text fragment include the combination relationship between the parts of speech of each text element in the labeled text fragment, and the sentence components of the predicted text fragment include the combination relationship between the parts of speech of each text element in the predicted text fragment.
[0162] The management server can first determine the sentence components of the annotated text fragment and predict the sentence components of the text fragment. The sentence components include the combination relationship between the parts of speech (such as nouns, verbs, adjectives, adverbs, and particles) of each text element (such as a single word) in the text fragment. For example, the part of speech composition of the text fragment, such as the sentence components of the annotated text fragment "unclear meaning" are "adverb + verb + noun", and there is a temporal relationship between the adverb, verb, and noun.
[0163] (4) The parameters of the natural language processing model are adjusted based on the difference data between the sentence components of the labeled text fragments and the sentence components of the predicted text fragments to obtain the information extraction model.
[0164] In this process, the management server uses the sentence components of the annotated text segments as supervisory information for predicting the sentence components of the predicted text segments. By calculating the difference between the sentence components of the annotated and predicted text segments, the server adjusts the parameters of the natural language processing (NLP) model using this difference data. Through multiple iterations of training, when the output of the trained NLP model reaches the preset accuracy requirement, an information extraction model is obtained. Using this method, this application can, based on the NLP model, perform targeted reinforcement training on the document evaluation lexicon characteristics and the NLP model to ensure the accuracy of the model's output. Based on the model's output, document quality evaluation data can be determined, thereby improving the accuracy of document quality evaluation.
[0165] In one possible implementation, the management server can calculate the loss between the sentence components of the labeled text segment and the sentence components of the predicted text segment, and then use the loss to tune the natural language processing model. The loss can be one or more of the following loss functions: Mean Squared Error Loss, Mean Absolute Error Loss, Huber Loss, Quantile Loss, Cross Entropy Loss, and Hinge Loss; this application does not limit the specific loss function used.
[0166] Please see Figure 5a , Figure 5aThis is a schematic diagram illustrating an reinforcement training method for an NLP model provided in this application embodiment. The middle section shows the sample data used for reinforcement training, such as evaluation data of sample texts. The left side shows the labeled text segments corresponding to the evaluation data of the sample texts, and the right side shows the sentence components corresponding to the labeled text segments. The training process is as follows: First, the evaluation data of the sample texts (e.g., "Many parameters are unclear and I don't understand their meaning") is input into the natural language processing model for processing, resulting in predicted text segments (e.g., "I don't understand the meaning"). Then, sentence components are extracted from both the predicted text segments and the labeled text segments (e.g., "I don't understand the meaning"), resulting in sentence components for the predicted text segments (e.g., "adverb + verb + noun") and the labeled text segments (e.g., "adverb + verb + noun"). The difference data (e.g., cross-entropy loss) between the sentence components of the predicted text segments and the labeled text segments is then calculated to tune the parameters of the natural language processing model. The natural language processing model is iteratively trained using evaluation data from multiple sample texts and corresponding labeled text segments, ultimately yielding an information extraction model. It should be noted that the evaluation data, corresponding labeled text fragments, and sentence components of each sample text listed in the above reinforcement training are merely exemplary, and the embodiments of this application do not impose any limitations on them.
[0167] In one possible implementation, the management server can also output improvement instructions for the document to be inspected. These instructions provide feedback on each positive or negative factor in the document's score, allowing the primary target to understand how to modify the document and providing a reference for document modification and optimization strategies. Therefore, after obtaining the quality evaluation data of the document to be inspected, the management server can also perform the following steps:
[0168] (1) Determine the positive score item of the document based on each positive text fragment and the evaluation parameters corresponding to each positive text fragment.
[0169] (2) Determine the negative score item of the document based on each negative text segment and the evaluation parameters corresponding to each negative text segment.
[0170] (3) Determine the improvement instruction information of the document to be tested based on the positive score items and negative score items of the document. The improvement instruction information is used to instruct the first target user to edit the document to be tested.
[0171] In steps (1)-(3) above, the management server uses a positive text fragment and its corresponding evaluation parameters as a positive score item for a document, resulting in multiple positive score items. Similarly, the management server uses a negative text fragment and its corresponding evaluation parameters as a negative score item for a document, resulting in multiple negative score items. Finally, the management server uses the generated multiple positive and multiple negative score items together as improvement instruction information for the document to be tested. The first target user can be the document author. The improvement instruction information can be used to provide the document author with a document score and a list of additions and subtractions, facilitating document improvement.
[0172] In one possible implementation, the management server can also use only the negative scores of the document as improvement instructions for the document to be tested, so that the first target object (the document author) can make more clear and targeted modifications to the document. Since less data is shown to the first target object, the efficiency of the first target object in making document modifications based on the improvement instructions based on less data can be improved.
[0173] Please see Figure 5b This diagram illustrates a process for generating document improvement instructions and quality evaluation labels according to an embodiment of this application. The management server first obtains the evaluation data set of the document to be tested, then completes scoring and a list of bonus and deduction items across multiple evaluation dimensions (including obtaining target text fragments through information extraction processing, and matching the target text fragments with the evaluation dictionary to obtain matching results). The management server then generates improvement instructions from the list of bonus and deduction items in the matching results and outputs them. Once the document author receives the improvement instructions, they can consider how to improve the document based on these instructions.
[0174] In one possible implementation, the management server can output quality evaluation tags for the document to be tested, allowing document users to quickly and intuitively understand the document's characteristics, its strengths and weaknesses, and thus make targeted use of the document to ensure a better user experience. Therefore, after obtaining the quality evaluation data of the document to be tested, the management server can also perform the following steps:
[0175] (1) Determine the frequency of occurrence of each positive text segment and each negative text segment in the matching text segment based on the content of the text segment.
[0176] (2) Determine the weight parameters of each positive text segment based on the frequency of occurrence and evaluation parameters of each positive text segment in the matched text segment, and determine the weight parameters of each negative text segment based on the frequency of occurrence and evaluation parameters of each negative text segment in the matched text segment.
[0177] In steps (1)-(2) above, the management server divides multiple matching text fragments based on their content, grouping those with the same content into the same group, thereby determining the frequency of occurrence of positive text fragments and negative text fragments corresponding to different content. The management server then calculates the weight parameters of each group of positive text fragments and each group of negative text fragments based on the frequency of occurrence of positive text fragments corresponding to different content and the evaluation parameters. The weight parameters can be regarded as the overall score of one or more positive text fragments (or negative text fragments) corresponding to the same content.
[0178] For example, taking a positive text fragment as an example, the positive text fragment includes phrases such as "clear structure," "easy to find," "easy to understand," "detailed introduction," and "accurate description." The frequencies of these positive text fragments are 10, 8, 4, 7, and 8, respectively, and the corresponding evaluation parameters are 0.8, 0.6, 0.9, 0.8, and 0.8, respectively. Therefore, based on the text content, the weight parameter for the above positive text fragments is calculated as follows: "clear structure" is 8, "easy to find" is 4.8, "easy to understand" is 3.6, "detailed introduction" is 5.6, and "accurate description" is 6.4.
[0179] (3) Determine the quality evaluation label of the document to be detected based on the weight parameters of each positive text segment and each negative text segment. The quality evaluation label is used to instruct the second target user to use the document to be detected based on the quality evaluation label.
[0180] The management server selects a subset of positive and negative text segments that characterize the document's features as quality evaluation labels for the document under test, based on the weight parameters of each positive and negative text segment. For example, it selects at least one positive text segment with a higher weight parameter from multiple positive text segments and at least one negative text segment with a lower weight parameter from multiple negative text segments, using these as the quality evaluation labels for the document under test. The second target user can be a document user, such as a user of after-sales documentation.
[0181] In one possible implementation, the step (3) above, which determines the quality evaluation label of the document to be detected based on the weight parameters of each positive text segment and each negative text segment, can be achieved through the following steps:
[0182] (1) Sort the weight parameters of each positive text segment in descending order to obtain the K positive text segments corresponding to the weight parameters in the first K positions, where K is a positive integer.
[0183] The management server can first sort the weight parameters of each positive text segment in descending order to determine the top K weight parameters, and then determine the positive text segment corresponding to each weight parameter in the top K weight parameters, finally obtaining the K positive text segments corresponding to the top K weight parameters.
[0184] The K positive text segments corresponding to the K weight parameters in the top K positions.
[0185] (2) Sort the weight parameters of each negative text segment in descending order to obtain the L negative text segments corresponding to the weight parameters in the first L positions, where L is a positive integer.
[0186] The management server can first sort the weight parameters of each negative text segment in descending order to determine the top L weight parameters, and then determine the negative text segment corresponding to each weight parameter in the top L weight parameters, finally obtaining the L negative text segments corresponding to the top L weight parameters.
[0187] (3) Determine the quality evaluation labels of the document to be tested based on K positive text fragments and L negative text fragments.
[0188] The management server combines K positive text fragments and L negative text fragments to obtain corresponding text fragment combinations, which are then used as quality evaluation tags for the document under test. For example, each positive text fragment in the K positive text fragments serves as a quality evaluation tag for the document under test, and each negative text fragment in the L negative text fragments serves as a quality evaluation tag for the document under test, ultimately resulting in K+L quality evaluation tags for the document under test.
[0189] It should be noted that the number of positive and negative text snippets can be the same, such as 3 for both; or the number can differ, such as 3 for positive and 10 for negative. Adjusting the number of positive and negative text snippets allows for more targeted presentation of information to the second target user. For example, showing more negative text snippets can guide the user to focus on the content quality issues indicated by the negative text snippets when using the document.
[0190] It should be noted that the number of positive text segments can also be 0. In this case, the management server only displays the quality evaluation tags of the document to be tested corresponding to L negative text segments to the second target user. The number of negative text segments can also be 0. In this case, the management server only displays the quality evaluation tags of the document to be tested corresponding to K positive text segments to the second target user. This application embodiment does not limit the number of positive and negative text segments.
[0191] For example, taking positive text fragments as an example, the management server sorts the weight parameters of five positive text fragments, namely "clear structure", "easy to find", "easy to understand", "detailed introduction" and "accurate description", in descending order to obtain the sorting result (8>6.4>5.6>4.8>3.6). Then, the positive text fragments corresponding to the top K weight parameters (e.g., K is 3, including weight parameters 8, 6.4, and 5.6), such as "clear structure", "accurate description" and "detailed introduction", are selected as the quality evaluation tags of the document to be tested.
[0192] Please see Figure 5c This figure is a schematic diagram of a quality evaluation label provided in an embodiment of this application. The quality evaluation label includes positive labels and negative labels. For example, positive labels include the three positive text segments with the highest weight parameters, namely "clear structure," "accurate description," and "detailed introduction." Negative labels include the three negative text segments with the highest weight parameters, namely "blurry image," "incorrect specifications," and "how to configure." It should be noted that each quality evaluation label listed above is merely exemplary, and this embodiment of the application does not impose any limitations on it.
[0193] The overall process of document quality inspection provided in the embodiments of this application will be described below. Please refer to... Figure 5d This figure is a schematic diagram of a document quality evaluation process provided in an embodiment of this application, which mainly includes the following parts:
[0194] Document quality evaluation indicators (evaluation dimensions): Document quality is divided into four dimensions: accessibility, completeness, accuracy, and comprehensibility.
[0195] Evaluation Dictionary: This dictionary summarizes evaluation criteria for product documentation based on common evaluation dimensions.
[0196] Word segmentation: The evaluation data of the document to be tested is segmented into words.
[0197] Natural Language Processing: Through self-developed language processing, we enhance the training of natural language processing models (NLP models) to better identify verbs, nouns, adverbs, particles, adjectives, and other words in technical documents.
[0198] Evaluation and scoring model: Evaluate the natural language processed text (such as target text fragments) according to the dimensions of document quality evaluation indicators.
[0199] Scoring: Provides a score for the document in each evaluation dimension.
[0200] Improve instruction information: Use forms to provide feedback on the document's strengths and weaknesses, allowing authors to understand how to revise the document.
[0201] Quality evaluation labels: Categorize by positive and negative, and add TopK evaluation labels at the end of the document according to the rule of score * frequency.
[0202] Specifically, document quality inspection can include the following steps:
[0203] 1. The management server obtains the evaluation data (such as user evaluation text) of the document to be tested.
[0204] 2. The management server performs word segmentation on the evaluation data.
[0205] 3. The management server performs natural language processing on the segmented results.
[0206] 4. The management server uses an evaluation scoring model to score the results of natural language processing.
[0207] 5. The management server provides an overall score for the document to be tested (including four evaluation dimensions: content accessibility, content completeness, content accuracy, and content comprehensibility, each with its corresponding quality score).
[0208] 6. The management server provides improvement instructions for the document to be tested, making it easier for the author to improve it.
[0209] 7. The management server provides quality evaluation tags for the documents to be tested, making it easier for users to use the documents.
[0210] In steps 2-5, the results of natural language processing are matched from four evaluation dimensions using an evaluation dictionary to obtain matching text fragments. Then, the quality scores corresponding to the four evaluation dimensions are calculated using the matching text fragments and their evaluation parameters.
[0211] In this embodiment, positive evaluation parameters are determined by each positive text segment and its corresponding evaluation parameters, and negative evaluation parameters are determined by each negative text segment and its corresponding evaluation parameters. Then, a quality score corresponding to each evaluation dimension is calculated using both the positive and negative evaluation parameters. Finally, the quality scores corresponding to each evaluation dimension are combined as the quality evaluation data of the document to be tested, thereby judging the document quality from multiple dimensions and improving the accuracy of document evaluation. Furthermore, improvement instructions for the document to be tested can be output, providing feedback on each point added or subtracted, so that the primary target audience understands how to modify the document and provides a reference for document modification and optimization strategies. Quality evaluation tags can also be output, allowing document users to quickly and intuitively understand the document's characteristics for targeted use and ensuring a good user experience.
[0212] Meanwhile, because sentiment-related words constitute a large proportion of the natural language processing models in related technologies, their performance is poor in business environments such as after-sales documents and product usage documents (where the evaluation data of the document to be tested needs to be irregular behavioral descriptions rather than sentiment words). Therefore, this application embodiment trains the natural language processing model using evaluation data and labeled data from sample documents, such as strengthening the training of sentence components like nouns, verbs, adjectives, adverbs, and particles, to obtain an information extraction model, thereby improving the accuracy of the target text fragments output by the model. Determining the document quality evaluation data based on accurate target text fragments improves the accuracy of document quality evaluation.
[0213] Furthermore, this application proposes a sentiment analysis-based method to determine the reference text fragments (including positive and negative text fragments) included in the four evaluation dimensions of the evaluation parameter comparison data, as well as the evaluation parameters for each reference text fragment. The sentiment score can be mapped to a value between 0 and 1 through normalization. Using this method, the evaluation parameters corresponding to the reference text fragments can be set in close conjunction with actual business conditions, improving the flexibility of evaluation parameter setting.
[0214] Please see Figure 6 This application's embodiments are based on Figure 1 The diagram shown illustrates another document quality assessment method provided by the document quality assessment system. This method can be applied to... Figure 1 The management server 101 in the system may specifically include the following steps:
[0215] 601. Obtain the evaluation data of the document to be tested.
[0216] 602. Perform natural language processing on the evaluation data to obtain the target text fragment.
[0217] 603. Match and score the target text fragment with the evaluation dictionary.
[0218] 604. Determine the quality scores and bonus / penalty items corresponding to the four evaluation dimensions.
[0219] The scoring criteria include both positive and negative points for the document.
[0220] 605. Feedback on the quality scores and bonus / penalty items corresponding to the four evaluation dimensions.
[0221] 606. Generate corresponding quality evaluation tags based on different evaluation dimensions of the document to be tested.
[0222] The quality evaluation tags are used to facilitate document users in obtaining and using the documents.
[0223] In one possible implementation, the management server can generate corresponding quality evaluation tags for each evaluation dimension. For example, the management server can generate a positive quality evaluation tag (e.g., "clear structure") and a negative quality evaluation tag (e.g., "no links") for content accessibility; a positive quality evaluation tag (e.g., "quite comprehensive") and a negative quality evaluation tag (e.g., "what are the specifications") for content completeness; a positive quality evaluation tag (e.g., "accurate description") and a negative quality evaluation tag (e.g., "different from the actual interface") for content accuracy; and a positive quality evaluation tag (e.g., "easy to understand") and a negative quality evaluation tag (e.g., "what does it mean") for content comprehensibility.
[0224] It should be noted that the specific implementation of steps 601-606 can be found in the detailed description of the relevant steps in the foregoing embodiments, and will not be repeated here.
[0225] Please see Figure 7This diagram illustrates a document quality evaluation process provided in an embodiment of this application. From a user's perspective, the overall workflow is as follows: Users can refer to website operators, document authors, document users, etc. Website operators can understand the document scores for each product; document authors can understand the quality scores and improvement instructions for their documents in each evaluation dimension; and document users can understand the strengths and weaknesses and characteristics of documents through document quality evaluation tags, making their use of documents more informed. Specifically, users can obtain the document evaluation data set (either through organization or automatic reading by the system) through the management server, and then input the evaluation data set into the document quality evaluation system in this embodiment of the application (including the functional modules corresponding to the document quality evaluation methods mentioned in each embodiment) for document quality evaluation processing. Finally, the document quality evaluation system provides feedback on the document score (such as quality evaluation data).
[0226] The document scoring includes quality scores for four evaluation dimensions. For the score related to accessibility, the evaluation dictionary is matched using a list of bonus items (e.g., "clear structure," "easy to search") and a list of deduction items (e.g., "no links," "how to download," "how to obtain"). For the score related to completeness, the evaluation dictionary is matched using a list of bonus items (e.g., "quite comprehensive," "complete content," "detailed introduction") and a list of deduction items (e.g., "what are the specifications," "how to configure," "no guidance"). For the score related to correctness, the evaluation dictionary is matched using a list of bonus items (e.g., "accurate description," "good content") and a list of deduction items (e.g., "different from the actual interface," "incorrect content," "incorrect specifications"). For the score related to comprehensibility, the evaluation dictionary is matched using a list of bonus items (e.g., "easy to understand," "straightforward content," "vivid") and a list of deduction items (e.g., "what does it mean," "don't understand the meaning," "blurry images," "unclear description"). The document quality score and a list of bonus and deduction points were obtained using the above method. Finally, improvement instructions were output to the document author, and quality evaluation tags were output to the document user. It should be noted that the list of bonus and deduction items used in the evaluation dictionary matching process for each evaluation dimension listed above is only exemplary, and this application embodiment does not limit it.
[0227] It is worth noting that the document quality evaluation method provided in this application can be applied to document quality evaluation processing for various document types, such as product documents, after-sales documents, and questionnaires. This application does not limit the specific document types to which it can be applied. Taking questionnaires as an example, since document evaluation and questionnaires are very similar, this method can also be applied to automatically extract user opinions from questionnaires. By performing relevant analysis on user opinions, the efficiency of questionnaire analysis can be improved.
[0228] It is understood that in the specific implementation of this application, data such as the document to be tested and the corresponding evaluation data, the sample document and the corresponding evaluation data, each reference text fragment included in the evaluation dictionary, and annotation data are involved. When the above embodiments of this application are applied to specific products or technologies, permission or consent from relevant individuals, units or organizations is required, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.
[0229] Please see Figure 8 This is a schematic diagram of a document quality assessment device according to an embodiment of this application. The device includes:
[0230] The acquisition module 801 is used to acquire the evaluation data of the document to be tested;
[0231] Processing module 802 is used to obtain at least one target text fragment based on the evaluation data;
[0232] The processing module 802 is further configured to match the at least one target text fragment with an evaluation dictionary from multiple evaluation dimensions to obtain a matching result. The matching result includes matching information corresponding to the multiple evaluation dimensions. The matching information corresponding to each evaluation dimension includes a matching text fragment and an evaluation parameter corresponding to the matching text fragment. The evaluation dictionary includes at least one reference text fragment. The evaluation parameter is used to indicate the score of the advantageous or disadvantageous features of the document content described by each matching text fragment.
[0233] Evaluation module 803 is used to determine the quality evaluation data of the document to be detected based on the matching results; wherein, the quality evaluation data includes quality scores corresponding to the multiple evaluation dimensions.
[0234] Optionally, the evaluation dictionary includes a sub-dictionary corresponding to each of the evaluation dimensions;
[0235] When the processing module 802 matches the at least one target text fragment with the evaluation dictionary from multiple evaluation dimensions to obtain a matching result, it is specifically used for:
[0236] The at least one target text fragment is matched with the reference text fragments in the target sub-dictionary corresponding to the target evaluation dimension to obtain a matching text fragment that matches the at least one target text fragment; the matching text fragment includes positive text fragments and negative text fragments; wherein, the target evaluation dimension is any one of the plurality of evaluation dimensions, the positive text fragment is used to describe the advantageous features of the content of the document to be detected, and the negative text fragment is used to describe the disadvantageous features of the content of the document to be detected;
[0237] The evaluation parameters corresponding to the matched text fragment are determined from the evaluation parameter comparison data;
[0238] Based on the matched text fragment and the corresponding evaluation parameters, determine the matching information corresponding to the target evaluation dimension;
[0239] The matching result is determined based on the matching information corresponding to the multiple evaluation dimensions.
[0240] Optionally, when the evaluation module 803 is used to determine the quality evaluation data of the document to be detected based on the matching result, it is specifically used for:
[0241] The evaluation parameters corresponding to each positive text segment and the evaluation parameters corresponding to each negative text segment are determined from the matching information corresponding to the target evaluation dimension.
[0242] The positive evaluation parameters are determined based on each of the positive text segments and the evaluation parameters corresponding to each of the positive text segments;
[0243] The negative evaluation parameters are determined based on each negative text segment and the evaluation parameters corresponding to each negative text segment;
[0244] The quality score corresponding to the target evaluation dimension is determined based on the positive evaluation parameters and the negative evaluation parameters;
[0245] The quality evaluation data of the document to be tested is determined based on the quality scores corresponding to the multiple evaluation dimensions.
[0246] Optionally, when the processing module 802 is used to obtain at least one target text fragment based on the evaluation data, it is specifically used to:
[0247] Based on the information extraction model, lexical analysis, syntactic analysis and semantic analysis are performed on the evaluation data to obtain at least one target text fragment;
[0248] The information extraction model is obtained by strengthening the training of the natural language processing model through the syntactic analysis based on the evaluation data and labeled data of the sample documents. The labeled data includes at least one labeled text fragment of the evaluation data of the sample documents.
[0249] Optionally, before performing lexical, syntactic, and semantic analysis on the evaluation data based on the information extraction model to obtain at least one target text fragment, the processing module 802 is further configured to:
[0250] Obtain evaluation data and annotation data of sample documents, wherein the annotation data includes at least one annotated text fragment of the evaluation data of the sample documents;
[0251] Information is extracted from the evaluation data of the sample documents using a natural language processing model to obtain predicted text fragments;
[0252] The sentence components of the labeled text fragment and the sentence components of the predicted text fragment are determined. The sentence components of the labeled text fragment include the combination relationship between the parts of speech of each text element in the labeled text fragment, and the sentence components of the predicted text fragment include the combination relationship between the parts of speech of each text element in the predicted text fragment.
[0253] The parameters of the natural language processing model are adjusted based on the difference data between the sentence components of the labeled text segment and the sentence components of the predicted text segment to obtain the information extraction model.
[0254] Optionally, the processing module 802 is further configured to:
[0255] The document's positive score is determined based on each of the positive text segments and the evaluation parameters corresponding to each of the positive text segments;
[0256] The document's negative score item is determined based on each negative text segment and the evaluation parameters corresponding to each negative text segment;
[0257] Improvement indication information for the document to be tested is determined based on the document's positive score and negative score, and the improvement indication information is used to instruct the first target user to edit the document to be tested.
[0258] Optionally, the processing module 802 is further configured to:
[0259] The frequency of occurrence of each positive text segment and the frequency of occurrence of each negative text segment in the matched text segment are determined based on the content of the text segment.
[0260] The weight parameter of each positive text segment is determined based on the occurrence frequency and evaluation parameter of each positive text segment in the matched text segment; the weight parameter of each negative text segment is determined based on the occurrence frequency and evaluation parameter of each negative text segment in the matched text segment.
[0261] The quality evaluation label of the document to be detected is determined based on the weight parameters of each positive text segment and the weight parameters of each negative text segment. The quality evaluation label is used to instruct a second target user to use the document to be detected based on the quality evaluation label.
[0262] Optionally, when the processing module 802 determines the quality evaluation label of the document to be detected based on the weight parameters of each positive text segment and the weight parameters of each negative text segment, it is specifically used for:
[0263] The weight parameters of each positive text segment are sorted in descending order to obtain the K positive text segments corresponding to the weight parameters in the first K positions, where K is a positive integer;
[0264] The weight parameters of each negative text segment are sorted in descending order to obtain the L negative text segments corresponding to the weight parameters in the first L positions, where L is a positive integer;
[0265] The quality evaluation tags of the document to be detected are determined based on the K positive text segments and the L negative text segments.
[0266] Optionally, the multiple evaluation dimensions include content accessibility, content completeness, content accuracy, and content comprehensibility; the content accessibility is used to indicate the ease of obtaining the content in the document to be tested, the content completeness is used to indicate the completeness of the content in the document to be tested, the content accuracy is used to indicate the accuracy of the content in the document to be tested, and the content comprehensibility is used to indicate the ease of understanding the content in the document to be tested.
[0267] It should be noted that the functions of each functional module of the document quality evaluation device in this application embodiment can be specifically implemented according to the methods in the above method embodiments. The specific implementation process can be referred to the relevant descriptions in the above method embodiments, which will not be repeated here.
[0268] Please see Figure 9 This is a schematic diagram of a computing device according to an embodiment of this application. The computing device described in this embodiment can correspond to the management server mentioned above. The computing device includes a processor 901, a memory 902, and a communication interface 903. The processor 901, the memory 902, and the communication interface 903 can exchange data.
[0269] The memory 902 may include volatile memory, such as random-access memory (RAM); the memory 902 may also include non-volatile memory, such as flash memory, solid-state drive (SSD), etc.; the memory 902 may also include a combination of the above types of memory.
[0270] The processor 901 may be a central processing unit (CPU). In one embodiment, the processor 901 may also be a graphics processing unit (GPU). The processor 901 may also be a combination of a CPU and a GPU.
[0271] In one embodiment, the memory 902 is used to store program instructions, and the processor 901 can call the program instructions to perform the following operations:
[0272] Obtain evaluation data for the document to be tested;
[0273] Processing data to obtain at least one target text fragment;
[0274] The at least one target text fragment is matched with an evaluation dictionary from multiple evaluation dimensions to obtain a matching result. The matching result includes matching information corresponding to the multiple evaluation dimensions. The matching information corresponding to each evaluation dimension includes a matched text fragment and an evaluation parameter corresponding to the matched text fragment. The evaluation dictionary includes at least one reference text fragment. The evaluation parameter is used to indicate the score of the strength or weakness features of the document content described by each matched text fragment.
[0275] The quality evaluation data of the document to be detected is determined based on the matching results; wherein, the quality evaluation data includes quality scores corresponding to the multiple evaluation dimensions.
[0276] Optionally, the evaluation dictionary includes a sub-dictionary corresponding to each of the evaluation dimensions;
[0277] When the processor 901 matches the at least one target text fragment with the evaluation dictionary from multiple evaluation dimensions to obtain a matching result, it is specifically used for:
[0278] The at least one target text fragment is matched with the reference text fragments in the target sub-dictionary corresponding to the target evaluation dimension to obtain a matching text fragment that matches the at least one target text fragment; the matching text fragment includes positive text fragments and negative text fragments; wherein, the target evaluation dimension is any one of the plurality of evaluation dimensions, the positive text fragment is used to describe the advantageous features of the content of the document to be detected, and the negative text fragment is used to describe the disadvantageous features of the content of the document to be detected;
[0279] The evaluation parameters corresponding to the matched text fragment are determined from the evaluation parameter comparison data;
[0280] Based on the matched text fragment and the corresponding evaluation parameters, determine the matching information corresponding to the target evaluation dimension;
[0281] The matching result is determined based on the matching information corresponding to the multiple evaluation dimensions.
[0282] Optionally, when the processor 901 determines the quality assessment data of the document to be detected based on the matching result, it is specifically used for:
[0283] The evaluation parameters corresponding to each positive text segment and the evaluation parameters corresponding to each negative text segment are determined from the matching information corresponding to the target evaluation dimension.
[0284] The positive evaluation parameters are determined based on each of the positive text segments and the evaluation parameters corresponding to each of the positive text segments;
[0285] The negative evaluation parameters are determined based on each negative text segment and the evaluation parameters corresponding to each negative text segment;
[0286] The quality score corresponding to the target evaluation dimension is determined based on the positive evaluation parameters and the negative evaluation parameters;
[0287] The quality evaluation data of the document to be tested is determined based on the quality scores corresponding to the multiple evaluation dimensions.
[0288] Optionally, when the processor 901 is used to obtain at least one target text fragment based on the evaluation data, it is specifically used to:
[0289] Based on the information extraction model, lexical analysis, syntactic analysis and semantic analysis are performed on the evaluation data to obtain at least one target text fragment;
[0290] The information extraction model is obtained by strengthening the training of the natural language processing model through the syntactic analysis based on the evaluation data and labeled data of the sample documents. The labeled data includes at least one labeled text fragment of the evaluation data of the sample documents.
[0291] Optionally, before performing lexical, syntactic, and semantic analysis on the evaluation data based on the information extraction model to obtain at least one target text fragment, the processor 901 is further configured to:
[0292] Obtain evaluation data and annotation data of sample documents, wherein the annotation data includes at least one annotated text fragment of the evaluation data of the sample documents;
[0293] Information is extracted from the evaluation data of the sample documents using a natural language processing model to obtain predicted text fragments;
[0294] The sentence components of the labeled text fragment and the sentence components of the predicted text fragment are determined. The sentence components of the labeled text fragment include the combination relationship between the parts of speech of each text element in the labeled text fragment, and the sentence components of the predicted text fragment include the combination relationship between the parts of speech of each text element in the predicted text fragment.
[0295] The parameters of the natural language processing model are adjusted based on the difference data between the sentence components of the labeled text segment and the sentence components of the predicted text segment to obtain the information extraction model.
[0296] Optionally, the processor 901 is further configured to:
[0297] The document's positive score is determined based on each of the positive text segments and the evaluation parameters corresponding to each of the positive text segments;
[0298] The document's negative score item is determined based on each negative text segment and the evaluation parameters corresponding to each negative text segment;
[0299] Improvement indication information for the document to be tested is determined based on the document's positive score and negative score, and the improvement indication information is used to instruct the first target user to edit the document to be tested.
[0300] Optionally, the processor 901 is further configured to:
[0301] The frequency of occurrence of each positive text segment and the frequency of occurrence of each negative text segment in the matched text segment are determined based on the content of the text segment.
[0302] The weight parameter of each positive text segment is determined based on the occurrence frequency and evaluation parameter of each positive text segment in the matched text segment; the weight parameter of each negative text segment is determined based on the occurrence frequency and evaluation parameter of each negative text segment in the matched text segment.
[0303] The quality evaluation label of the document to be detected is determined based on the weight parameters of each positive text segment and the weight parameters of each negative text segment. The quality evaluation label is used to instruct a second target user to use the document to be detected based on the quality evaluation label.
[0304] Optionally, when determining the quality evaluation label of the document to be detected based on the weight parameters of each positive text segment and the weight parameters of each negative text segment, the processor 901 is specifically used for:
[0305] The weight parameters of each positive text segment are sorted in descending order to obtain the K positive text segments corresponding to the weight parameters in the first K positions, where K is a positive integer;
[0306] The weight parameters of each negative text segment are sorted in descending order to obtain the L negative text segments corresponding to the weight parameters in the first L positions, where L is a positive integer;
[0307] The quality evaluation tags of the document to be detected are determined based on the K positive text segments and the L negative text segments.
[0308] Optionally, the multiple evaluation dimensions include content accessibility, content completeness, content accuracy, and content comprehensibility; the content accessibility is used to indicate the ease of obtaining the content in the document to be tested, the content completeness is used to indicate the completeness of the content in the document to be tested, the content accuracy is used to indicate the accuracy of the content in the document to be tested, and the content comprehensibility is used to indicate the ease of understanding the content in the document to be tested.
[0309] In specific implementation, the processor 901, memory 902, and communication interface 903 described in the embodiments of this application can execute the embodiments of this application. Figure 2 , Figure 4 and Figure 6 The implementation methods described in the relevant embodiments of the provided method can also be used to execute the embodiments of this application. Figure 8 The implementation methods described in the relevant embodiments of the provided device will not be repeated here.
[0310] Those skilled in the art will also understand that the various illustrative logical blocks and steps listed in the embodiments of this application can be implemented by electronic hardware, computer software, or a combination of both. Whether such functionality is implemented through hardware or software depends on the specific application and the overall system design requirements. Those skilled in the art can implement the described functionality using various methods for each specific application, but such implementation should not be construed as exceeding the scope of protection of the embodiments of this application.
[0311] This application also provides a computer-readable storage medium having a computer program stored thereon, the computer program including program instructions that, when executed by a computer, implement the functions of any of the above method embodiments.
[0312] The aforementioned computer-readable storage media include, but are not limited to, flash memory, hard disk, and solid-state drive.
[0313] This application also provides a computer program product that, when executed by a computer, implements the functions of any of the above method embodiments.
[0314] Those skilled in the art will also understand that the various illustrative logical blocks and steps listed in the embodiments of this application can be implemented by electronic hardware, computer software, or a combination of both. Whether such functionality is implemented through hardware or software depends on the specific application and the overall system design requirements. Those skilled in the art can use various methods to implement the described functionality for corresponding applications, but such implementation should not be construed as exceeding the scope of protection of the embodiments of this application.
[0315] The solutions described in this application can be implemented in various ways. For example, these technologies can be implemented in hardware, software, or a combination of hardware. For hardware implementation, the processing unit for executing these technologies at a communication device (e.g., access network equipment, capability reduction terminal equipment, core network equipment, or chip) can be implemented in one or more general-purpose processors, digital signal processors (DSPs), digital signal processing devices, application-specific integrated circuits (ASICs), programmable logic devices, field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or any combination thereof. The general-purpose processor can be a microprocessor; alternatively, it can also be any conventional processor, controller, microcontroller, or state machine. The processor can also be implemented through a combination of computing devices, such as a digital signal processor and a microprocessor, multiple microprocessors, one or more microprocessors combined with a digital signal processor core, or any other similar configuration.
[0316] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be an apparatus that implements the functions of the capability-reduced terminal device, the access network device corresponding to the second cell, the access network device corresponding to the first cell, and the core network device serving the capability-reduced terminal device in any of the above embodiments. For example, the apparatus can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable apparatus. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another.
[0317] In this application, the use of singular pronouns to denote "one or more" rather than "one and only one," unless otherwise specified. In this application, unless otherwise specified, "at least one" is intended to mean "one or more," and "more than" is intended to mean "two or more."
[0318] Furthermore, the terms "system" and "network" are often used interchangeably in this paper. The term "and / or" in this paper is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three cases: A exists alone, A and B exist simultaneously, and B exists alone. Here, A can be singular or plural, and B can be singular or plural.
[0319] In this application, presets (such as preset sequences) can be understood as definitions, pre-defined, stored, pre-stored, pre-negotiated, pre-configured, solidified, or pre-burned.
[0320] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.
[0321] The same or similar parts between the various embodiments in this application can be referred to mutually. In the various embodiments of this application, and in the various implementation methods / methods / implementations within each embodiment, unless otherwise specified or logically conflicting, the terminology and / or descriptions between different embodiments and between the various implementation methods / methods / implementations within each embodiment are consistent and can be mutually referenced. The technical features in different embodiments and the various implementation methods / methods / implementations within each embodiment can be combined according to their inherent logical relationships to form new embodiments, implementation methods, methods, or implementation approaches. The above-described embodiments of this application do not constitute a limitation on the scope of protection of this application.
Claims
1. A document quality evaluation method, characterized in that, The method includes: Obtain evaluation data for the document to be tested; At least one target text fragment is obtained based on the evaluation data; The at least one target text fragment is matched with an evaluation dictionary from multiple evaluation dimensions to obtain a matching result. The matching result includes matching information corresponding to the multiple evaluation dimensions. Each evaluation dimension's matching information includes a matched text fragment and evaluation parameters corresponding to that matching text fragment. The evaluation dictionary includes at least one reference text fragment. The evaluation parameters indicate the scores of the strengths or weaknesses of the document content described by each matched text fragment. The multiple evaluation dimensions include content accessibility, content completeness, content accuracy, and content comprehensibility. Content accessibility indicates the ease of accessing the content in the document to be tested; content completeness indicates the completeness of the content in the document to be tested; content accuracy indicates the accuracy of the content in the document to be tested; and content comprehensibility indicates the ease of understanding the content in the document to be tested. The quality evaluation data of the document to be detected is determined based on the matching results; wherein, the quality evaluation data includes quality scores corresponding to the multiple evaluation dimensions; The evaluation dictionary includes a sub-dictionary corresponding to each of the evaluation dimensions; The matching of the at least one target text fragment with the evaluation dictionary from multiple evaluation dimensions to obtain the matching result includes: The at least one target text fragment is matched with the reference text fragments in the target sub-dictionary corresponding to the target evaluation dimension to obtain a matching text fragment that matches the at least one target text fragment; the matching text fragment includes positive text fragments and negative text fragments; wherein, the target evaluation dimension is any one of the plurality of evaluation dimensions, the positive text fragment is used to describe the advantageous features of the content of the document to be detected, and the negative text fragment is used to describe the disadvantageous features of the content of the document to be detected; The evaluation parameters corresponding to the matching text fragment are determined from the evaluation parameter comparison data, and the matching information corresponding to the target evaluation dimension is determined based on the matching text fragment and the evaluation parameters corresponding to the matching text fragment. The matching result is determined based on the matching information corresponding to the multiple evaluation dimensions; The method further includes: The frequency of occurrence of each positive text segment and the frequency of occurrence of each negative text segment in the matched text segment are determined based on the content of the text segment. The weight parameter of each positive text segment is determined based on the occurrence frequency and evaluation parameter of each positive text segment in the matched text segment; the weight parameter of each negative text segment is determined based on the occurrence frequency and evaluation parameter of each negative text segment in the matched text segment. The quality evaluation label of the document to be detected is determined based on the weight parameters of each positive text segment and the weight parameters of each negative text segment. The quality evaluation label is used to instruct a second target user to use the document to be detected based on the quality evaluation label.
2. The method according to claim 1, characterized in that, The step of determining the quality evaluation data of the document to be detected based on the matching result includes: The evaluation parameters corresponding to each positive text segment and the evaluation parameters corresponding to each negative text segment are determined from the matching information corresponding to the target evaluation dimension. The positive evaluation parameters are determined based on each of the positive text segments and the evaluation parameters corresponding to each of the positive text segments; The negative evaluation parameters are determined based on each negative text segment and the evaluation parameters corresponding to each negative text segment; The quality score corresponding to the target evaluation dimension is determined based on the positive evaluation parameters and the negative evaluation parameters; The quality evaluation data of the document to be tested is determined based on the quality scores corresponding to the multiple evaluation dimensions.
3. The method according to claim 1 or 2, characterized in that, The process of obtaining at least one target text fragment based on the evaluation data includes: Based on the information extraction model, lexical analysis, syntactic analysis and semantic analysis are performed on the evaluation data to obtain at least one target text fragment; The information extraction model is obtained by strengthening the training of the natural language processing model through the syntactic analysis based on the evaluation data and labeled data of the sample documents. The labeled data includes at least one labeled text fragment of the evaluation data of the sample documents.
4. The method according to claim 3, characterized in that, Before performing lexical, syntactic, and semantic analysis on the evaluation data based on the information extraction model to obtain at least one target text fragment, the method further includes: Obtain evaluation data and annotation data of sample documents, wherein the annotation data includes at least one annotated text fragment of the evaluation data of the sample documents; Information is extracted from the evaluation data of the sample documents using a natural language processing model to obtain predicted text fragments; The sentence components of the labeled text fragment and the sentence components of the predicted text fragment are determined. The sentence components of the labeled text fragment include the combination relationship between the parts of speech of each text element in the labeled text fragment, and the sentence components of the predicted text fragment include the combination relationship between the parts of speech of each text element in the predicted text fragment. The parameters of the natural language processing model are adjusted based on the difference data between the sentence components of the labeled text segment and the sentence components of the predicted text segment to obtain the information extraction model.
5. The method according to claim 1 or 2, characterized in that, The method further includes: The document's positive score is determined based on each of the positive text segments and the evaluation parameters corresponding to each of the positive text segments; The document's negative score item is determined based on each negative text segment and the evaluation parameters corresponding to each negative text segment; Improvement indication information for the document to be tested is determined based on the document's positive score and negative score, and the improvement indication information is used to instruct the first target user to edit the document to be tested.
6. The method according to claim 5, characterized in that, The step of determining the quality evaluation label of the document to be detected based on the weight parameters of each positive text segment and the weight parameters of each negative text segment includes: The weight parameters of each positive text segment are sorted in descending order to obtain the K positive text segments corresponding to the weight parameters in the first K positions, where K is a positive integer; The weight parameters of each negative text segment are sorted in descending order to obtain the L negative text segments corresponding to the weight parameters in the first L positions, where L is a positive integer; The quality evaluation labels of the document to be detected are determined based on the K positive text fragments and the L negative text fragments.
7. A document quality evaluation device, characterized in that, The device includes: The acquisition module is used to acquire evaluation data for the document to be tested; The processing module is used to obtain at least one target text fragment based on the evaluation data; The processing module is further configured to match the at least one target text fragment with an evaluation dictionary from multiple evaluation dimensions to obtain a matching result. The matching result includes matching information corresponding to the multiple evaluation dimensions. Each evaluation dimension's matching information includes a matched text fragment and evaluation parameters corresponding to that matching text fragment. The evaluation dictionary includes at least one reference text fragment. The evaluation parameters indicate the scores of the strengths or weaknesses of the document content described by each matched text fragment. The multiple evaluation dimensions include content accessibility, content completeness, content accuracy, and content comprehensibility. Content accessibility indicates the ease of accessing the content in the document to be detected, content completeness indicates the completeness of the content in the document to be detected, content accuracy indicates the accuracy of the content in the document to be detected, and content comprehensibility indicates the ease of understanding the content in the document to be detected. An evaluation module is used to determine the quality evaluation data of the document to be detected based on the matching results; wherein, the quality evaluation data includes quality scores corresponding to the multiple evaluation dimensions; The evaluation dictionary includes a sub-dictionary corresponding to each evaluation dimension; the processing module is specifically used for: The at least one target text fragment is matched with reference text fragments in the target sub-dictionary corresponding to the target evaluation dimension to obtain a matching text fragment that matches the at least one target text fragment; the matching text fragment includes positive text fragments and negative text fragments; wherein, the target evaluation dimension is any one of the plurality of evaluation dimensions, the positive text fragment is used to describe the advantageous features of the content of the document to be detected, and the negative text fragment is used to describe the disadvantageous features of the content of the document to be detected; the evaluation parameters corresponding to the matching text fragment are determined from the evaluation parameter comparison data, and the matching information corresponding to the target evaluation dimension is determined based on the matching text fragment and the evaluation parameters corresponding to the matching text fragment; the matching result is determined based on the matching information corresponding to the plurality of evaluation dimensions; The processing module is further configured to determine the frequency of occurrence of each positive text segment and each negative text segment in the matched text segment based on the content of the text segment; determine the weight parameter of each positive text segment based on the frequency of occurrence of each positive text segment and the evaluation parameter; determine the weight parameter of each negative text segment based on the frequency of occurrence of each negative text segment and the evaluation parameter; and determine the quality evaluation tag of the document to be detected based on the weight parameter of each positive text segment and the weight parameter of each negative text segment, wherein the quality evaluation tag is used to instruct a second target user to use the document to be detected based on the quality evaluation tag.
8. A computing device, characterized in that, The computing device includes a processor and a memory; the processor and the memory are electrically connected, wherein the memory is used to store computer program instructions; the processor is used to execute the program instructions so that the computing device performs the document quality evaluation method as described in any one of claims 1-6.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that are executed by a processor to perform the document quality evaluation method as described in any one of claims 1-6.
10. A computer program product, characterized in that, It includes a computer program that, when executed by a computer processor, implements the document quality evaluation method as described in any one of claims 1-6.
Citation Information
Patent Citations
Auxiliary bid evaluation method and system based on text mining
CN112632228A
Public opinion information intelligent processing method and device, and storage medium
CN114090756A
