Sentence evaluation method and apparatus

Abstract

The application relates to a sentence evaluation method and device. The sentence evaluation method comprises the following steps: obtaining first text coding information of a first sentence to be evaluated; performing syntax dependency relationship analysis and part-of-speech analysis on the first sentence to obtain a first syntax dependency sequence and a first part-of-speech sequence; determining first syntax information for quality evaluation of the first sentence based on the first text coding information, the first syntax dependency sequence and the first part-of-speech sequence; and performing quality evaluation processing on the first sentence based on the first syntax information to obtain a first quality of the first sentence as a standard sentence. The sentence evaluation method and device can solve the problem that the quality of a sentence used for human-computer interaction cannot be judged, can perform standardized evaluation on the sentence, reduce the workload of manual input or screening of the sentence, and improve the interaction quality of a human-computer interaction system.

Description

Technical Field

[0001] This application relates to the field of natural language processing technology, and in particular to a method and apparatus for evaluating sentences. Background Technology

[0002] With the development of information and internet technologies, human-computer interaction systems such as dialogue systems can provide the ability to interact with users. Based on natural language processing (NLP), they can interact with users to answer questions raised by users and judge the user's intentions.

[0003] In the process of human-computer interaction, the construction of standard statements such as knowledge base is important for improving the quality of interaction. Specifically, the quality of statements is related to natural language understanding (NLU) in human-computer interaction systems and indirectly affects natural language generation (NLG).

[0004] However, existing technologies lack a scheme for evaluating statements, making it impossible to determine the quality of statements used for human-computer interaction. Summary of the Invention

[0005] This application provides a method and apparatus for evaluating statements, to at least solve the problem in related technologies that the quality of statements used for human-computer interaction cannot be determined. The technical solution of this application is as follows:

[0006] According to a first aspect of the embodiments of this application, a statement evaluation method is provided, the statement evaluation method comprising: acquiring first text encoding information to be evaluated; performing grammatical dependency analysis and part-of-speech analysis on the first statement to obtain a first grammatical dependency sequence and a first part-of-speech sequence; the first grammatical dependency sequence representing the grammatical dependency relations in the first statement, and the first part-of-speech sequence representing the part of speech of each word in the first statement; determining first grammatical information for quality evaluation of the first statement based on the first text encoding information, the first grammatical dependency sequence and the first part-of-speech sequence; and performing quality evaluation processing on the first statement based on the first grammatical information to obtain a first quality of the first statement as a standard statement.

[0007] According to a second aspect of the embodiments of this application, a statement evaluation apparatus is provided, the statement evaluation apparatus comprising: an acquisition unit, configured to acquire first text encoding information of a first statement to be evaluated; an analysis unit, configured to perform grammatical dependency relation analysis and part-of-speech analysis on the first statement to obtain a first grammatical dependency sequence and a first part-of-speech sequence; wherein the first grammatical dependency sequence represents the grammatical dependency relations in the first statement, and the first part-of-speech sequence represents the part-of-speech of words in the first statement; and a determination unit, configured to determine first grammatical information for quality evaluation of the first statement based on the first text encoding information, the first grammatical dependency sequence, and the first part-of-speech sequence; wherein the determination unit is further configured to perform quality evaluation processing on the first statement based on the first grammatical information to obtain a first quality of the first statement as a standard statement.

[0008] According to a third aspect of the embodiments of this application, an electronic device is provided, the electronic device comprising: a processor; and a memory for storing processor-executable instructions, wherein, when the processor executes the processor, the processor causes the processor to perform the statement evaluation method according to this application.

[0009] According to a fourth aspect of the embodiments of this application, a computer-readable storage medium is provided, which, when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the statement evaluation method according to this application.

[0010] According to a fifth aspect of the embodiments of this application, a computer program product is provided, which includes computer instructions that, when executed by a processor, implement the statement evaluation method according to this application.

[0011] The technical solutions provided by the embodiments of this application bring at least the following beneficial effects:

[0012] By performing grammatical dependency analysis and part-of-speech analysis on the statement to be evaluated, and based on the obtained grammatical dependency sequence, part-of-speech sequence, and the statement's encoding information, the grammatical information of the statement can be obtained. This allows for the evaluation of the statement's quality as a standard statement, thus addressing the lack of a statement quality evaluation scheme in existing related technologies. It enables standardized statement evaluation, reduces the workload of manual input or statement filtering, provides guidance for the construction of standard statement databases such as knowledge bases, and helps interactive systems accurately match standard statements corresponding to user input, thereby improving the interaction quality of human-computer interaction systems.

[0013] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0014] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application, and do not constitute an undue limitation of this application.

[0015] Figure 1 This is an example implementation scenario diagram of a statement evaluation method according to an exemplary embodiment of this application.

[0016] Figure 2 This is a schematic flowchart illustrating the determination of the first quality of a first statement in a statement evaluation method according to an exemplary embodiment of this application.

[0017] Figure 3 This is a schematic flowchart illustrating the step of determining a first quality based on first syntactic fusion information in a statement evaluation method according to an exemplary embodiment of this application.

[0018] Figure 4 This is a flowchart illustrating an example of determining the first quality of a first statement in a statement evaluation method according to an exemplary embodiment of this application.

[0019] Figure 5 This is a schematic flowchart illustrating the steps of determining the second quality of a second statement in a statement evaluation method according to an exemplary embodiment of this application.

[0020] Figure 6 This is an illustrative flowchart illustrating the step of determining a first semantic similarity based on encoded information in a statement evaluation method according to an exemplary embodiment of this application.

[0021] Figure 7 This is a schematic flowchart illustrating a second example of the step of determining a first semantic similarity in a statement evaluation method according to an exemplary embodiment of this application.

[0022] Figure 8 This is an illustrative flowchart illustrating the step of determining a first semantic similarity based on minimum edit distance in a statement evaluation method according to an exemplary embodiment of this application.

[0023] Figure 9 This is an illustrative flowchart illustrating the steps of determining a second semantic similarity in a statement evaluation method according to an exemplary embodiment of this application.

[0024] Figure 10 This is a flowchart illustrating an example of determining the second quality of a second statement in a statement evaluation method according to an exemplary embodiment of this application.

[0025] Figure 11 This is a schematic block diagram of a statement evaluation apparatus according to an exemplary embodiment of this application.

[0026] Figure 12This is a block diagram of an electronic device according to an exemplary embodiment of this application. Detailed Implementation

[0027] To enable those skilled in the art to better understand the technical solutions of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0028] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0029] It should be noted that in this application, "at least one of several items" refers to three parallel cases: "any one of the several items", "a combination of any number of the several items", and "all of the several items". For example, "including at least one of A and B" includes the following three parallel cases: (1) including A; (2) including B; (3) including A and B. As another example, "performing at least one of step one and step two" indicates the following three parallel cases: (1) performing step one; (2) performing step two; (3) performing both step one and step two.

[0030] The statement evaluation scheme according to this application relates to a human-computer interaction system. A human-computer interaction system can be a system that can understand the user's intention based on the statement input by the user and make a corresponding response. The following is an example of a dialogue system.

[0031] A dialogue system (also known as an "intelligent dialogue system") is a system that can interact and converse with users by understanding and processing natural language. Its applications are diverse, including task-oriented dialogue systems, chat-based dialogue systems, and question-and-answer dialogue systems. With the development of information technology, dialogue systems have been widely applied in various fields, providing great convenience for production and daily life.

[0032] A knowledge base can provide language support for a dialogue system. Specifically, the knowledge base can store multiple standard questions and multiple similar questions (also known as "extended questions") corresponding to each standard question, and pre-assign standard answers to each standard question and its corresponding similar questions. Standard questions can be some common statements related to the application scenario, and similar questions can be statements with similar meanings to the corresponding standard questions but expressed differently.

[0033] When the dialogue system receives a statement input by the user, it can search the knowledge base for a matching standard question or similar question, and then output a pre-specified standard answer for the searched standard question or similar question to achieve interactive dialogue with the user.

[0034] In existing solutions, when building or expanding a knowledge base, statements are generally input manually as standard or similar questions, or statements can be generated automatically. However, there is no clear and quantitative evaluation scheme to assess whether these statements are suitable as standard or similar questions, whether they are manually input or automatically generated.

[0035] For example, during manual input, limitations in individual language proficiency may result in inputting standard questions that do not conform to grammatical rules or are ambiguous, which could affect the quality of the dialogue. Furthermore, during the automatic generation of similar questions, there may be some repetitive or similarly worded questions. Expanding the knowledge base with such similar questions could lead to information redundancy, hindering the refinement of statements within the knowledge base.

[0036] To address the aforementioned issues, existing technologies lack a scheme for evaluating statement quality, thus failing to provide guidance for constructing or expanding statements in a knowledge base.

[0037] To address at least one of the aforementioned problems, this application proposes a statement evaluation scheme. The following will refer to... Figures 1 to 11 This application describes in detail the statement evaluation method, statement evaluation apparatus, electronic device, computer-readable storage medium, and computer program product according to exemplary embodiments of the present application.

[0038] It should be noted that although the application scenarios of common dialogue systems are used as examples in the context for ease of understanding, the application scenarios of this application are not limited to these. The statement evaluation scheme of this application can also be applied to other application scenarios involving the construction or expansion of statement knowledge base or the establishment of standard statements, such as search prediction and search related statements in the application scenario of search engines.

[0039] The following will refer to Figures 1 to 10A statement evaluation method according to exemplary embodiments of this application is described. Here, an example implementation scenario of the statement evaluation method according to exemplary embodiments of this application is first described.

[0040] Reference Figure 1 This example implementation scenario includes a user terminal and a statement evaluation platform. Users can request a statement quality evaluation from the statement evaluation platform 1000 via a data transmission line on a client (e.g., mobile phone 111, desktop computer 112, tablet computer 113, etc.). Here, data transmission can be based on communication methods such as Bluetooth or wireless / wired networks.

[0041] As an example, the statement evaluation platform 1000 can obtain the first text encoding information of the first statement to be evaluated sent by the client through the data transmission line, and can also perform grammatical dependency analysis and part-of-speech analysis on the first statement to obtain the first grammatical dependency sequence and the first part-of-speech sequence. The first grammatical dependency sequence can represent the grammatical dependency relations in the first statement, and the first part-of-speech sequence can represent the part of speech of each word in the first statement.

[0042] The statement evaluation platform 1000 can determine the first grammatical information for quality evaluation of the first statement based on the aforementioned first text encoding information, first grammatical dependency sequence, and first part-of-speech sequence. Thus, the statement evaluation platform 1000 can perform quality evaluation processing on the first statement based on the first grammatical information to obtain the first quality of the first statement as a standard statement.

[0043] The statement evaluation platform 1000 can send the determined first quality of the first statement to the client. Here, the client can be, but is not limited to, [a specific type of client]. Figure 1 The mobile phone 111, desktop computer 112, and tablet computer 113 shown can also be smart wearable devices, smart home appliances, or other devices with information interaction capabilities; the statement evaluation platform can be, but is not limited to, servers, desktop computers, mobile phones, tablet computers, or other devices with information processing functions.

[0044] The above is just one example implementation scenario of the statement evaluation method. It is not limited to this and can also be applied to other implementation scenarios. For example, you can also directly input statements on the statement evaluation platform 1000 for statement evaluation.

[0045] Figure 2 This is a schematic flowchart illustrating the determination of the first quality of a first statement in a statement evaluation method according to an exemplary embodiment of this application. Figure 2 The statement evaluation method described above can be executed by a server, which can be... Figure 1 The statement evaluation platform described herein can have a server that is a standalone physical server, a server cluster consisting of multiple servers, or a cloud server capable of cloud computing. For example... Figure 2 As shown, the statement evaluation method may include the following steps:

[0046] Step S21: Obtain the first text encoding information of the first statement to be evaluated.

[0047] The first statement can be a candidate for a standard statement. Before adding it to the knowledge base or standard statement library, its quality as a standard statement can be evaluated.

[0048] It should be noted that in this application, a statement can also be called a "query," but it can be an interrogative sentence, a declarative sentence, or a combination sequence of phrases or words, etc. This disclosure does not impose any particular limitation on the way statements are expressed. The standard statements mentioned in this application can be standard questions in the knowledge base mentioned above, or representative statements used to match user input in natural language. The similar statements mentioned in this application can be similar or extended questions in the knowledge base mentioned above, or statements that are similar, extended, or related expressions of other standard statements.

[0049] It should also be noted that the statements described in this application can be in text form or in voice form, and this disclosure does not impose any special restrictions on the input method of the statements.

[0050] In one embodiment, the first statement can be converted into an encoded form through encoding so that it can be further processed through quantization to obtain the first text encoded information.

[0051] As an example, the first sentence can be converted into a semantic encoding vector (e.g., word embedding) based on a predetermined encoding model, and this semantic encoding vector can be used as the first text encoding information. Here, the encoding model can be, for example, but not limited to, the Simcse model. The Simcse model is a contrastive learning framework that achieves state-of-the-art (SOTA) performance in sentence vector representation. Generally, alignment (the pairs of samples constituting positive samples should match on the most similar features, thus avoiding variations on irrelevant features) and uniformity (feature vectors should be distributed roughly uniformly on a hypersphere, preserving as much data information as possible) can be used as evaluation metrics for this model. However, the encoding model in this application is not limited to this; it can also be other models based on BERT, such as the Mengzi model, CPM model, MT5 model, etc.

[0052] Here, during the encoding process, a pre-trained language model (PTMS) such as the Simcse model can be trained. Specifically, sentence samples from the same domain as the first sentence and their annotation information can be input into the aforementioned model for learning, resulting in a fine-tuned model. Then, the first sentence is encoded based on this model, yielding a semantic encoding vector representation of the first sentence, which serves as the word embedding result. Essentially, the pre-trained language model learns a good feature representation for words by running a self-supervised learning method on a massive corpus.

[0053] Step S22: Perform grammatical dependency analysis and part-of-speech analysis on the first statement to obtain the first grammatical dependency sequence and the first part-of-speech sequence.

[0054] Here, the first grammatical dependency sequence represents the grammatical dependency relations in the first statement, and the first part-of-speech sequence represents the part of speech of each word in the first statement.

[0055] In this step, dependency parsing can be performed on the first sentence to analyze the grammatical relationships and parts of speech of the words in the sentence. Grammatical relationships can be, for example, subject-verb, verb-object, or attributive-head relationships. Grammatical relationships are relative and involve two or more words, reflecting the sentence structure. Parts of speech refer to the grammatical classification of words, such as adjectives, nouns, pronouns, etc.

[0056] Here, syntactic dependency parsing can be implemented based on any syntactic dependency parsing or syntactic dependency detection framework, such as, but not limited to, using the PaddlePaddle framework or the LTP semantic analysis model.

[0057] It should be noted that the term "words" in this application refers to linguistic units that are defined for any language and according to any rules, such as Chinese characters or words, or English words.

[0058] The first grammatical dependency sequence can represent the grammatical relationships between words in the first sentence, and the first part-of-speech sequence can represent the part of speech of words in the first sentence. For example, the first sentence can be "I am a student", which includes the words "I, am, a, student". After grammatical dependency analysis, the first grammatical dependency sequence is "subject-predicate relation, core, attributive-head relation, verb-object relation", and the first part-of-speech sequence is "pronoun, verb, quantifier, noun".

[0059] Here, the sequence contains the order of the items in the sequence, and these ordering relationships are also a kind of information about the statement.

[0060] For this step, this application recognizes that in evaluating whether the first sentence is suitable as a standard sentence, the standard sentence should be a grammatically correct sentence, that is, it should have standard grammatical relations and accurate part-of-speech usage. Therefore, in this step, this application introduces grammatical dependency sequences and part-of-speech sequences to judge the quality of the first sentence as a standard sentence from the perspective of grammatical relation standardization and accurate part-of-speech usage. The higher the quality of the standard sentence, the more likely it is to derive high-quality similar sentences, and the better the trained model will perform.

[0061] Step S23: Determine the first grammatical information for quality evaluation of the first statement based on the first text encoding information, the first grammatical dependency sequence, and the first part-of-speech sequence.

[0062] In this step, the first syntactic dependency sequence and the first part-of-speech sequence can be encoded separately to obtain the second encoding information (e.g., an embedding) of the first syntactic dependency sequence and the third encoding information (e.g., an embedding) of the first part-of-speech sequence. The second and third encoding information are then concatenated with the first text encoding information to obtain the first syntactic information. Here, the first syntactic information can be in the form of a concatenated word vector.

[0063] Here, a grammatical relation encoding library and a part-of-speech encoding library can be established. The grammatical relation encoding library can store each grammatical relation and its corresponding unique code, and the part-of-speech encoding library can store each part of speech and its corresponding unique code. In this way, the first grammatical dependency sequence and the first part-of-speech sequence can be represented by encoding, allowing them to be fused with the first text encoding information. The fused first grammatical information carries the original features of the first sentence (i.e., the first text encoding information), the grammatical relation features of the first sentence (i.e., the second encoding information of the first grammatical dependency sequence), and the part-of-speech features of the first sentence (i.e., the third encoding information of the first part-of-speech sequence). In the above method, grammatical relations and word parts of speech can be expressed using mathematical language, and features from different dimensions can be concatenated through encoding fusion for subsequent quality evaluation. This allows for quantitative evaluation of sentence quality, making the evaluation results more reliable.

[0064] In addition to semantic dependency analysis, according to exemplary embodiments of this application, the type of statement can also be considered, and the statement type information can be fused with the first text encoding information, the first grammatical dependency sequence and the first part-of-speech sequence described above.

[0065] Specifically, the statement evaluation method may also include: classifying the first statement to obtain the statement type of the first statement; the statement type may include written language or spoken language.

[0066] In this step, the first sentence can be classified using a written / spoken analysis model to determine whether it is written or spoken language. Here, the written / spoken analysis model is, for example, but not limited to, the Roberta model.

[0067] In this step, this application recognizes that, in evaluating whether the first statement is suitable as a standard statement, standard statements are generally written language and rarely spoken language. Therefore, in this step, this application introduces an analysis of statement types to judge the quality of the first statement as a standard statement from the perspective of statement type.

[0068] In this example, the first grammatical information can be obtained by fusing the statement type, the first text encoding information, the first grammatical dependency sequence, and the first part-of-speech sequence.

[0069] For example, in addition to the third encoding information of the first grammatical dependency sequence and the fourth encoding information of the first part-of-speech sequence mentioned above, the sentence type can also be encoded to obtain the fourth encoding information. For example, written language and spoken language correspond to different codes.

[0070] Thus, the fused first grammatical information carries the original features of the first statement (i.e., the first text encoding information), the grammatical relation features of the first statement (i.e., the second encoding information), and the part-of-speech features of the first statement (i.e., the third encoding information), as well as the features of the statement type (i.e., the fourth encoding information), thereby allowing for the evaluation of statement quality from multiple perspectives and improving the accuracy of the evaluation results.

[0071] It should be noted that the step numbers and the order of description in this application do not represent the necessary execution order for implementing the exemplary embodiments of this application, and can be adjusted according to actual needs. For example, the order of determining the first text encoding information, the second encoding information, the third encoding information and the fourth encoding information can be arbitrary.

[0072] In step S24, the first statement can be evaluated based on the first grammatical information to obtain the first quality of the first statement as a standard statement.

[0073] In one example, the first syntactic information can be processed for quality evaluation through a linear layer and a softmax layer to obtain the probability / score of the first statement as a standard statement, thereby determining the first quality.

[0074] In another example, long-distance dependency analysis can be performed on the first grammatical information, and the analysis results can be integrated into the original first grammatical information. After passing through a linear layer and a softmax layer, the probability / score of the first statement as a standard statement can be obtained to determine the first quality.

[0075] Specifically, such as Figure 3 As shown, the steps for performing quality evaluation on the first statement based on the first syntactic information to obtain the first quality of the first statement as a standard statement may include:

[0076] In step S31, semantic relationship analysis between words in the first statement can be performed based on the first grammatical information to obtain long-distance dependency features.

[0077] Here, long-distance dependency features represent the semantic relationships between words in the first sentence. These semantic relationships can be the semantic relationships between each word and all words in the first sentence (including the aforementioned words themselves).

[0078] In this step, long-range dependency features can be obtained by inputting the first syntactic information into an encoding layer that incorporates a multi-head attention mechanism. Multi-head attention is a mechanism that captures long-range dependencies by focusing on different parts of the input vector.

[0079] In addition, the encoding layer can be implemented using the BERT model. BERT (Bidirectional Encoder Representation from Transformers) is a pre-trained language representation model that emphasizes that instead of using traditional unidirectional language models or shallow concatenation of two unidirectional language models for pre-training, it adopts a new masked language model (MLM) to generate deep bidirectional language representations.

[0080] For example, taking the above example sentence "I am a student", we can calculate the similarity between the word vector of the word "I" in the first grammatical information and the word vector of each word "I", "is", "a" and "student". The obtained similarity can be used as a long-distance dependency feature.

[0081] In step S32, long-distance dependency features can be fused with the first grammatical information to obtain the first grammatical fusion information.

[0082] Long-distance dependency features can be added to the first grammatical information to obtain the first grammatical fusion information. In one embodiment, the first grammatical information can be in vector form. Here, the long-distance dependency features obtained in step S31 (such as the similarity mentioned above) can be multiplied by the first grammatical information as weights to obtain the first grammatical fusion information.

[0083] In step S33, the first statement can be evaluated based on the first syntactic fusion information to determine the first quality of the first statement.

[0084] In one embodiment, the first syntax fusion information can be input into a linear layer to obtain the linearly transformed first syntax fusion information. For example, the linear layer can be in the form of a vector. The first syntax fusion information can be multiplied with the vector of the linear layer to obtain a probability value. This probability value can be input into a softmax layer to classify the first statement based on a preset probability threshold. For example, if the probability value is greater than or equal to the probability threshold, the first statement is a standard statement; if the probability value is less than the probability threshold, the first statement is not a standard statement.

[0085] In this step, since the first grammatical fusion information includes the aforementioned long-distance dependency features, the accuracy of the first quality assessment obtained based on this first grammatical fusion information is higher. In other words, as mentioned above, the first grammatical information carries the inherent features of the first sentence, the grammatical relation features of the first sentence, the part-of-speech features of the first sentence, and the features of the sentence type. These features represent the information of the words themselves or the information between words with grammatical relations. The long-distance dependency features can more comprehensively reflect the semantic relations of all words on this basis. Even the semantic relations between words without grammatical relations can be reflected through the long-distance dependency features, thereby enabling a more comprehensive evaluation of the first sentence and improving the accuracy of the evaluation.

[0086] Furthermore, according to an exemplary embodiment of this application, the step of obtaining the first quality of the first statement as a standard statement may further include: performing quality evaluation processing on the first statement based on first syntactic information to obtain the initial quality of the first statement as a standard statement; and performing weighted processing on the initial quality of the first statement as a standard statement based on a predetermined first weighting rule to obtain the first quality of the first statement as a standard statement.

[0087] Here, the first weighting rule can be a rule specified for a specific application domain. It can weight the initial quality of a statement according to the actual application requirements, and use the weighted initial quality as the first quality. For example, compared with a statement in the form of a declarative sentence, a statement in the form of a question can be given a greater weighting value for its initial quality, so that the first statement is used as the first quality of the standard statement.

[0088] Thus, based on the general statement evaluation method described in steps S21 to S24 above, predefined weighting rules can be added to make the statement evaluation method applicable to specific domains and meet specific business needs, thereby making the statement evaluation method more flexible and adaptable to a wider range of environments.

[0089] Figure 4This is a flowchart illustrating an example of determining the first quality of a first statement in a statement evaluation method according to an exemplary embodiment of this application. Figure 4 This is an illustrative example using a knowledge base application scenario.

[0090] like Figure 4 As shown, in step S41, the first statement to be evaluated (also known as a query) can be entered, for example, what conditions are required for a loan.

[0091] In step S42, the first statement can be input into the Simcse model, and the semantic encoding vector (word embedding) of the first statement can be obtained through the encoding method.

[0092] In step S43, the first statement can be input into the dependency parsing module of the PaddlePaddle framework to obtain the first grammatical dependency sequence and the first part-of-speech sequence.

[0093] In step S44, the first sentence can be input into the Roberta model to obtain the written / spoken classification of the first sentence.

[0094] In step S45, the first grammatical dependency sequence, the first part-of-speech sequence, and the written / spoken language classification can be sequentially encoded and added layer by layer to the semantic encoding vector (Word embedding) of the first sentence. By concatenating them, a new fused vector representation containing dependency relations and other information is formed, which is the first grammatical information.

[0095] In step S46, the first syntactic information can be obtained by using the encoder layer and a multihead attention mechanism to obtain information containing long-distance dependency features, thus obtaining the first syntactic fusion information.

[0096] In step S47, the first syntax fusion information can be passed through a linear layer to obtain the first syntax fusion information after linear transformation.

[0097] In step S48, the first grammar fusion information after linear transformation can be passed through the Softmax layer to predict whether it is a standard sentence in a probabilistic manner, thus obtaining the initial quality.

[0098] In step S49, the initial quality can be post-processed according to the rules of the standard statements in the proprietary domain to obtain the first quality. For example, the initial quality can be weighted according to the rules of the standard statements to obtain the first quality.

[0099] In step S410, the final output can be: whether the first statement is a standard statement, and the standard score.

[0100] The above reference Figures 2 to 4 The process of determining the first quality of a first statement as a standard statement in a statement evaluation method is described. The statement evaluation method according to an exemplary embodiment of this application can also evaluate the quality of similar statements.

[0101] For example, after obtaining the first quality of the first statement as the standard statement, the statement evaluation method may also include: if the first quality is greater than or equal to the quality threshold, then the first statement is taken as the target standard statement.

[0102] Here, the first quality of the first statement being greater than or equal to the quality threshold can characterize that the first statement can be used as a standard statement.

[0103] According to the statement evaluation method of the exemplary embodiments of this application, the first statement can also be used as a target standard statement to determine whether the second statement can be used as a similar statement to the target standard statement. Reference will be made below. Figures 5 to 10 This describes the process of determining the second quality of a second statement in a statement evaluation method according to an exemplary embodiment of this application.

[0104] like Figure 5 As shown, the statement evaluation method according to an exemplary embodiment of this application may further include the following steps:

[0105] In step S51, the second statement, the target standard statement corresponding to the second statement, and the existing similar statements corresponding to the target standard statement can be obtained.

[0106] In this step, the second statement can be a statement similar to the target standard statement. Before adding it to the knowledge base or standard statement library, its quality as a similar statement to the target standard statement can be evaluated.

[0107] Existing similar statements can be known statements that correspond to the target standard statement, such as similar statements that exist in the current knowledge base.

[0108] Here, the target standard statement can be a standard statement that has already been determined. As an example, the target standard statement can be a standard statement determined by the evaluation method of the first statement described in the above exemplary embodiments of this application.

[0109] For example, the second statement could be "How to apply for a loan". Through manual or model recognition, the corresponding target standard statement could be "How to apply for a loan". In the knowledge base, there could be existing similar statements corresponding to the target standard statement, such as "How to apply for a loan".

[0110] In step S52, the first semantic similarity between the second statement and the target standard statement can be determined.

[0111] In this step, the first semantic similarity characterizes the degree of semantic closeness between the second statement and the target standard statement. Semantic similarity, also known as textual similarity (STS), can be determined using existing similarity detection schemes. Several examples of determining the first semantic similarity are provided in this application and will be described in detail below.

[0112] In step S53, a second semantic similarity can be determined between the second statement and existing similar statements.

[0113] In this step, the second semantic similarity characterizes the degree of semantic proximity between the second statement and existing similar statements of the target standard statement. The method for determining the second semantic similarity will be described in detail below.

[0114] In step S54, the second statement can be evaluated based on the first semantic similarity and the second semantic similarity to obtain the second quality of the second statement as a similar statement to the target standard statement.

[0115] As an example, the first semantic similarity and the second semantic similarity can be used as feature values ​​input into a pre-trained Support Vector Machine (SVM) classifier. The SVM classifier then outputs whether the second statement can be considered a similar statement to the target standard statement. Furthermore, according to exemplary embodiments of this application, other classifiers, such as the LS-SVM classifier, can also be used.

[0116] Here, the first semantic similarity is positively correlated with the second quality, and the second semantic similarity is negatively correlated with the second quality.

[0117] Specifically, this application recognizes that when evaluating whether a statement is suitable as a similar statement to a target standard statement, the syntactic structure of the similar statements does not need to be complete and standardized. However, it is necessary to consider the following: on the one hand, the semantics of the similar statements should be clear and as close as possible to the semantics of the target standard statement, that is, the higher the similarity between the similar statements and the target standard statement, the better; on the other hand, for the same target standard statement, as many different interpretations as possible can be stored in the knowledge base. That is, while keeping the semantics unchanged, it is desirable for the statements to be more diverse and to cover as many statements that users may use as possible. Therefore, for the same target standard statement, the lower the similarity between its multiple similar statements, the richer and more diverse the expression of the similar statements, the more user statements they can cover, thereby better understanding user intent and improving the quality of user interaction.

[0118] Based on the above considerations, according to the statement evaluation method of this application, the quality of the second statement as a similar statement to the target standard statement can be determined based on the first semantic similarity and the second semantic similarity. In this way, the second statement can be evaluated in terms of semantic expression and diversity, so as to more accurately evaluate whether the second statement is suitable as a similar statement.

[0119] The following will describe an example of determining the first semantic similarity mentioned in step S52 above.

[0120] In one example, the first semantic similarity between the second statement and the target standard statement can be determined based on the encoded information.

[0121] Specifically, such as Figure 6 As shown, step S52 may include the following steps:

[0122] In step S61, the second statement can be encoded to obtain the encoded information of the second statement.

[0123] Here, the second statement can be converted into second statement encoding information (e.g., Wordembedding) based on a predetermined encoding model, wherein the encoding model can be, for example, but not limited to, the Simcse model.

[0124] In step S62, the target standard statement can be encoded to obtain the standard statement encoding information.

[0125] Here, the target standard statement can be converted into standard statement encoding information (e.g., Word embedding) based on a predetermined encoding model, wherein the encoding model can be, for example, but not limited to, the Simcse model.

[0126] In step S63, the first semantic similarity is determined based on the second statement encoding information and the standard statement encoding information.

[0127] In step S63, as a first example, the first semantic similarity can be determined based on vector distance.

[0128] Specifically, the first vector distance between the second statement encoding information and the standard statement encoding information can be determined; similarity is calculated based on the first vector distance to obtain the first semantic similarity.

[0129] Here, vector distance can also be called vector similarity. The first vector distance can be, for example, but not limited to, Euclidean distance, Manhattan distance, Mahalanobis distance, etc., and can be obtained by any method of calculating vector distance. The smaller the first vector distance, the higher the semantic similarity between the second statement and the target standard statement; conversely, the larger the first vector distance, the lower the semantic similarity between the second statement and the target standard statement.

[0130] exist Figure 6 In the example, the method of calculating vector distance can determine the first semantic similarity with less computation and faster computation speed.

[0131] In step S63, as a second example, the first semantic similarity can be determined based on grammatical dependency analysis.

[0132] Specifically, such as Figure 7 As shown, step S63 may include the following steps:

[0133] In step S71, grammatical dependency analysis and part-of-speech analysis can be performed on the second statement to obtain a second grammatical dependency sequence and a second part-of-speech sequence; the second grammatical dependency sequence represents the grammatical dependency relations in the second statement, and the second part-of-speech sequence represents the part of speech of each word in the second statement.

[0134] In step S72, second grammatical information for quality evaluation of the second statement can be determined based on the second statement encoding information, the second grammatical dependency sequence, and the second part-of-speech sequence of the second statement.

[0135] In step S73, similarity can be calculated based on the second grammatical information and the first grammatical information to obtain the first semantic similarity.

[0136] Here, since the first statement is taken as the target standard statement, the similarity can be calculated based on the second grammatical information and the grammatical information of the first statement to obtain the first semantic similarity.

[0137] For example, the second grammatical information and the first grammatical information can be in the form of encoded vectors. The first semantic similarity can be determined by calculating the vector distance between the two. Here, the vector distance can be calculated using any vector distance calculation method, such as, but not limited to, Euclidean distance, Manhattan distance, Mahalanobis distance, etc.

[0138] exist Figure 7 In the example, by encoding the grammatical dependency sequence and part-of-speech sequence, the grammatical relation information and part-of-speech information of the second statement and its corresponding target standard statement can be represented mathematically, thereby quantifying the similarity between the two and determining the first semantic similarity between the second statement and the target standard statement.

[0139] Returning to step S52, in another example, the first semantic similarity between the second statement and the target standard statement can be determined based on the minimum edit distance.

[0140] Specifically, such as Figure 8 As shown, step S52 may include the following steps:

[0141] In step S81, grammatical dependency analysis and part-of-speech analysis can be performed on the second statement to obtain a second grammatical dependency sequence and a second part-of-speech sequence; the second grammatical dependency sequence represents the grammatical dependency relations in the second statement, and the second part-of-speech sequence represents the part of speech of the words in the second statement.

[0142] In step S82, a first minimum edit distance between the second syntactic dependency sequence and the first syntactic dependency sequence can be determined.

[0143] In step S83, the second minimum edit distance between the second part-of-speech sequence and the first part-of-speech sequence can be determined.

[0144] In steps S83 and S84 above, the first minimum edit distance and the second minimum edit distance can be determined based on existing minimum edit distance algorithms. Specifically, the minimum edit distance (also known as the "shortest edit path") algorithm can extract the minimum edit distance for each language unit in one language sequence to be transformed into each language unit in another language sequence. Operations that implement the minimum edit distance include deletion, addition, and replacement.

[0145] For example, the second syntactic dependency sequence can be "subject-verb relation, core, modifier-head relation", while the first sentence syntactic dependency sequence can be "subject-verb relation, core, verb-object relation". The second syntactic dependency sequence can be transformed into the standard sentence syntactic dependency sequence by "no operation (for 'subject-verb relation'), no operation (for 'core'), deletion operation (for 'modifier-head relation'), and addition operation (for 'verb-object relation')". The number of steps of the above operations can be used as the first minimum edit distance.

[0146] For example, if the second part-of-speech sequence is "pronoun, verb, quantifier, noun", while the first sentence's part-of-speech sequence is "noun, verb, pronoun", the second part-of-speech sequence can be transformed into the first sentence's part-of-speech sequence through "replacement operation (for 'pronoun'), no operation (for 'verb'), deletion operation (for 'quantifier'), and replacement operation (for 'noun')". The number of steps of the above operations can be used as the second minimum edit distance.

[0147] In step S84, similarity can be calculated based on the first minimum edit distance and the second minimum edit distance to obtain the first semantic similarity.

[0148] As an example, the first minimum edit distance and the second minimum edit distance can be weighted and summed according to preset rules, or the first minimum edit distance and the second minimum edit distance can be summed directly, and the summed value can be used as the first semantic similarity.

[0149] Based on the above method, by introducing syntactic dependency analysis and determining the first semantic similarity by calculating the minimum edit distance, a faster computation speed can be achieved while ensuring computational accuracy.

[0150] The following will describe an example of determining the second semantic similarity mentioned in step S53 above.

[0151] like Figure 9 As shown, step S53 may include the following steps:

[0152] In step S91, existing similar statements can be encoded to obtain the encoding information of similar statements.

[0153] Here, the target standard statement can correspond to an existing sequence of similar statements. For example, for the target standard statement "how to apply for a credit line", the existing sequence of similar statements can be: ["how to operate a loan", "how to apply for a loan", "how to apply for a loan"]. The existing sequence of similar statements includes three existing similar statements.

[0154] As an example, an existing sequence of similar sentences can be converted into a sequence of encoded information of existing similar sentences (e.g., a Word embedding list) based on a predetermined encoding model, wherein the encoding model can be, for example, but not limited to, the Simcse model.

[0155] In step S92, a second vector distance can be determined between the second statement encoding information and the similar statement encoding information of existing similar statements.

[0156] In one embodiment, the second statement can be converted into second statement encoding information (e.g., Word embedding) based on a predetermined encoding model, wherein the encoding model can be, for example, but not limited to, the Simcse model.

[0157] For example, a second vector distance can be determined between the second statement encoding information and the existing similar statement encoding information in the existing similar statement encoding information sequence, resulting in a second vector distance list. Here, the vector distance can be implemented using any vector distance calculation method, which can be, for example, but not limited to, Euclidean distance, Manhattan distance, Mahalanobis distance, etc.

[0158] In step S93, the second semantic similarity between the second statement and existing similar statements can be determined based on the second vector distance.

[0159] Here, determining the second semantic similarity based on the second vector distance can accurately determine the degree of similarity between the second statement and existing similar statements while ensuring computational speed, providing quantitative guidance for statement evaluation.

[0160] In this step, as an example, the target standard statement may correspond to multiple existing similar statements. The minimum vector distance between the encoding information of the second statement and the encoding information of similar statements among the multiple existing similar statements can be determined as the second semantic similarity. For example, the minimum value in the list of second vector distances can be taken as the diversity score of the second statement and other similar statements in the corresponding cluster of the target standard statement.

[0161] Thus, the semantic similarity between the second statement and the closest existing similar statement is used as the diversity score of the second statement. Therefore, the semantic similarity between the second statement and other existing similar statements is greater than the minimum value mentioned above, so the minimum diversity score of the second statement can be taken into account, thereby improving the reliability of the evaluation results of the second statement.

[0162] Furthermore, according to an exemplary embodiment of this application, the step of obtaining the second quality of the second statement as a target standard statement may further include: performing quality evaluation processing on the second statement based on the first semantic similarity and the second semantic similarity to obtain the initial quality of the similar statements of the second statement as the target standard statement; and performing weighted processing on the initial quality of the similar statements of the second statement as the target standard statement based on a predetermined second weighting rule to obtain the second quality of the similar statements of the second statement as the target standard statement.

[0163] Here, the second weighting rule can be a rule specified for a specific application domain. It can weight the initial quality of statements according to actual application needs, and use the weighted initial quality as the first quality. For example, compared to the initial quality of the second statement being an interrogative sentence and the target standard statement being a declarative sentence, a larger weighting value can be assigned to the initial quality of both the second statement and the target standard statement being declarative sentences, thus obtaining the second quality of statements similar to the second statement as the target standard statement.

[0164] Thus, based on the general statement evaluation method described in steps S51 to S54 above, predefined weighting rules can be added to make the statement evaluation method applicable to specific domains and meet specific business needs, thereby making the statement evaluation method more flexible and adaptable to a wider range of environments.

[0165] Figure 10 This is a flowchart illustrating an example of determining the second quality of a second statement in a statement evaluation method according to an exemplary embodiment of this application. Figure 10 This is an illustrative example using a knowledge base application scenario.

[0166] like Figure 10 As shown, in step S101, a second statement to be evaluated (also known as a query) can be entered, for example, "How do I apply for a loan?"

[0167] In step S102, the second statement can be input into the Simcse model, and the semantic encoding vector (word embedding) of the second statement can be obtained through the encoding method.

[0168] In step S103, you can input the target standard statement corresponding to the second statement to be evaluated, such as how to apply for a credit line.

[0169] In step S104, the target standard sentence can be input into the Simcse model, and the semantic encoding vector (Word embedding) of the target standard sentence can be obtained through the encoding method.

[0170] In step S105, the vector similarity (i.e., the first vector distance) between the semantic encoding vector of the second statement and the semantic encoding vector of the target standard statement can be calculated as the first semantic similarity (similarity_score).

[0171] In step S106, an existing sequence of similar statements corresponding to the target standard statement to be evaluated can be input, such as [“How to operate a loan”, “How to apply for a loan”, “How to apply for a loan”].

[0172] In step S107, existing similar sentences can be input into the Simcse model one by one, and the encoding vector list (Word embedding list) of existing similar sentence sequences can be obtained through the encoding method.

[0173] In step S108, the distance between the semantic encoding vector of the second statement and the vector similarity of the existing similar statement encoding vectors in the encoding vector list (i.e., the second vector distance) can be calculated one by one to form the second vector distance list.

[0174] In step S109, the minimum value in the second vector distance list can be taken as the diversity score between the second statement and other existing similar statements in the corresponding cluster, that is, the second semantic similarity.

[0175] In step S1010, the first semantic similarity and the second semantic similarity can be used as feature values ​​and input into the trained SVM classifier.

[0176] In step S1011, the initial quality of whether the second statement can be used as a similar statement to the target standard statement can be output.

[0177] In step S1012, rules for similar statements in the proprietary domain can be input to post-process the initial quality and obtain a second quality. For example, based on the rules for similar statements, the initial quality can be weighted to obtain the second quality of the second statement as a similar statement to the target standard statement.

[0178] In step S1013, the final output can be whether the second statement can be used as a similar statement to the target standard statement.

[0179] It should be noted that the statement evaluation scheme according to the exemplary embodiments of this application can be applied to various fields, such as intelligent customer service dialogue, chatbots, search engines and other application fields, and this application does not impose any special limitations on it.

[0180] For example, in the field of intelligent financial customer service dialogue applications, one can input the statement "What conditions are required for a loan?" According to the statement evaluation scheme of the exemplary embodiments of this application, the quality of the statement as a standard statement can be evaluated. For example, a score can be given for the statement as a standard statement. When the score is greater than or equal to a predetermined threshold, the statement can be used as a standard statement to build or expand the knowledge base of the intelligent financial customer service dialogue; when the score is less than the predetermined threshold, the statement will not be used as a standard statement to build or expand the knowledge base.

[0181] For example, in the field of search engine applications, one can input the statement "smartphone brand". According to the statement evaluation scheme of the exemplary embodiment of this application, the quality of similar statements of the statement as the target standard statement "smartphone brand" can be evaluated. For example, a score value can be given for the statement as a similar statement of the target standard statement. When the score value is greater than or equal to a predetermined threshold, the statement can be used as a similar statement of the target standard statement to build or expand the knowledge base of the search engine; when the score value is less than the predetermined threshold, the statement will not be used as a similar statement of the target standard statement to build or expand the knowledge base.

[0182] According to an exemplary embodiment of this application, the statement evaluation method described above can be implemented using a pre-trained statement evaluation model.

[0183] As an example, a statement evaluation model can include a standard question evaluation model, which can be used to perform quality evaluation on the first statement to obtain the first statement as the first quality of the standard statement. The standard question evaluation model can include an encoding network, a syntactic dependency parsing network, and a first quality evaluation network.

[0184] The step of encoding the first statement to obtain the first text encoding information can be performed through the encoding network in the standard question evaluation model.

[0185] Here, the encoding network can be, for example, the Simcse model.

[0186] The steps of performing syntactic dependency analysis on the first statement to obtain the first syntactic dependency sequence and the first part-of-speech sequence, and obtaining the first syntactic information by fusing the first text encoding information, the first syntactic dependency sequence and the first part-of-speech sequence, can be performed through the syntactic dependency analysis network in the standard question evaluation model.

[0187] Here, the grammar dependency parsing network can be, for example, the grammar dependency detection module of the PaddlePaddle framework.

[0188] The step of performing quality evaluation on the first statement based on the first syntactic information to obtain the first quality of the first statement as a standard statement can be performed through the first quality evaluation network in the standard question evaluation model.

[0189] Here, the first quality evaluation network may include, for example, an encoding layer, a linear layer, and a softmax layer.

[0190] As an example, the standard question evaluation model can be trained based on a first training statement sample and the first statement annotation information corresponding to the first training statement sample, where the first statement annotation information represents the quality of the first training statement sample as a standard statement. Here, the first training statement sample can be, for example, an existing standard statement.

[0191] According to an exemplary embodiment of this application, a standard evaluation model including an encoding network, a grammatical dependency parsing network, and a first quality evaluation network can be trained as a whole. In this way, the output accuracy of the model can be adjusted as a whole by adjusting the parameters of each network in the model, thereby improving the efficiency of model training.

[0192] Furthermore, the standard question evaluation model may also include a statement classification network. The step of classifying the first statement to obtain its statement type can be performed through the statement classification network in the standard question evaluation model.

[0193] Here, the sentence classification network could be, for example, the Roberta model.

[0194] According to the statement evaluation method of this application, the statement evaluation model may further include a similarity evaluation model. As an example, the similarity evaluation model may include a semantic similarity analysis network and a second quality evaluation network.

[0195] The steps of determining the first semantic similarity between the second statement and the target standard statement, and the steps of determining the second semantic similarity between the second statement and existing similar statements, can be performed through the semantic similarity analysis network in the similarity evaluation model.

[0196] The step of performing quality evaluation on the second statement based on the first and second semantic similarities to obtain the second quality of the second statement as a similar statement to the target standard statement can be performed through the second quality evaluation network in the similarity evaluation model.

[0197] Here, the second quality assessment network can be, for example, an SVM classifier.

[0198] As an example, the similarity evaluation model is trained based on a second training statement sample, a corresponding training standard statement, training similar statements corresponding to the training standard statements, and second statement annotation information corresponding to the second training statement sample. The second statement annotation information indicates the quality of the second training statement sample as a similar statement to its corresponding training standard statement.

[0199] As an example, the semantic similarity analysis network may also include a first encoding network, a second encoding network, a first semantic similarity analysis network, and a second semantic similarity analysis network.

[0200] The step of encoding the second statement to obtain its encoded information can be performed through the first encoding network. Similarly, the step of encoding the target standard statement to obtain its encoded information can be performed through the second encoding network.

[0201] Here, the first and second encoding networks can be, for example, the Simcse model.

[0202] The step of calculating the first semantic similarity based on the first vector distance can be performed by a first semantic similarity analysis network. Here, the first semantic similarity analysis network can be, for example, the syntax dependency detection module of the PaddlePaddle framework.

[0203] The step of determining the second semantic similarity between the second statement and existing similar statements can be performed using a second semantic similarity analysis network. Here, the second semantic similarity analysis network can be, for example, the Simcse model.

[0204] Figure 11 This is a schematic block diagram of a statement evaluation apparatus according to an exemplary embodiment of this application.

[0205] like Figure 11 As shown, the statement evaluation device 100 may include an acquisition unit 110, an analysis unit 120, and a determination unit 130.

[0206] The acquisition unit 110 can be used to acquire the first text encoding information of the first statement to be evaluated.

[0207] The analysis unit 120 can be used to perform grammatical dependency analysis and part-of-speech analysis on the first statement to obtain a first grammatical dependency sequence and a first part-of-speech sequence; the first grammatical dependency sequence represents the grammatical dependency relations in the first statement, and the first part-of-speech sequence represents the part of speech of each word in the first statement.

[0208] The determining unit 130 can be used to determine first grammatical information for quality evaluation of the first statement based on the first text encoding information, the first grammatical dependency sequence and the first part-of-speech sequence.

[0209] The determining unit 130 can also be used to perform quality evaluation processing on the first statement based on the first syntax information to obtain the first quality of the first statement as a standard statement.

[0210] As an example, the statement evaluation device 100 may further include a classification unit, which may be used to classify the first statement to obtain the statement type of the first statement; the statement type may include written language or spoken language. In this example, the determination unit 130 may also be used to fuse the statement type, the first text encoding information, the first grammatical dependency sequence, and the first part-of-speech sequence to obtain the first grammatical information.

[0211] As an example, the determining unit 130 can also be used to: perform semantic relationship analysis between words in the first statement based on the first grammatical information to obtain long-distance dependency features, wherein the long-distance dependency features represent the semantic relationship between each word in the first statement; fuse the long-distance dependency features with the first grammatical information to obtain first grammatical fusion information; and perform quality evaluation processing on the first statement based on the first grammatical fusion information to determine the first quality of the first statement.

[0212] As an example, the statement evaluation device 100 may further include a second statement evaluation unit, which may be used to: if the first quality is greater than or equal to a quality threshold, then take the first statement as the target standard statement.

[0213] As an example, the second statement evaluation unit can also be used to: obtain a second statement, a target standard statement corresponding to the second statement, and existing similar statements corresponding to the target standard statement; determine the first semantic similarity between the second statement and the target standard statement; determine the second semantic similarity between the second statement and existing similar statements; perform quality evaluation processing on the second statement based on the first semantic similarity and the second semantic similarity to obtain the second quality of the second statement as a similar statement to the target standard statement; the first semantic similarity is positively correlated with the second quality, and the second semantic similarity is negatively correlated with the second quality.

[0214] As an example, the second statement evaluation unit can also be used to: encode the second statement to obtain second statement encoding information; encode the target standard statement to obtain standard statement encoding information; and determine the first semantic similarity based on the second statement encoding information and the standard statement encoding information.

[0215] As an example, the second statement evaluation unit can also be used to: determine the first vector distance between the second statement encoding information and the standard statement encoding information; and calculate the similarity based on the first vector distance to obtain the first semantic similarity.

[0216] As an example, the second statement evaluation unit can also be used to: perform grammatical dependency analysis and part-of-speech analysis on the second statement to obtain a second grammatical dependency sequence and a second part-of-speech sequence; the second grammatical dependency sequence represents the grammatical dependency relations in the second statement, and the second part-of-speech sequence represents the part of speech of each word in the second statement; determine second grammatical information for quality evaluation of the second statement based on the second statement encoding information, the second grammatical dependency sequence and the second part-of-speech sequence; and calculate the similarity based on the second grammatical information and the first grammatical information to obtain a first semantic similarity.

[0217] As an example, the second statement evaluation unit can also be used to: perform grammatical dependency analysis and part-of-speech analysis on the second statement to obtain a second grammatical dependency sequence and a second part-of-speech sequence; determine the first minimum edit distance between the second grammatical dependency sequence and the first grammatical dependency sequence; determine the second minimum edit distance between the second part-of-speech sequence and the first part-of-speech sequence; and calculate the similarity based on the first minimum edit distance and the second minimum edit distance to obtain a first semantic similarity.

[0218] As an example, the second statement evaluation unit can also be used to: encode existing similar statements to obtain similar statement encoding information of existing similar statements; determine the second vector distance between the second statement encoding information and the similar statement encoding information of existing similar statements; and determine the second semantic similarity between the second statement and existing similar statements based on the second vector distance.

[0219] As an example, the target standard statement corresponds to multiple existing similar statements; the second statement evaluation unit can also be used to determine the minimum vector distance between the second statement encoding information and the similar statement encoding information of multiple existing similar statements as the second semantic similarity.

[0220] As an example, the statement evaluation device 100 may further include a weighting unit, which may be used to: perform quality evaluation processing on the first statement based on first syntactic information to obtain the initial quality of the first statement as a standard statement; and perform weighting processing on the initial quality of the first statement as a standard statement based on a predetermined first weighting rule to obtain the first quality of the first statement as a standard statement.

[0221] As an example, the weighting unit can also be used to: perform quality evaluation processing on the second statement based on the first semantic similarity and the second semantic similarity to obtain the initial quality of the second statement as a similar statement to the target standard statement; and perform weighting processing on the initial quality of the second statement as a similar statement to the target standard statement based on a predetermined second weighting rule to obtain the second quality of the second statement as a similar statement to the target standard statement.

[0222] As an example, encoding the first statement to obtain the first text encoding information is performed through the encoding network in the standard question evaluation model; performing syntactic dependency analysis on the first statement to obtain the first syntactic dependency sequence and the first part-of-speech sequence, and fusing the first text encoding information, the first syntactic dependency sequence, and the first part-of-speech sequence to obtain the first syntactic information, is performed through the syntactic dependency analysis network in the standard question evaluation model; and performing quality evaluation processing on the first statement based on the first syntactic information to obtain the first quality of the first statement as a standard statement is performed through the first quality evaluation network in the standard question evaluation model.

[0223] As an example, the standard question evaluation model is trained based on the first training statement sample and the first statement annotation information corresponding to the first training statement sample; the first statement annotation information represents the quality of the first training statement sample as a standard statement.

[0224] As an example, the first semantic similarity between the second statement and the target standard statement is determined; the second semantic similarity between the second statement and existing similar statements is determined by the semantic similarity analysis network in the similarity evaluation model; the second quality of the second statement as a similar statement to the target standard statement is obtained by the second quality evaluation network in the similarity evaluation model based on the first and second semantic similarities.

[0225] As an example, the similarity evaluation model is trained based on the second training statement sample, the training standard statement corresponding to the second training statement sample, the training similar statement corresponding to the training standard statement, and the second statement annotation information corresponding to the second training statement sample; the second statement annotation information represents the quality of the second training statement sample as a similar statement to the corresponding training standard statement.

[0226] As an example, the similarity question evaluation model is deployed within the statement analysis model, which in turn deploys the standard question evaluation model.

[0227] Figure 12 This is a block diagram of an electronic device according to an exemplary embodiment of this application.

[0228] like Figure 12 As shown, the electronic device 10 may include a processor 101 and a memory 102 for storing processor-executable instructions, wherein the processor-executable instructions, when executed by the processor, cause the processor to execute the statement evaluation method according to this application.

[0229] As an example, electronic device 10 is not necessarily a single device, but can be a collection of any means or circuits capable of executing the aforementioned instructions (or instruction sets) individually or in combination. Electronic device 10 can also be part of an integrated control system or system manager, or can be configured to interconnect with a server locally or remotely (e.g., via wireless transmission).

[0230] In electronic device 10, processor 101 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. By way of example and not limitation, processor 101 may also include analog processors, digital processors, microprocessors, multi-core processors, processor arrays, neural network processors, etc.

[0231] The processor 101 can execute instructions or code stored in the memory 102, which can also store data. Instructions and data can also be sent and received via a neural network through a neural network interface device, which can employ any known transmission protocol.

[0232] The memory 102 may be integrated with the processor 101, for example, by placing RAM or flash memory within an integrated circuit microprocessor. Alternatively, the memory 102 may include a separate device, such as an external disk drive, a storage array, or any other storage device usable by a database system. The memory 102 and the processor 101 may be operatively coupled, or may communicate with each other, for example, via I / O ports, neural network connections, etc., enabling the processor 101 to read files stored in the memory 102.

[0233] In addition, the electronic device 10 may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, mouse, touch input device, etc.). All components of the electronic device 10 may be interconnected via a bus and / or a neural network.

[0234] In exemplary embodiments of this application, a computer-readable storage medium may also be provided, which, when executed by a processor of an electronic device, enables the electronic device to perform the statement evaluation method as described in the exemplary embodiments above. The computer-readable storage medium may be, for example, a memory including instructions. Optionally, the computer-readable storage medium may be: a read-only memory (ROM), a random access memory (RAM), a random access programmable read-only memory (PROM), an electrically erasable programmable read-only memory (EEPROM), a dynamic random access memory (DRAM), a static random access memory (SRAM), flash memory, non-volatile memory, a CD-ROM, a CD-R, a CD+R, a CD-RW, a CD+RW, a DVD-ROM, a DVD-R, a DVD+R, a DVD-RW, a DVD+RW, a DVD-RAM, a BD-ROM, a BD-R, or a BD-R. LTH, BD-RE, Blu-ray or optical disc storage, hard disk drive (HDD), solid-state drive (SSD), card storage (such as multimedia cards, secure digital (SD) cards, or ultra-fast digital (XD) cards), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, and any other device configured to store a computer program and any associated data, data files, and data structures in a non-transitory manner and to provide the computer program and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the aforementioned computer-readable storage medium can run in an environment deployed in computer devices such as clients, hosts, agent devices, servers, etc. Furthermore, in one example, the computer program and any associated data, data files, and data structures are distributed across a networked computer system, such that the computer program and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner through one or more processors or computers.

[0235] In an exemplary embodiment of this application, a computer program product may also be provided, which includes computer instructions that, when executed by a processor, implement the statement evaluation method as described in the exemplary embodiments above.

[0236] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0237] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A statement evaluation method, characterized in that, The statement evaluation method includes: Obtain the first text encoding information of the first statement to be evaluated; Perform grammatical dependency analysis and part-of-speech analysis on the first statement to obtain a first grammatical dependency sequence and a first part-of-speech sequence; the first grammatical dependency sequence represents the grammatical dependency relations in the first statement, and the first part-of-speech sequence represents the part of speech of each word in the first statement; Based on the first text encoding information, the first grammatical dependency sequence, and the first part-of-speech sequence, first grammatical information for quality evaluation of the first statement is determined. The first statement is evaluated based on the first grammatical information to obtain the first quality of the first statement as a standard statement; if the first quality is greater than or equal to the quality threshold, the first statement is taken as the target standard statement. The statement evaluation method also includes: Obtain the second statement, the target standard statement corresponding to the second statement, and existing similar statements corresponding to the target standard statement; Determine the first semantic similarity between the second statement and the target standard statement; Determine the second semantic similarity between the second statement and the existing similar statements; The second statement is evaluated based on the first semantic similarity and the second semantic similarity to obtain the second quality of the second statement as a similar statement to the target standard statement; the first semantic similarity is positively correlated with the second quality, and the second semantic similarity is negatively correlated with the second quality.

2. The statement evaluation method according to claim 1, characterized in that, The step of determining the first grammatical information for quality evaluation of the first statement based on the first text encoding information, the first grammatical dependency sequence, and the first part-of-speech sequence includes: The first statement is classified to obtain the statement type of the first statement; the statement type includes written language or spoken language; The statement type, the first text encoding information, the first grammatical dependency sequence, and the first part-of-speech sequence are fused to obtain the first grammatical information.

3. The statement evaluation method according to claim 1 or 2, characterized in that, The process of performing quality evaluation on the first statement based on the first grammatical information to obtain the first quality of the first statement as a standard statement includes: Based on the first grammatical information, semantic relationship analysis between words in the first statement is performed to obtain long-distance dependency features; The long-distance dependency features are fused with the first syntactic information to obtain the first syntactic fusion information; The first statement is evaluated based on the first syntactic fusion information to determine its first quality.

4. The statement evaluation method according to claim 1, characterized in that, Determining the first semantic similarity between the second statement and the target standard statement includes: The second statement is encoded to obtain the encoded information of the second statement; The target standard statement is encoded to obtain standard statement encoding information; The first semantic similarity is determined based on the second statement encoding information and the standard statement encoding information.

5. The statement evaluation method according to claim 4, characterized in that, Determining the first semantic similarity based on the second statement encoding information and the standard statement encoding information includes: Determine the first vector distance between the second statement encoding information and the standard statement encoding information; The first semantic similarity is obtained by calculating the similarity based on the first vector distance.

6. The statement evaluation method according to claim 4, characterized in that, Determining the first semantic similarity based on the second statement encoding information and the standard statement encoding information includes: Perform grammatical dependency analysis and part-of-speech analysis on the second statement to obtain a second grammatical dependency sequence and a second part-of-speech sequence; the second grammatical dependency sequence represents the grammatical dependency relations in the second statement, and the second part-of-speech sequence represents the part of speech of each word in the second statement; Based on the second statement encoding information, the second grammatical dependency sequence, and the second part-of-speech sequence, second grammatical information is determined for quality evaluation of the second statement; The first semantic similarity is obtained by calculating the similarity between the second grammatical information and the first grammatical information.

7. The sentence evaluation method according to claim 1, characterized by, Determining the first semantic similarity between the second statement and the target standard statement includes: Perform grammatical dependency analysis and part-of-speech analysis on the second statement to obtain a second grammatical dependency sequence and a second part-of-speech sequence; the second grammatical dependency sequence represents the grammatical dependency relations in the second statement, and the second part-of-speech sequence represents the part of speech of each word in the second statement; determine the first minimum edit distance between the second grammatical dependency sequence and the first grammatical dependency sequence; Determine the second minimum edit distance between the second part-of-speech sequence and the first part-of-speech sequence; The first semantic similarity is obtained by calculating the similarity based on the first minimum edit distance and the second minimum edit distance.

8. The sentence evaluation method according to any one of claims 1-2, 4-7, characterized by, Determining the second semantic similarity between the second statement and the existing similar statements includes: The existing similar statements are encoded to obtain the similar statement encoding information of the existing similar statements; Determine the second vector distance between the second statement encoding information of the second statement and the similar statement encoding information of the existing similar statements; Based on the second vector distance, a second semantic similarity is determined between the second statement and the existing similar statements.

9. The sentence evaluation method according to claim 8, characterized by, The number of existing similar statements corresponding to the target standard statement is N; the number of second vector distances is N, and there is a second vector distance between the encoding information of each existing similar statement and the encoding information of the second statement; the step of determining the second semantic similarity between the second statement and the existing similar statements based on the second vector distance includes: The minimum vector distance among the N examples of second vectors is determined as the second semantic similarity.

10. The method of claim 1, wherein, The encoding of the first statement to obtain the first text encoding information is performed through the encoding network in the standard question evaluation model; The process of performing syntactic dependency analysis on the first statement to obtain the first syntactic dependency sequence and the first part-of-speech sequence, and obtaining the first syntactic information by fusing the first text encoding information, the first syntactic dependency sequence and the first part-of-speech sequence, is performed through the syntactic dependency analysis network in the standard question evaluation model. The quality evaluation process performed on the first statement based on the first grammatical information to obtain the first quality of the first statement as a standard statement is executed by the first quality evaluation network in the standard evaluation model. The standard question evaluation model is trained based on a first training sample set, which includes a first training statement sample and first statement annotation information corresponding to the first training statement sample. The first statement annotation information indicates the quality of the first training statement sample as a standard statement.

11. The method of claim 1, wherein, The determination of the first semantic similarity between the second statement and the target standard statement, and the determination of the second semantic similarity between the second statement and the existing similar statements, are both performed through the semantic similarity analysis network in the similarity evaluation model. The quality evaluation process of the second statement based on the first semantic similarity and the second semantic similarity, to obtain the second quality of the second statement as a similar statement to the target standard statement, is performed by the second quality evaluation network in the similarity evaluation model; The similarity evaluation model is trained based on a second training sample set, which includes a second training statement sample, a training standard statement corresponding to the second training statement sample, a training similar statement corresponding to the training standard statement, and second statement annotation information corresponding to the second training statement sample. The second statement annotation information indicates the quality of the second training statement sample as a similar statement to the training standard statement.

12. A sentence evaluation device characterized by comprising: The statement evaluation device includes: The acquisition unit is used to acquire the first text encoding information of the first statement to be evaluated. The analysis unit is used to perform grammatical dependency analysis and part-of-speech analysis on the first statement to obtain a first grammatical dependency sequence and a first part-of-speech sequence; the first grammatical dependency sequence represents the grammatical dependency relations in the first statement, and the first part-of-speech sequence represents the part of speech of the words in the first statement; The determining unit is configured to determine first grammatical information for quality evaluation of the first statement based on the first text encoding information, the first grammatical dependency sequence, and the first part-of-speech sequence. The determining unit is further configured to perform quality evaluation processing on the first statement based on the first grammatical information to obtain the first quality of the first statement as a standard statement. The second statement evaluation unit is used to take the first statement as the target standard statement if the first quality is greater than or equal to the quality threshold. The second statement evaluation unit is also used for: Obtain the second statement, the target standard statement corresponding to the second statement, and existing similar statements corresponding to the target standard statement; Determine the first semantic similarity between the second statement and the target standard statement; Determine the second semantic similarity between the second statement and the existing similar statements; The second sentence is subjected to quality evaluation processing based on the first semantic similarity and the second semantic similarity, to obtain a second quality of the second sentence as a similar sentence of the target standard sentence; the first semantic similarity is positively correlated with the second quality, and the second semantic similarity is negatively correlated with the second quality.

Images