Method and equipment for evaluating speech manuscripts

Abstract

The present invention discloses a method and device for evaluating speech manuscripts, which is aimed at solving the problems of large sample data, few sample types, and low validity and fairness of evaluation results in the technical field of information processing in traditional evaluation methods. The technical solution includes the steps of obtaining a number of speech manuscripts, respectively dividing each of the speech manuscripts into several sections, using a neural network model to recognize all the sections and a number of different predetermined questions, taking each of the predetermined questions and all of the sections as input data in turn, the neural network model extracts feature data from the input data, and outputs ranking information of all the sections to the predetermined questions according to the feature data, indicating the approval degree of each of the sections to the answer to the predetermined question, and determining an evaluation result of each speech manuscript according to the ranking information of all the sections to each of the predetermined questions.

Description

[Technical field]

[0001] FIELD OF THE DISCLOSURE The present invention relates to the technical field of information processing, and more particularly to a method and apparatus for evaluating a speech manuscript. [Background technology]

[0002] Speech is a way of speaking and communication in a special environment, and professional speech training requires repeated practice. Through timely scoring and evaluation during the training process, deficiencies can be identified and training progress can be accelerated. Speech scoring includes facial expression, posture, speech rate, and speech content. Summary of the Invention [Problem to be solved by the invention]

[0003] The most common scoring method for speech content is to establish a regression model, that is, to establish a large amount of data set, collect speech content in different score ranges, design features artificially or automatically extract features by machine, calculate the contribution of each feature to the score, extract effective features, and establish the relationship between the features and the score. Training the regression model extracts features from the speech manuscript data set, establishes the relationship between the features and the score, and stores it in the form of a weight matrix. However, this method relies on a large amount of data, and the samples need to cover the data of each score range, theme, etc., otherwise the scoring results will be randomly distributed, which will affect the effectiveness and fairness of the entire scoring. In practical application, when speech training begins, there are only a few excellent samples, and there are very few samples with medium to low scores. And other speech data sets in open data resources are similar, only the best speech cases are reserved, so they cannot be directly learned by transfer learning.

[0004] In view of this, the embodiment of the present invention provides a method and device for evaluating a speech manuscript, and the main objective of the present invention is to solve the problems of traditional evaluation methods, such as the large amount of sample data required, the small number of sample types, and the low validity and fairness of the evaluation results. [Means for solving the problem]

[0005] To solve the above problems, the embodiments of the present invention mainly provide the following technical solutions:

[0006] According to a first aspect, an embodiment of the present invention provides a method for evaluating a speech manuscript, the method comprising: obtaining a plurality of speech transcripts; Dividing each of said speech manuscripts into a number of sections; The method includes the steps of using a neural network model to recognize all of the sections and a plurality of different predetermined questions, taking each of the predetermined questions and all of the sections in turn as input data, the neural network model extracting feature data from the input data, and outputting ranking information for all of the sections with respect to the predetermined questions based on the feature data, and indicating the approval degree of each of the sections for the answer to the predetermined question.

[0007] An evaluation result for each speech manuscript is determined based on the ranking information for each of the predetermined questions for all of the sections.

[0008] Preferably, before using a neural network model to recognise all said sections and a plurality of different predetermined questions, Obtaining a plurality of training data, each of which includes a plurality of sample answers, one predetermined question, and ranking information of each sample answer with respect to the predetermined question; The method further includes the steps of training the neural network model using the plurality of training data, the neural network model outputting ranking information based on a plurality of sample answers and a predetermined question, and optimizing model parameters based on a difference between the output ranking information and the ranking information in the training data.

[0009] Preferably, the step of acquiring a plurality of training data specifically comprises: Crawling through a number of designated web pages for contexts and corresponding answer contents associated with the predetermined questions; and acquiring the ranking information based on a ranking status of each answer content on the web page.

[0010] Preferably, the step of determining an evaluation result of each speech manuscript based on the ranking information of all the sections with respect to each of the predetermined questions specifically includes: obtaining the highest ranking information for each of the predetermined questions of each of the sections belonging to the same speech manuscript from the ranking information for each of the predetermined questions of all of the sections; and obtaining an evaluation result for the speech manuscript based on the highest ranking information of each of the same speech manuscripts.

[0011] Preferably, the ranking information corresponds to a predetermined score value, and the evaluation result is a score value obtained based on each predetermined score value.

[0012] Preferably, the speech manuscripts are text data obtained by performing voice recognition on speech recordings, and voice pauses are recorded during the voice recognition process, and in the step of dividing each of the speech manuscripts into several sections, the speech manuscripts are divided into several sections based on the semantics of the speech manuscripts and the pauses in the speech audio.

[0013] Preferably, the ranking information includes null rankings and / or parallel ranking information.

[0014] Preferably, during the process of the neural network model extracting feature data for the input data, an attention mechanism is used to process the text data from the section and the text data from the predetermined question, and ranking information is output based on the processed feature data.

[0015] According to a second aspect, an embodiment of the present invention provides a speech manuscript evaluation apparatus, the apparatus comprising at least one processor and a memory communicatively connected to the at least one processor, the memory storing instructions executable by the at least one processor, the instructions being executed by the at least one processor to cause the at least one processor to perform the speech manuscript evaluation method described above.

[0016] According to a third aspect, an embodiment of the present invention provides a computer program product comprising instructions which, when run on a computer, cause said computer to perform the method for evaluating a manuscript speech as defined above. Effect of the Invention

[0017] According to the speech manuscript evaluation method and apparatus provided by the present invention, the evaluation task for the entire speech manuscript is converted into an evaluation for the approval degree of questions and answers, and there is no need to provide a large number of speech manuscripts of different qualities as learning samples for the neural network. By pre-setting questions related to the speech manuscript theme and preparing corresponding answers with different approval degrees, the neural network model can be trained, and the evaluation for multiple speech manuscripts can be completed, solving the problem in the prior art that speech manuscripts are difficult to evaluate due to a lack of samples, and increasing the accuracy rate of the present solution. [Brief description of the drawings]

[0018] Various other advantages will become apparent to those skilled in the art from the detailed description of the preferred embodiments below. The drawings are not intended to limit the scope of the present invention, but merely to illustrate the preferred embodiments. Like reference numerals refer to like elements throughout the drawings. [Figure 1] 1 is a flowchart of a speech manuscript evaluation method according to an embodiment of the present invention. [Diagram 2] 4 is a flowchart of another semantic evaluation method according to an embodiment of the present invention. [Diagram 3] 4 is a schematic diagram of obtaining an evaluation result for a paragraph in an embodiment of the present invention; [Figure 4] FIG. 2 is a schematic diagram of the operation process of a neural network model in an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] Hereinafter, exemplary embodiments of the present disclosure will be described in more detail with reference to the drawings. Although the drawings show exemplary embodiments of the present disclosure, it is understood that the present disclosure is not limited to the embodiments described herein, and can be realized in various forms. On the contrary, these embodiments are provided to allow the present disclosure to be more fully understood and to fully convey the scope of the present disclosure to those skilled in the art.

[0020] The present invention provides a method for evaluating a speech manuscript, which is executed by an electronic device such as a computer, a server, etc., and as shown in FIG. 1, the method includes the following steps:

[0021] 101: Obtain multiple speech manuscripts.

[0022] 102: Each of the speech manuscripts is divided into several sections.

[0023] In the embodiment of the present invention, the several sections are paragraphs expressing a certain point, and may be one formal paragraph or several formal paragraphs. For example, if the several formal paragraphs all describe the advantages of the product, these several formal paragraphs are regarded as one paragraph. C11...C1n is the section of the first speech manuscript, C21...C2n is the section of the second speech manuscript, and Cn1...Cnn are the n sections of the nth speech manuscript. The speech manuscript division method includes, but is not limited to, the following methods, that is, the whole manuscript is divided into several sections according to the semantics of the text content by the conventional semantic recognition technology. Furthermore, the sections may be formed by synthesizing the pauses and semantics according to the results of speech recognition, and there is no specific limitation.

[0024] 103: Using a neural network model to recognize all the sections and a plurality of different predetermined questions, each of the predetermined questions and all of the sections being input data in turn, the neural network model extracts feature data from the input data, and based on the feature data, outputs ranking information of all the sections with respect to the predetermined questions, indicating the approval degree of each of the sections with respect to the predetermined questions.

[0025] The approval level may be interpreted as popularity, and is learned by using a neural network to recognize training data. For example, when training a neural network, answers and questions are prepared artificially, and the ranking status of these answers for the corresponding questions is given, that is, the popularity / approval level of each answer is determined artificially, and of course, data may be transferred from other question and answer databases as sample data for training the neural network. There are multiple methods for training a neural network and setting and obtaining sample data, which will be specifically introduced in the following embodiments. As can be seen, the neural network of this solution does not recognize the shallow semantic relationship between answers and questions, but ranks multiple answers by simulating human thinking based on learned knowledge, and the higher the answer, the higher its approval / popularity, and the more the answer is preferred. However, from a semantic perspective, the higher the answer is not necessarily more relevant to a given question than the lower answers.

[0026] Fig. 4 is a schematic diagram of the operation process of the neural network, and in this solution, the trained neural network is used to recognize the divided sections and output the ranking information. Specifically, the given question and the divided sections are input to the network, for example, question 1+C11...Cnn is input, and the ranking information of C11...Cnn for the given question 1 is output; similarly, question 2+C11...Cnn is input, and the ranking information of C11...Cnn for the given question 2 is output. The given questions are set according to the content of the speech manuscript, and specifically, related questions are set as the given questions according to the theme of the speech manuscript. The higher the section, the higher the recognition / popularity of the answer to the given question.

[0027] 104: Determine an evaluation result for each speech manuscript based on the ranking information for each of the predetermined questions for all of the sections.

[0028] In one embodiment, the evaluation result is a score value, and before giving the score according to the ranking situation, a corresponding relationship between the rank and the score is set, for example, if it is ranked 1st, it corresponds to 10 points, if it is ranked 2nd, it corresponds to 8 points, etc. There are two questions in a given paragraph, and the paragraphs C11...C1n for the two questions have corresponding ranking situations, and the one with the highest rank is selected, for example, if C11 ranks 2nd (highest) for question 1, it will receive a score of 8, and if C14 ranks 3rd (highest) for question 2, it will receive a score of 6. The method of calculating the total score for the two questions of the speech manuscript includes, but is not limited to, direct addition, weighted addition, weighted average, etc.

[0029] In another embodiment, the evaluation result may be a classification result, for example, categories such as "excellent", "good", "average", and "poor" are pre-set, and the ranking information output from the neural network is classified to obtain the category to which each speech manuscript belongs.

[0030] According to the speech manuscript evaluation method and apparatus provided by the present invention, the evaluation task for the entire speech manuscript is converted into an evaluation for the approval degree of questions and answers, and there is no need to provide a large number of speech manuscripts with different qualities as learning samples for the neural network. By pre-setting questions related to the speech manuscript theme and preparing corresponding answers with different approval degrees, the neural network model can be trained, and the evaluation for multiple speech manuscripts can be completed, solving the problem in the prior art that it is difficult to evaluate speech manuscripts due to a lack of samples, and increasing the accuracy rate of the present solution.

[0031] An embodiment of the present invention further provides a method for evaluating a speech manuscript, as shown in FIG. 2, the method includes the following steps:

[0032] 201: Obtain a plurality of training data, each of which includes a plurality of sample answers, one predetermined question, and ranking information of each sample answer with respect to the predetermined question.

[0033] In an embodiment of the present invention, a neural network model is trained, and a set of training data includes a given question and a number of corresponding candidate answers, for example, a given question may be "Which basic fact is described below?", there are 40 corresponding candidate answers, and the label is a ranking status of the 40 answers, which is assigned based on the subjective will of humans. The higher the rank, the higher the quality of the answer, and the higher the sample answer, the higher its approval and popularity, and the more it is preferred.

[0034] In the embodiment of the present invention, the method of acquiring multiple training data is to crawl the context and corresponding answer contents associated with the given question from several specified web pages, and if there are duplicated answer contents, they are merged to acquire sample answers, and the ranking information is acquired according to the ranking status of each answer content in the web page. For example, for a given question in a speech script on a certain topic, the Internet (e.g., a Q&A system) can be searched for matching questions and answers can be obtained, and the questioner can select one of the answers as the best answer, and the other answers can be lined up after it, and the ranking can be directly used as the label of the training data according to the interaction status between the questioner and the answerer, such as the ranking of popularity.

[0035] Here, the number of ranks is not equal to the number of candidate answers, for example, there are 40 candidate answers, but the number of ranks is 10. There may be parallel and blanks in the ranks, for example, the first answer has 0, the second answer has 0, and the third answer has 2 or 3, and thus the ranking information includes blank rankings and / or parallel ranking information.

[0036] The predetermined questions are set based on the content of the speech manuscript, and each evaluation corresponds to a speech manuscript of the same theme. When setting the questions in advance, several questions are set for the theme of the speech manuscript, for example, "Which basic facts are described below?", "What advantages does it have over other products?", "What benefits can users obtain?", "What did the protagonist do?", "What advanced achievements has the protagonist made?", etc. Referring to the answer ranking method of a normal Q&A system, k candidate answers are selected and combined with the speech manuscript. The topK algorithm first finds top-n (for example, n=10) related documents for a specified question sentence, and may use tf-idf or bm25 algorithm. Then, n documents are divided into paragraphs to obtain a set of candidate answers that is much larger than n, and topk candidate answers (for example, k=40) are selected from them, where the above exemplary related documents and candidate answers are not limited in number and are merely examples.

[0037] 202: Training the neural network model using the plurality of training data, the neural network outputting ranking information based on a plurality of sample answers and a predetermined question, and optimizing model parameters based on a difference between the output ranking information and the ranking information in the training data.

[0038] In an embodiment of the present invention, a two-layer feedforward neural network is selected to input a given question and several candidate answers, and output the labels of several candidate answers. The above multiple training data are used to train the network, and the difference between the ranks and labels output by the network is used to determine the loss and optimize the network parameters.

[0039] The network can be denoted as f(xi) = ReLU(xiAT + b1)BT + b2.

[0040] where Xi denotes the attention feature, A∈Rm*d and B∈R1*m are the optimized weight matrix parameters, and b1∈Rm and b2∈R are the linear bias vectors.

[0041] 203: Obtain multiple speech manuscripts.

[0042] 204: Divide each of the speech manuscripts into several sections.

[0043] In a specific implementation process, the speech transcripts are text data obtained by performing voice recognition on speech recordings, and voice pauses are recorded during the voice recognition process. In the step of dividing each of the speech transcripts into several sections, the speech transcripts are divided into several sections based on the semantics of the speech transcripts and the pauses in the speech recordings.

[0044] 205: Using a neural network model to recognize all the sections and a plurality of different predetermined questions, each of the predetermined questions and all the sections in turn being input data, the neural network model extracts feature data for the input data, and based on the feature data, output ranking information of all the sections with respect to the predetermined question, indicating the approval / popularity of each of the sections with respect to the answer to the predetermined question.

[0045] FIG. 3 shows the evaluation process of the neural network model. In a preferred embodiment, during the process in which the neural network model extracts feature data from the input data, it uses an attention mechanism to process the text data from the section and the text data from the predetermined question, and outputs ranking information based on the processed feature data.

[0046] 206: From the ranking information of all the sections with respect to each of the predetermined questions, the highest ranking information of each section belonging to the same speech manuscript with respect to each of the predetermined questions is obtained.

[0047] In an embodiment of the present invention, each ranking information corresponds to a predetermined score value, for example, the first ranked situation corresponds to 10 points, the second ranked situation corresponds to 8 points, etc., and the correspondence between the rank and the score value can be specifically set based on actual needs.

[0048] 207: Obtain an evaluation result for the speech manuscript based on the highest ranking information of each of the same speech manuscripts.

[0049] In an embodiment of the present invention, for each question in a speech manuscript, the most relevant answer is found from the speech content to determine the score. For example, there are two given questions, and paragraphs C11...C1n for the two questions have corresponding ranking situations, and the one with the highest rank is selected. For example, if C11 is ranked 2nd (highest) for question 1, the score is 8, and if C14 is ranked 3rd (highest) for question 2, the score is 6.

[0050] Here, the method of calculating the total score may be to directly add up the scores of each question to obtain the total score, and weights may be set according to the importance of the questions, for example, the scores of each answer may be added up to obtain a final score of 35+30, as shown in Fig. 1. Weights may also be set according to the importance of the questions, for example, if the weight of question 1 is set to 1.2 and the weight of question 2 is set to 0.8, the final score is 1.2*35+0.8*30, and there is no limitation on the specific method of calculating the total score and the setting of weights.

[0051] In an embodiment of the present invention, the technology and data of a knowledge question and answer system are transferred to construct an evaluation method for speech manuscripts that are difficult to score due to a lack of data, and the evaluation model has a high accuracy rate and strong interpretability, and can provide scores and similar cases with corresponding score values.

[0052] An embodiment of the present invention further provides a speech manuscript evaluation device, comprising at least one processor and a memory communicatively connected to the at least one processor, the memory storing instructions executable by the at least one processor, the instructions being executed by the at least one processor to cause the at least one processor to perform a method according to the above embodiment.

[0053] An embodiment of the present invention further provides a computer program product comprising instructions, which when run on a computer, cause said computer to perform the method according to the above embodiments.

[0054] As will be appreciated by those skilled in the art, the present invention may be implemented as a method, a system, or a computer program product. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. The present invention may take the form of a computer program product embodied in one or more computer usable storage media (including, but not limited to, magnetic disk memories, CD-ROMs, optical memories, etc.) that contain computer usable program code.

[0055] The present invention will be described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention, where each flow and / or block in the flowcharts and / or block diagrams, and combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, a special purpose computer, an embedded processor, or other programmable data processing device to generate a machine, and the instructions executed by the processor of the computer or other programmable data processing device generate an apparatus that implements the functions specified in one or more flows in the flowcharts and / or one or more blocks in the block diagrams.

[0056] These computer program instructions may further be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a particular manner such that the instructions stored in the computer readable memory generate an article of manufacture including a command device, which implements the functions specified in one or more flows of the flowcharts and / or one or more blocks of the block diagrams.

[0057] These computer program instructions, when loaded into a computer or other programmable data processing device, cause the computer or other programmable device to perform a series of operational steps to generate a computer-implemented process, and the instructions executed on the computer or other programmable device provide steps to implement the functions specified in one or more flows of the flowcharts and / or one or more blocks of the block diagrams.

[0058] Obviously, the above examples do not limit the embodiments, but are merely for the purpose of clearly illustrating the examples. Those skilled in the art may make other different modifications or improvements based on the above description. It is not possible and not necessary to cover all the embodiments here. Any obvious modifications or improvements obtained therefrom still fall within the scope of protection of the present invention.

Claims

1. A method for evaluating a speech manuscript, comprising: obtaining a plurality of speech transcripts; Dividing each of said speech manuscripts into a number of sections; using a neural network model to recognize all of the sections and a plurality of different predetermined questions, taking each of the predetermined questions and all of the sections as input data in turn, the neural network model extracting feature data from the input data, and outputting ranking information for all of the sections with respect to the predetermined questions based on the feature data, thereby indicating the approval degree of each of the sections with respect to the answer to the predetermined question; determining an evaluation result for each speech manuscript based on the ranking information for each of the predetermined questions for all of the sections.

2. before using a neural network model to recognize all of said sections and a plurality of different predetermined questions; Obtaining a plurality of training data each including a plurality of sample answers, one predetermined question, and ranking information of each sample answer with respect to the predetermined question; 2. The method of claim 1, further comprising: training the neural network model using the plurality of training data, the neural network model outputting ranking information based on a plurality of sample answers and a predetermined question, and optimizing model parameters based on a difference between the output ranking information and ranking information in the training data.

3. Specifically, the steps of acquiring multiple training data are as follows: Crawling through a number of designated web pages for contexts and corresponding answer contents associated with the predetermined questions; The method of claim 2 , further comprising: obtaining the ranking information based on a ranking status of each answer content on the web page.

4. The step of determining an evaluation result of each speech manuscript based on the ranking information of all the sections with respect to each of the predetermined questions specifically includes: obtaining the highest ranking information for each of the predetermined questions of each of the sections belonging to the same speech manuscript from the ranking information for each of the predetermined questions of all of the sections; The method according to any one of claims 1 to 3, further comprising the step of: obtaining an evaluation result for the speech manuscript based on the highest ranking information of each of the same speech manuscripts.

5. The method of claim 4 , wherein the ranking information corresponds to predetermined score values, and the evaluation result is a score value obtained based on each predetermined score value.

6. The speech manuscript is text data obtained by performing voice recognition on a speech recording, and voice pauses are recorded during the voice recognition process; 2. The method of claim 1, wherein the step of dividing each of the speech manuscripts into several sections comprises dividing the speech manuscripts into several sections based on semantics of the speech manuscript and pauses in the speech audio.

7. 3. The method according to claim 1 or 2, wherein the ranking information comprises null rankings and / or parallel ranking information.

8. 3. The method according to claim 1, wherein during the process of the neural network model extracting feature data for the input data, an attention mechanism is used to process the text data from the section and the text data from the predetermined question, and ranking information is output based on the processed feature data.

9. 9. An apparatus for evaluating a speech manuscript, comprising at least one processor and a memory connected in communication with said at least one processor, said memory storing instructions executable by said one processor, said instructions being executed by said at least one processor to cause said at least one processor to perform a method according to any one of claims 1 to 8.

Images