Method and device for evaluating the quality of a lecture manuscript
A neural network-based method for evaluating lecture manuscripts identifies key points and their positions to objectively assess quality, addressing inefficiencies and subjectivity in manual evaluation and improving accuracy with extensible models.
Patent Information
- Application Number
- JP2023577907
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2022-11-25
- Filing Date
- 2023-04-19
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2043-04-19
AI Technical Summary
Manually evaluating the quality of lecture manuscripts is inefficient and susceptible to subjective factors, while existing automated methods fail to accurately assess the content and logical structure of the manuscripts.
A method using a neural network model to identify the meanings of text units and determine the hit positions of key points in a lecture manuscript, calculating an evaluation result based on the hit positions and the order of key points, with optional manual adjustments and penalties for deviations.
Accurately evaluates the quality of lecture manuscripts by considering the content and logical structure, providing objective and extensible assessments without requiring large-scale training on specialized data.
Smart Images

Figure 2024545380000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to the field of natural language processing, and more particularly to a method and apparatus for evaluating the quality of a lecture manuscript. [Background technology]
[0002] A lecture is a linguistic communication activity in which a person presents his / her views and opinions on a specific issue clearly and completely in a public place, using spoken language as the main means and body language as an auxiliary means, explaining facts and expressing emotions, and promoting or inciting others. A lecture manuscript is the basis for a lecture, and it must contain a number of predetermined key points.
[0003] A lecture manuscript is always written in a certain order, such as by arranging or summarizing first, and then describing the main points, and such order may be used as an index for evaluating the quality of the lecture manuscript. It is clear that manually evaluating the quality of a lecture manuscript is relatively inefficient and easily influenced by the subjective factors of the evaluator.
[0004] The Chinese patent document CN113361275A discloses a method for evaluating the logical structure of a lecture manuscript, which evaluates the overall logical structure by identifying the conjunctions and their distribution in the lecture manuscript. Although this solution can evaluate the logic of the lecture manuscript more accurately, it ignores the content that the lecture manuscript is trying to express, and a lecture manuscript can only use conjunctions rationally, and cannot fully demonstrate its relatively high quality. Summary of the Invention [Means for solving the problem]
[0005] In view of this, the present invention provides a quality evaluation method for a manuscript for a lecture, comprising: obtaining a manuscript for a lecture and a predetermined sequence of key points, the predetermined sequence of key points including a plurality of key points; dividing the manuscript for a lecture into a plurality of text units; performing semantic identification of the plurality of text units using a neural network model to determine a hit position in the manuscript for each of the key points; and calculating an evaluation result of the manuscript for a lecture based on the hit position and the sequence of the plurality of key points.
[0006] As an option, using a neural network model to semantically identify the multiple text units and determine the hit position of each of the main points in the lecture manuscript specifically includes: the neural network model identifying each of the text units that match each of the main points; and obtaining the number of the text unit that matches each of the main points.
[0007] As an option, determining the evaluation result of the lecture manuscript based on the hit position specifically includes: determining the length of effective content based on the numbers of text units matching the first key content and the last key content; determining a desired position of each of the key content based on the number of the key content and the length of the effective content; determining whether the hit position of each of the key content matches the desired position; and calculating the evaluation result of the lecture manuscript based on all the determination results.
[0008] Alternatively, the length of the valid content is end-start, where start is the number of the text unit that matches the first gist content and end is the number of the text unit that matches the last gist content.
[0009] Alternatively, the desired location of the gist contents is a desired section determined based on the number of the gist contents, the length of the useful content, and the number of each gist content.
[0010] Alternatively, the desired interval of the i-th gist may be [start+(i-1) * (end-start) / n,start+i * (end-start) / n], where start is the number of the text unit that matches the first gist content, end is the number of the text unit that matches the last gist content, and n is the number of gist contents.
[0011] Optionally, the method further comprises obtaining manually entered desired section engagement instructions for adjusting the desired section of any of the gist content.
[0012] As an option, determining whether the hit position of each of the gist contents coincides with the desired position specifically includes: determining whether the number of the text unit corresponding to the i-th gist content is within the interval; and if the number of the text unit corresponding to the i-th gist content is within the interval, determining that the hit position coincides with the desired position.
[0013] As an option, calculating the evaluation result of the lecture manuscript based on all the judgment results specifically includes, in the case of judgment results that do not reach a consistent conclusion, determining the amount of punishment based on the degree of deviation of the hit position from the desired position, and calculating the evaluation result of the lecture manuscript based on all the punishment amounts.
[0014] The present invention further provides a quality evaluation device for a manuscript for a lecture, comprising a processor and a memory connected to the processor, wherein instructions that can be executed by the processor are stored in the memory, and the instructions are executed by the processor to cause the processor to execute the quality evaluation method for the manuscript for a lecture. Effect of the Invention
[0015] According to the method and device for evaluating the quality of a manuscript for a lecture according to the embodiment of the present invention, the evaluator is first allowed to provide the key points to be presented in the manuscript for a lecture that he / she predicts, and the solution uses a neural network model to sequentially determine the positions where each key point is mentioned in the evaluated manuscript for a lecture, and evaluates whether the logical structure of the manuscript for a lecture is appropriate and whether it is related to all the key points to be presented according to the order of the key points and the order of their mentioned positions, and can accurately evaluate the quality of the manuscript for a lecture based mainly on the content that the manuscript is intended to express. In addition, the solution does not need to train a neural network model with a large amount of specialized language materials, and can realize the identification of the hit position of the key point content by using a semantically identifiable model trained with open source data in a general field, and has higher scalability and practicality. [Brief description of the drawings]
[0016] In order to more clearly describe the specific embodiments of the present invention or the technical solutions of the prior art, the drawings necessary for describing the specific embodiments or the prior art are briefly described below. Obviously, the drawings described below are only examples of the embodiments of the present invention, and those skilled in the art can further obtain other drawings based on these drawings without any creative efforts. [Figure 1] 1 is a flowchart showing a method for evaluating the quality of a manuscript for a lecture according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a schematic diagram illustrating data classification by a neural network model according to an embodiment of the invention. [Diagram 3] FIG. 13 is a schematic diagram showing a hit situation of the main points according to the embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0017] The technical solution of the present invention will be described clearly and completely below with reference to the drawings. Of course, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, any other embodiments obtained by a person skilled in the art without inventive efforts fall within the protection scope of the present invention.
[0018] An embodiment of the present invention provides a method for evaluating the quality of a lecture manuscript, which may be implemented by an electronic device such as a computer or a server, and as shown in FIG. 1, the method includes the following steps S1 to S4.
[0019] S1: Obtain a lecture manuscript and a predetermined sequence of key points, the predetermined sequence of key points including a plurality of key points, which are in an orderly manner and are a summary of the contents to be expressed by the lecture manuscript, and may be, for example, a single word, a phrase, or a single word.
[0020] The evaluator may set multiple key points based on factors such as the subject matter and audience of the presentation manuscript. For example, if there are n key points, the evaluator will indicate that the evaluated presentation manuscript is expected to contain written content related to these n key points.
[0021] S2 Divide the presentation manuscript into a number of text units. The division may be performed by referring to the method described in CN113361275A, or by a simpler method, for example, by dividing the manuscript by punctuation marks such as periods, question marks, and exclamation marks that indicate the end of a word, and each text unit is a single word.
[0022] S3: A neural network model is used to identify the meaning of multiple text units, and to determine the hit position of each key point in the presentation manuscript. The neural network model in this solution should identify whether the content expressed by each word matches each predetermined key point, and specific algorithms may be zero-resource classification model, similarity judgment model, etc. The predetermined key point content is general-purpose, and is not an initial character manually extracted from the presentation manuscript. For example, if the presentation manuscript to be evaluated is related to the recommendation of an electronic product, one predetermined key point may be "hardware performance of the electronic product", and the neural network model should not identify whether this word exists in the presentation manuscript, but should identify each text unit and determine whether its meaning matches the hardware performance of the electronic product.
[0023] Assuming there are m text units and n gist contents, the neural network model sequentially determines the position where each gist is hit in the lecture manuscript. For example, as shown in FIG. 2, for the i-th gist content, the neural network model sequentially determines the position where each gist is hit in the lecture manuscript. i …text unit m j If it is determined that the content expressed by matches the i-th gist, then the text unit m i …text unit m j The position of the i-th main point in the entire lecture manuscript is the hit position of the i-th main point in the lecture manuscript.
[0024] In addition, the number of text units explaining a certain key point in a lecture manuscript may be one or more, and there is a risk that the text unit explaining the key point will not be identified, i.e., there are no characters expressing the key point in the entire lecture manuscript.
[0025] S4: Calculate the evaluation result of the lecture manuscript according to the hit position and the sequence of multiple key points. The core idea of this solution is that a high-quality lecture manuscript should meet the requirement that the sequence of a given key point and the sequence of the hit key points in the lecture manuscript have a linear correlation, for example, the hit position of the i-th key point content should be before the hit position of the i+1-th key point content and after the hit position of the i-1-th key point content, and if the hit positions of all key points meet the above relationship, a better evaluation result is obtained; conversely, if the above relationship is not met or some key points do not have hit positions, a worse evaluation result is obtained.
[0026] The evaluation result may be a score value or a classification result such as excellent, good, fair, or bad. There are various specific implementation logics. The more cases that do not conform to the above linear relationship, the larger the gap is and the more negative situations such as no hits occur, so the score will be lower or the classification result will be worse; conversely, the score will be higher or the classification result will be better. Therefore, a punishment can be calculated for a negative situation or a reward can be calculated for a positive situation, and an evaluation result can be obtained based on the punishment or reward.
[0027] In a preferred embodiment, in step S3, the neural network model identifies the text units that match each gist content, and obtains the numbers of the text units that match each gist content. If a lecture manuscript is divided into m words, the numbers of m text units are obtained, and the number of the text unit that matches the i-th gist content is the hit position of the i-th gist content.
[0028] Furthermore, in step S4, the length of the valid content is determined based on the numbers of the text units matching the first and last gist contents. Since the beginning and end of a lecture manuscript are generally characters that are not related to all gist contents, in order to accurately evaluate whether the distribution of the hit positions of each gist content is uniform, the length of the valid content is first determined here, the number of the text unit matching the first gist content is recorded as start, and the number of the text unit matching the last gist content is recorded as end, so that the length of the valid content is end-start.
[0029] The desired position of each gist content is determined based on the number of gist contents and the length of the effective content. In order to determine whether the hit positions are uniformly distributed, the desired positions need to be determined in combination with the total number of the predetermined gist contents. For example, when the number of predetermined gist contents is relatively small, for example, only three, it is reasonable that the hit position of the first predetermined gist is in the top third of the effective content of the lecture manuscript, and the top third of the effective content is the desired position of the first predetermined gist; similarly, the desired position of the second predetermined gist is the middle third of the effective content, and the desired position of the third predetermined gist is the last third of the effective content; when the number of predetermined gist contents is relatively large, for example, ten, the desired positions of the predetermined gist are also adjusted accordingly.
[0030] It is judged whether the hit position of each main point coincides with the desired position, and the evaluation result of the lecture manuscript is calculated based on all the judgment results. A situation where the hit position is not located at the intended position is a negative situation, and the further the hit position deviates from the intended position, the worse the judgment result for the main point content becomes and the higher the penalty value may be embodied. By summarizing the judgment results of the hit positions of all the main points, a comprehensive evaluation result can be calculated.
[0031] Furthermore, the desired location of the gist content is a desired section determined based on the number of gist contents, the length of the effective content, and the number of each gist content. The length of the desired section of each given gist may be denoted as (end-start) / n, and the desired section of the i-th gist content is [start+(i-1) * (end-start) / n,start+i * (end-start) / n], where start is the number of the text unit that matches the first gist, end is the number of the text unit that matches the last gist, and n is the number of gist contents.
[0032] According to the above preferred solution, the device implementing the method can automatically determine the desired section of each gist according to the actual situation of the speech script and the predetermined gist. In an optional embodiment, the desired section can also be manually adjusted.
[0033] For example, if some key points are more important or less important than others, the content expressing the more important key points in the manuscript should be longer, and therefore the desired section should be longer, and vice versa. The length of the desired section determined automatically is an average, and in order to obtain a more accurate evaluation result, a preferred embodiment may first automatically obtain the desired sections of all key points, and then obtain a desired section engagement command manually inputted to adjust the desired section of any key point to diverge to both ends, and any desired section can be shortened or extended as necessary.
[0034] After obtaining the desired section of each gist content, it is determined whether the number of the text unit corresponding to the i-th gist content is within the section, and if the number of the text unit corresponding to the i-th gist content is within the section, it is determined that the hit position coincides with the desired position. For example, if the number of the text unit corresponding to the i-th gist content is m i …m j If m i …mj ∈[start+(i-1) * (end-start) / n,start+i * Determine whether the expression [(end-start) / n] holds.
[0035] If the judgment results in an inconsistent conclusion, i.e., the above conditions are not met, the amount of penalty is determined based on the degree of deviation of the hit position from the desired position. The farther the hit position deviates from the intended section, the greater the amount of penalty. In a specific embodiment, the amount of penalty is defined as the percentage of the total length of the valid content that is occupied by the deviation from the desired section.
[0036] Finally, the evaluation result of the presentation manuscript is calculated based on all the penalty amounts. Figure 3 shows a specific example, where the ordinate indicates the predetermined gist contents 1 to n from top to bottom, and the abscissa indicates the text units 1 to m from left to right, and the color patches in the ordinate indicate the hit status of the gist contents in the text units, with dark color patches indicating that the text units match the gist contents, and light color patches indicating that they do not match. The recognition result by the neural network model is the degree of match, and the darker the color in the legend, the higher the degree of match of the color patches, and vice versa. The contents in the dotted line area indicate that the order of matching with the predetermined gist and the order of hitting the gist in the presentation manuscript have a linear correlation, while the outside of the dotted line area indicates the position in the presentation manuscript of the text units that hit the gist contents, and if the position does not match the order of the gist contents themselves, a penalty amount will be generated, and finally, the final result can be calculated based on all the penalty amounts.
[0037] As will be appreciated by those skilled in the art, embodiments of the present invention may be provided as a method, a system, or a computer program product. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. The present invention may also take the form of a computer program product embodied on one or more computer usable storage mediums (including, but not limited to, magnetic disk storage devices, CD-ROMs, optical storage devices, etc.) that contain computer usable program code.
[0038] The present invention has been described with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, and combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, a special purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, whereby the instructions executed by the processor of the computer or other programmable data processing apparatus create an apparatus for implementing the function specified in a process or processes in the flowcharts and / or a block or blocks in the block diagrams.
[0039] These computer program instructions may be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to operate in a particular manner, such that the instructions stored in the computer-readable memory create an article of manufacture that includes an instruction apparatus that implements the functions specified in a process or processes in the flowcharts and / or a block or blocks in the block diagrams.
[0040] These computer program instructions may be loaded into a computer or other programmable data processing apparatus to cause the computer or other programmable device to perform a series of operational steps to generate a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in a process or processes in the flowcharts and / or in a block or blocks in the block diagrams.
[0041] Obviously, the above examples are merely examples given for clarity of explanation, and are not intended to limit the embodiments. Those skilled in the art may make other different types of changes or variations based on the above description. It is not possible or necessary to list all the embodiments here. And, obvious changes or variations derived from these are still within the scope of protection of the invention.
Claims
1. A method for evaluating the quality of a manuscript for a presentation, comprising the steps of: Obtaining a lecture manuscript and a predetermined sequence of key points, the predetermined sequence of key points including a plurality of key points; Dividing the manuscript into a plurality of text units; performing semantic identification of the plurality of text units using a neural network model to determine a hit position of each of the main points in the lecture manuscript; calculating an evaluation result of the presentation manuscript based on the hit positions and the order of the plurality of main points.
2. Specifically, the method of determining a hit position of each of the main points in the lecture manuscript by identifying the meaning of the plurality of text units using a neural network model includes the following steps: the neural network model identifying each of the text units that match each of the gist portions; and obtaining a number of the text units that match each of the gist of the text.
3. Specifically, determining an evaluation result of the manuscript based on the hit position includes: determining a length of valid content based on the numbers of text units matching the first gist and the last gist; determining a desired location for each of the gist contents based on the number of gist contents and the length of the useful content; determining whether the hit location of each of the gist contents corresponds to the desired location; 3. The method according to claim 2, further comprising: calculating an evaluation result of the presentation manuscript based on all the judgment results.
4. 4. The method of claim 3, wherein the length of the valid content is end-start, where start is the number of the text unit that matches the first gist content and end is the number of the text unit that matches the last gist content.
5. 4. The method according to claim 3, wherein the desired location of the gist is a desired section determined based on the number of gist contents, the length of the useful content, and the number of each gist content.
6. The desired section of the i-th gist is [start+(i-1) * (end-start) / n, start+i * (end-start) / n, where start is the number of the first text unit that matches the gist content, end is the number of the last text unit that matches the gist content, and n is the number of gist contents.
7. 7. The method of claim 5 or 6, further comprising obtaining manually entered desired section engagement instructions for adjusting the desired section of any of the gist content.
8. Determining whether the hit position of each of the gist contents coincides with the desired position is specifically determining whether the number of the text unit corresponding to the i-th gist content is within the interval; and determining that a hit position coincides with a desired position if the number of the text unit corresponding to the i-th gist is within the interval.
9. Specifically, the calculation of the evaluation result of the manuscript based on all the judgment results is as follows: determining an amount of punishment based on the degree of deviation of the hit position from the desired position when the conclusions are not consistent; 4. The method of claim 3, further comprising: calculating an evaluation result of the lecture manuscript based on all the penalties.
10. A quality evaluation device for a manuscript for a presentation, comprising: A quality evaluation device for a lecture manuscript comprising a processor and a memory connected to the processor, wherein instructions that can be executed by the processor are stored in the memory, and the instructions are executed by the processor to cause the processor to execute the quality evaluation method for a lecture manuscript described in any one of claims 1 to 9.
Citation Information
Patent Citations
Online course video resource content identification and evaluation method and intelligent system
CN111898441A
Document detection method and device, equipment and storage medium
CN113515628A
Presentation data retrieving system and its method and program
JP2004265097A
Device, System, and Method for Assigning Differential Weight to Meeting Participants and for Generating Meeting Summaries
US20200243095A1
