A review method, device, computer equipment and storage device

Abstract

The present disclosure provides a review method, device, computer equipment and storage medium, the method comprising: obtaining answer information to be reviewed and reference answer information; determining a first feature matrix for characterizing the semantic association relationship between the answer to be reviewed and the reference answer based on a trained neural network model, the answer information to be reviewed and the reference answer information; determining a second feature matrix for characterizing the statistical feature relationship between the answer to be reviewed and the reference answer based on the answer information to be reviewed and the reference answer information; determining a review result corresponding to the answer information to be reviewed based on the first feature matrix, the second feature matrix and a pre-trained feature extraction matrix. Based on this method, the answer information to be reviewed can be evaluated in combination with semantic features and statistical features, thereby improving the accuracy of the review result.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a review method, device, computer equipment and storage medium. Background Art

[0002] Student questions are generally divided into subjective questions and objective questions. Objective questions can include calculation questions, multiple-choice questions, etc. When reviewing objective questions, since the answers to objective questions are certain, when reviewing objective questions, it is only necessary to determine whether the student's answer is consistent with the standard answer; for subjective questions, since there is no absolutely standard answer, they need to be reviewed manually one by one.

[0003] However, during the manual review process, the user's subjective consciousness may be carried over, and different people may have different review results for the same subjective question, resulting in relatively low accuracy and efficiency in the review of subjective questions. Summary of the invention

[0004] The embodiments of the present disclosure at least provide a review method, apparatus, computer equipment, and storage medium.

[0005] In a first aspect, an embodiment of the present disclosure provides a review method, comprising:

[0006] Obtain information on answers to be reviewed and reference answers;

[0007] Based on the trained neural network model, the answer information to be reviewed and the reference answer information, determining a first feature matrix for characterizing the semantic association relationship between the answer to be reviewed and the reference answer;

[0008] Based on the answer information to be reviewed and the reference answer information, determining a second feature matrix for characterizing the statistical feature relationship between the answer to be reviewed and the reference answer;

[0009] Based on the first feature matrix, the second feature matrix and a pre-trained feature extraction matrix, a review result corresponding to the answer information to be reviewed is determined.

[0010] In a possible implementation manner, determining a first feature matrix for characterizing the semantic association relationship between the answer to be reviewed and the reference answer based on the trained neural network model, the answer to be reviewed information and the reference answer information includes:

[0011] Preprocessing the answer information to be reviewed and the reference answer information to obtain a representation matrix for representing the answer information to be reviewed and the reference answer information;

[0012] The representation matrix is ​​input into the neural network model to obtain the first feature matrix.

[0013] In a possible implementation manner, the preprocessing of the answer information to be reviewed and the reference answer information to obtain a representation matrix for representing the answer information to be reviewed and the reference answer information includes:

[0014] Performing word segmentation processing on the answer information to be reviewed to obtain at least one first keyword to be screened corresponding to the answer to be reviewed, and performing word segmentation processing on the reference answer information to obtain at least one second keyword to be screened corresponding to the reference answer information;

[0015] Based on a plurality of stop words stored in a stop word library, determining a target keyword from the first keywords to be screened and the second keywords to be screened;

[0016] The word vector corresponding to each target keyword is determined, and the representation matrix is ​​determined according to the order in which the target keywords appear in the answer information to be reviewed and the reference answer information.

[0017] In a possible implementation, determining the word vector corresponding to each target keyword includes:

[0018] Based on a pre-set keyword dictionary storing the correspondence between keywords and word vectors, the word vector corresponding to each target keyword is searched.

[0019] In a possible implementation, the determining the target keyword from the first keywords to be screened and the second keywords to be screened based on the multiple stop words stored in the stop word library includes:

[0020] Matching the first keyword to be screened and the second keyword to be screened with a plurality of stop words stored in a stop word library respectively, and determining the keyword to be screened that is unsuccessfully matched;

[0021] In the case that the number of keywords included in the keywords to be screened that failed to match exceeds the preset number, based on the order in which the keywords to be screened that failed to match appear in the answer information to be reviewed, the first N keywords included in the keywords to be screened are used as the target keywords, where N is the preset number and N is a positive integer.

[0022] In a possible implementation manner, determining the word vector corresponding to each target keyword and determining the representation matrix according to the order in which the target keywords appear in the answer information to be reviewed and the reference answer information includes:

[0023] Determine the word vector corresponding to the token;

[0024] The word vectors corresponding to the target keywords belonging to the answer information to be reviewed are concatenated in the order in which the target keywords appear in the answer information to be reviewed to obtain a first concatenated matrix; and the word vectors corresponding to the target keywords belonging to the reference answer information are concatenated in the order in which the target keywords appear in the reference answer information to obtain a second concatenated matrix;

[0025] The character vector corresponding to the marker, the first splicing matrix and the second splicing matrix are spliced ​​to obtain the representation matrix, and the marker is used to distinguish the first splicing matrix from the second splicing matrix.

[0026] In a possible implementation manner, determining a second feature matrix for characterizing the statistical feature relationship between the answer to be reviewed and the reference answer based on the answer to be reviewed information and the reference answer information includes:

[0027] Based on the answer information to be reviewed and the reference answer information, determining statistical feature information between the information to be reviewed and the reference answer information;

[0028] The second feature matrix is ​​determined based on statistical feature information between the information to be reviewed and the reference answer information.

[0029] In a possible implementation manner, the statistical feature information includes multiple types of the following information:

[0030] The editing distance between the information to be reviewed and the reference answer information, the number of identical words and the number of different words in the information to be reviewed and the reference answer information, the word set similarity, the N-gram similarity, and the difference in text length between the information to be reviewed and the reference answer information.

[0031] In a possible implementation manner, the review result corresponding to the answer information to be reviewed includes a similarity score between the answer information to be reviewed and the reference answer information;

[0032] The determining, based on the first feature matrix, the second feature matrix and a pre-trained feature extraction matrix, a review result corresponding to the answer information to be reviewed includes:

[0033] Vertically concatenate the first characteristic matrix and the second characteristic matrix to obtain a full characteristic matrix;

[0034] Performing feature extraction on the full feature matrix based on the feature extraction matrix to obtain a similarity matrix;

[0035] Based on the similarity matrix and the activation function, a similarity score between the answer to be reviewed and the reference answer is generated.

[0036] In a possible implementation, the method further includes training the neural network model according to the following method:

[0037] Acquire the sample answer information to be reviewed, sample reference answer information corresponding to the sample answer information to be reviewed, and annotation information of the sample answer information to be reviewed, wherein the annotation information is used to indicate the correctness of the sample answer information to be reviewed;

[0038] Determine a first sample feature matrix based on the sample answer information to be reviewed, the sample reference answer information, and the neural network model to be trained;

[0039] Determining a second sample feature matrix based on statistical feature information between the sample answer information to be reviewed and the sample reference answer information;

[0040] Determine a similarity matrix between the answer to be reviewed and the reference answer based on the first sample feature matrix, the second sample feature matrix, and the feature extraction matrix to be trained;

[0041] Based on the annotation information and the similarity matrix, the loss value in this training process is determined. When the loss value is greater than a set threshold, the model parameters of the neural network model and the feature extraction matrix are adjusted, and the neural network model is retrained.

[0042] In a second aspect, the present disclosure also provides a review device, including:

[0043] The acquisition module is used to obtain the answer information to be reviewed and the reference answer information;

[0044] A first determination module is used to determine a first feature matrix for characterizing the semantic association relationship between the answer to be reviewed and the reference answer based on the trained neural network model, the answer information to be reviewed and the reference answer information;

[0045] A second determination module, configured to determine, based on the answer information to be reviewed and the reference answer information, a second characteristic matrix for characterizing the statistical characteristic relationship between the answer to be reviewed and the reference answer;

[0046] The third determination module is used to determine the review result corresponding to the answer information to be reviewed based on the first feature matrix, the second feature matrix and a pre-trained feature extraction matrix.

[0047] In a possible implementation manner, the first determination module, when determining the first feature matrix for characterizing the semantic association relationship between the answer to be reviewed and the reference answer based on the trained neural network model, the answer information to be reviewed and the reference answer information, is used to:

[0048] Preprocessing the answer information to be reviewed and the reference answer information to obtain a representation matrix for representing the answer information to be reviewed and the reference answer information;

[0049] The representation matrix is ​​input into the neural network model to obtain the first feature matrix.

[0050] In a possible implementation manner, the first determination module, when preprocessing the answer information to be reviewed and the reference answer information to obtain a representation matrix for representing the answer information to be reviewed and the reference answer information, is used to:

[0051] Performing word segmentation processing on the answer information to be reviewed to obtain at least one first keyword to be screened corresponding to the answer to be reviewed, and performing word segmentation processing on the reference answer information to obtain at least one second keyword to be screened corresponding to the reference answer information;

[0052] Based on a plurality of stop words stored in a stop word library, determining a target keyword from the first keywords to be screened and the second keywords to be screened;

[0053] The word vector corresponding to each target keyword is determined, and the representation matrix is ​​determined according to the order in which the target keywords appear in the answer information to be reviewed and the reference answer information.

[0054] In a possible implementation manner, the first determination module, when determining the word vector corresponding to each target keyword, is used to:

[0055] Based on a pre-set keyword dictionary storing the correspondence between keywords and word vectors, the word vector corresponding to each target keyword is searched.

[0056] In a possible implementation manner, the first determination module, when determining the target keyword from the first keywords to be screened and the second keywords to be screened based on the multiple stop words stored in the stop word library, is used to:

[0057] Matching the first keyword to be screened and the second keyword to be screened with a plurality of stop words stored in a stop word library respectively, and determining the keyword to be screened that is unsuccessfully matched;

[0058] In the case that the number of keywords included in the keywords to be screened that failed to match exceeds the preset number, based on the order in which the keywords to be screened that failed to match appear in the answer information to be reviewed, the first N keywords included in the keywords to be screened are used as the target keywords, where N is the preset number and N is a positive integer.

[0059] In a possible implementation manner, the first determination module, when determining the word vector corresponding to each target keyword and determining the representation matrix according to the order in which the target keywords appear in the answer information to be reviewed and the reference answer information, is used to:

[0060] Determine the word vector corresponding to the token;

[0061] The word vectors corresponding to the target keywords belonging to the answer information to be reviewed are concatenated in the order in which the target keywords appear in the answer information to be reviewed to obtain a first concatenated matrix; and the word vectors corresponding to the target keywords belonging to the reference answer information are concatenated in the order in which the target keywords appear in the reference answer information to obtain a second concatenated matrix;

[0062] The character vector corresponding to the marker, the first splicing matrix and the second splicing matrix are spliced ​​to obtain the representation matrix, and the marker is used to distinguish the first splicing matrix from the second splicing matrix.

[0063] In a possible implementation manner, the second determination module, when determining the second feature matrix for characterizing the statistical feature relationship between the answer to be reviewed and the reference answer based on the answer to be reviewed information and the reference answer information, is used to:

[0064] Based on the answer information to be reviewed and the reference answer information, determining statistical feature information between the information to be reviewed and the reference answer information;

[0065] The second feature matrix is ​​determined based on statistical feature information between the information to be reviewed and the reference answer information.

[0066] In a possible implementation manner, the statistical feature information includes multiple types of the following information:

[0067] The editing distance between the information to be reviewed and the reference answer information, the number of identical words and the number of different words in the information to be reviewed and the reference answer information, the word set similarity, the N-gram similarity, and the difference in text length between the information to be reviewed and the reference answer information.

[0068] In a possible implementation manner, the review result corresponding to the answer information to be reviewed includes a similarity score between the answer information to be reviewed and the reference answer information;

[0069] The third determination module, when determining the review result corresponding to the answer information to be reviewed based on the first feature matrix, the second feature matrix and the pre-trained feature extraction matrix, is used to:

[0070] Vertically concatenate the first characteristic matrix and the second characteristic matrix to obtain a full characteristic matrix;

[0071] Performing feature extraction on the full feature matrix based on the feature extraction matrix to obtain a similarity matrix;

[0072] Based on the similarity matrix and the activation function, a similarity score between the answer to be reviewed and the reference answer is generated.

[0073] In a possible implementation manner, the review device further includes a training module, and the training module is used to:

[0074] Acquire the sample answer information to be reviewed, sample reference answer information corresponding to the sample answer information to be reviewed, and annotation information of the sample answer information to be reviewed, wherein the annotation information is used to indicate the correctness of the sample answer information to be reviewed;

[0075] Determine a first sample feature matrix based on the sample answer information to be reviewed, the sample reference answer information, and the neural network model to be trained;

[0076] Determining a second sample feature matrix based on statistical feature information between the sample answer information to be reviewed and the sample reference answer information;

[0077] Determine a similarity matrix between the answer to be reviewed and the reference answer based on the first sample feature matrix, the second sample feature matrix, and the feature extraction matrix to be trained;

[0078] Based on the annotation information and the similarity matrix, the loss value in this training process is determined. When the loss value is greater than a set threshold, the model parameters of the neural network model and the feature extraction matrix are adjusted, and the neural network model is retrained.

[0079] In a third aspect, an optional implementation of the present disclosure further provides a computer device, a processor, and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the processor is used to execute the machine-readable instructions stored in the memory, and when the machine-readable instructions are executed by the processor, the machine-readable instructions perform the steps of the above-mentioned first aspect, or any possible implementation of the first aspect.

[0080] In a fourth aspect, an optional implementation of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed, the steps of the above-mentioned first aspect, or any possible implementation of the first aspect are executed.

[0081] The answer review method, apparatus, computer equipment and storage medium provided in the embodiments of the present disclosure use a trained neural network model to extract the semantic association relationship between the answers to be reviewed and the reference answers to obtain a first feature matrix, and then combine the statistical features between the answers to be reviewed and the reference answers to generate the review results of the answers to be reviewed. Based on this method, the review results of the answers to be reviewed can be automatically generated based on the semantic association relationship and statistical features between the answers to be reviewed and the reference answers. Therefore, while improving the review speed, it avoids the influence of subjective color on the review results and improves the accuracy of the review results.

[0082] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following is a brief introduction to the drawings required for use in the embodiments. The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used together with the specification to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can also be obtained based on these drawings without creative work.

[0084] Figure 1 A flow chart of a review method provided by an embodiment of the present disclosure is shown;

[0085] Figure 2 A flowchart showing the training process of a neural network model in the review method provided by an embodiment of the present disclosure is shown;

[0086] Figure 3 A schematic diagram of a review device provided by an embodiment of the present disclosure is shown;

[0087] Figure 4 A schematic diagram of the structure of a computer device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0088] In order to make the purpose, technical scheme and advantages of the embodiments of the present disclosure clearer, the technical scheme in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the embodiments of the present disclosure generally described and shown here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure is not intended to limit the scope of the present disclosure claimed for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present disclosure.

[0089] According to research, student questions are generally divided into subjective questions and objective questions. Objective questions can include calculation questions, multiple-choice questions, etc. When reviewing objective questions, since the answers to objective questions are fixed, it is only necessary to judge whether the student answers are consistent with the standard answers. For subjective questions, since there is no absolute standard answer, manual review is required one by one. However, during the manual review process, the user's subjective consciousness may be carried, and different people may have different review results for the same subjective question, resulting in low accuracy and efficiency in reviewing subjective questions.

[0090] In the prior art, in order to improve the accuracy and efficiency of subjective question review, two identical long short-term memory (LSTM) feature extractors are used to extract features from student answers and reference answers respectively, and the similarity between student answers and reference answers is generated by numerically calculating the extracted vectors. Since the LSTM feature extractor cannot extract deep semantic features of the text, the review accuracy is low.

[0091] The answer review method, device, computer equipment and storage medium provided by the disclosed embodiment use a trained neural network model to extract the semantic association relationship between the answers to be reviewed and the reference answers to obtain a first feature matrix, and then combine the statistical features between the answers to be reviewed and the reference answers to generate the review results of the answers to be reviewed; the disclosed embodiment extracts the semantic features of the student answers and the reference answers based on the neural network model, and combines the statistical features of the student answers and the reference answers to generate the corresponding review results; compared with the above-mentioned LSTM feature extractor, the neural network model used in the disclosed embodiment can extract deeper semantic features and combine statistical features. Based on this method, while improving the review speed, it can avoid the influence of subjective color on the review results and improve the accuracy of the review results.

[0092] The defects existing in the above solutions are the results obtained by the inventor after practice and careful research. Therefore, the discovery process of the above problems and the solutions proposed by the present disclosure for the above problems below should be the contributions made by the inventor to the present disclosure during the disclosure process.

[0093] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0094] To facilitate understanding of the present embodiment, a review method disclosed in the present embodiment is first introduced in detail. The execution subject of the review method provided in the present embodiment is generally a computer device with certain computing capabilities, and the computer device includes, for example: a terminal device or a server or other processing device, and the terminal device can be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some possible implementations, the review method can be implemented by a processor calling a computer-readable instruction stored in a memory.

[0095] See also Figure 1 FIG. 1 is a flowchart of a review method provided in an embodiment of the present disclosure, wherein the method includes steps S101 to S104, wherein:

[0096] S101: Obtain information of answers to be reviewed and information of reference answers.

[0097] S102: Based on the trained neural network model, the answer information to be reviewed and the reference answer information, determine a first feature matrix for characterizing the semantic association relationship between the answer to be reviewed and the reference answer.

[0098] S103: Based on the information of the answer to be reviewed and the information of the reference answer, determine a second characteristic matrix for characterizing the statistical characteristic relationship between the answer to be reviewed and the reference answer.

[0099] S104: Determine a review result corresponding to the answer information to be reviewed based on the first feature matrix, the second feature matrix, and a pre-trained feature extraction vector.

[0100] Here are detailed instructions for the above steps:

[0101] For S101,

[0102] In specific implementations, there may be multiple methods for obtaining the answer information to be reviewed and the reference answer information.

[0103] When obtaining the answer information to be reviewed, any of the following methods may be used:

[0104] Method 1: Obtain the answer information to be reviewed entered by the user in the answer position area preset by the client.

[0105] The answer information to be reviewed may be answers to subjective questions preset in the client.

[0106] Among them, subjective questions can be subjective understanding of the paragraph or the whole text of the given article / poem, for example: "In the article "Old House", the two sentences "like a bird returning to its nest" and "like a man in his sixties" reflect the author's feelings about his old house?"

[0107] Correspondingly, the answer information to be reviewed may be the answer to the subjective question, such as: "It reflects the author's attachment to his old house and his feeling about the fleeting time."

[0108] In a possible implementation, a first location area and a second location area may be set respectively on the client, wherein the first location area is used to display the question; and the second location area is used to indicate that the user enters the answer information to be reviewed at this location. The size of the second location area may be a preset fixed value, or after obtaining the reference answer information, the size of the second location area may be determined based on the reference answer information.

[0109] Method 2: obtaining an image containing the answer information to be reviewed, and extracting information in the image to obtain the answer information to be reviewed.

[0110] Specifically, the server can call the camera of the electronic device deployed by the client to collect images, so as to obtain the image sent by the client containing the answer information to be reviewed. The image can be passed through optical character recognition (OCR) and other recognition technologies to obtain the answer information to be reviewed.

[0111] In a possible implementation, when obtaining the reference answer information, the pre-stored reference answer information corresponding to the question information of the answer information to be reviewed can be directly obtained, wherein the reference answer information can be stored in a local database or in a server, and the reference answer information can be entered by the question setter during the question setting stage.

[0112] Taking the above subjective question "The Old House" as an example, the corresponding reference answer information can be "reflecting the author's attachment to the warm old house and the sentiment towards the vicissitudinous old house, because the old house has condensed kinship and provided spiritual shelter; witnessing the changes of the world, the passage of time, and helplessness." When the question setter enters the question information, the corresponding reference answer information can be entered and stored at the same time.

[0113] In a possible implementation manner, after obtaining the answer information to be reviewed and the reference answer information, in response to the user's trigger operation on the preset reference answer display button of the client, the reference answer information can be displayed in the preset position area of the client.

[0114] Regarding S102,

[0115] In a possible implementation manner, when determining the first feature matrix for characterizing the semantic association relationship between the answer to be reviewed and the reference answer based on the trained neural network model, the answer information to be reviewed and the reference answer information, the answer information to be reviewed and the reference answer information can be preprocessed first to obtain a characterization matrix for characterizing the answer information to be reviewed and the reference answer information, and then the characterization matrix is input into the neural network to obtain the first feature matrix.

[0116] Among them, when preprocessing the answer information to be reviewed and the reference answer information, the steps of the preprocessing include the following steps:

[0117] First, perform word segmentation on the answer information to be reviewed to obtain at least one first keyword to be screened corresponding to the answer to be reviewed, and perform word segmentation on the reference answer information to obtain at least one second keyword to be screened corresponding to the reference answer information; then, based on multiple stop words stored in the stop word library, determine the target keywords from the first keywords to be screened and the second keywords to be screened; finally, determine the word vectors corresponding to each target keyword, and determine the characterization matrix according to the order in which the target keywords appear in the answer information to be reviewed and the reference answer information.

[0118] In specific implementation, the stop words can be words that have no semantic association with the answer information to be reviewed and the reference answer information, such as modal particles, adverbs, prepositions, conjunctions, etc. Usually, they have no clear meaning by themselves and only have a certain role when placed in a complete sentence, such as common words like "de" (的), "zai" (在), etc. The classification and corresponding situations in the stop word library can be shown in Table 1:

[0119] Table 1

[0120]

[0121]

[0122]

[0123] In a possible implementation, when determining a target keyword from a first keyword to be screened and a second keyword to be screened based on multiple stop words stored in a stop word library, the first keyword to be screened and the second keyword to be screened can be respectively matched with the multiple stop words stored in the stop word library to determine the keyword to be screened that failed to match; then, when the number of keywords contained in the keyword to be screened that failed to match exceeds a preset number, the first N keywords contained in the keyword to be screened can be used as the target keywords based on the order in which the keyword to be screened that failed to match appears in the answer information to be reviewed, where N is the preset number and N is a positive integer.

[0124] Exemplarily, N can be set to 42. Taking the subjective question "Old House" as an example, the first keyword to be screened that was unsuccessfully matched and determined by the corresponding answer information to be reviewed is "home, old house, attachment, time is fleeting, emotion"; the second keyword to be screened that was unsuccessfully matched is "warmth, old house, attachment, vicissitudes, emotion, cohesion, family affection, provision, heart, shelter, witness, changes in the world, passing of time, helplessness". Since the sum of the number of words of the first keyword to be screened and the second keyword to be screened is 44, which is greater than 42, the corresponding target keywords are "warmth, warmth, old, house, dependence, love, vicissitudes, mulberry, emotion, sigh, cohesion, gathering, family, affection, provision, heart, soul, shelter, protection, witness, witness, world, things, change, migration, years, flow, passing, helplessness, self, home, old, house, dependence, love, time, light, easy, passing". Among them, "emotion" in the first keyword to be screened is deleted.

[0125] In practical applications, since the number of target keywords determined based on different reference answer information and different answer information to be reviewed may be different, and the dimension of the representation matrix is ​​related to the number of determined target keywords, in order to ensure that the dimensions of each representation matrix input into the neural network model are the same, the maximum number of target keywords can be set to N.

[0126] In another possible implementation, when the number of keywords included in the keywords to be screened that did not match successfully does not exceed a preset number, all the keywords included in the keywords to be screened can be directly used as target keywords. In addition, to ensure that the dimensions of the representation matrix input into the neural network are the same, when the number of keywords included in the keywords to be screened does not exceed a preset number, in addition to determining the keywords included in the keywords to be screened as target keywords, "0" can also be used as the target keyword, wherein after adding "0", the number of target keywords can be N.

[0127] Exemplarily, N can be set to 50. Taking the subjective question "Old House" as an example, the second keyword to be screened that failed to match successfully determined by the corresponding reference answer information is "warmth, old house, attachment, vicissitudes, emotion, cohesion, family affection, provision, soul, shelter, witness, changes in the world, passing of time, helplessness"; the first keyword to be screened that failed to match successfully determined by the corresponding answer information to be reviewed is "own home, old house, attachment, time is fleeting, emotion". After adding the corresponding number of "0", the corresponding target keywords are "warmth, warmth, old, house, dependence, love, vicissitudes, emotion, sigh, cohesion, gathering, family, affection, provision, heart, soul, shelter, witness, world, things, change, migration, years, flow, passing, helplessness, self, home, old, house, dependence, love, time, light, easy, passing, emotion, sigh, 0, 0, 0, 0, 0, 0".

[0128] After the target keyword is determined, the characterization matrix may be determined in any one of the following ways.

[0129] Method 1: first combine the target keywords based on the order in which the target keywords appear in the answer information to be reviewed and the reference answer information, and then generate the representation matrix based on the word vector corresponding to each target keyword and the combination result.

[0130] It should be noted that since the answer information to be reviewed and the reference answer information need to be input into the neural network model at the same time after preprocessing, in order to enable the neural network model to distinguish the processed answer information to be reviewed and the reference answer information, a CLS marker can be inserted at the front before input, and a SEP marker can be inserted in the middle position of input for distinction. Here, the CLS marker and the SEP marker will also have the corresponding word vector.

[0131] Further, after combining the target keywords, the combination result is: "CLS marker, warmth, fragrance, old, house, dependence, love, vicissitudes, mulberry, feeling, sigh, condense, gather, close, affection, provide, heart, soul, shelter, protect, see, witness, world, things, change, migration, years, glory, flow, passing, helpless, SEP marker, self, home, old, house, dependence, love, time, light, easy, passing, feeling, sigh, 0, 0, 0, 0, 0, 0".

[0132] In a possible implementation, when determining the word vector corresponding to each target keyword, the word vector corresponding to each target keyword and marker can be searched based on a pre-set keyword dictionary storing the correspondence between keywords, markers and word vectors. Here, the keyword dictionary for the correspondence between keywords, markers and word vectors can be as shown in Table 2:

[0133] Table 2

[0134]

[0135] In the above table, One-Hot encoding is adopted, that is, one-bit effective encoding. Then the word vectors corresponding to each target keyword (50) and marker (2) are vertically spliced ​​to generate a 52*52 representation matrix.

[0136] Method 2: first determine the word vector corresponding to each target keyword, and then determine the representation matrix according to the word vector corresponding to the target keyword and the order in which the target keyword appears in the answer information to be reviewed and the reference answer information.

[0137] In determining the word vector corresponding to each target keyword and determining the representation matrix according to the order in which the target keywords appear in the answer information to be reviewed and the reference answer information, the word vector corresponding to the marker can be determined first; then, the word vectors corresponding to the target keywords belonging to the answer information to be reviewed are spliced ​​according to the order in which the target keywords appear in the answer information to be reviewed to obtain a first splicing matrix; and the word vectors corresponding to the target keywords belonging to the reference answer information are spliced ​​according to the order in which the target keywords appear in the reference answer information to obtain a second splicing matrix; finally, the word vectors corresponding to the marker, the first splicing matrix and the second splicing matrix are spliced ​​to obtain the representation matrix, the marker is used to distinguish the first splicing matrix from the second splicing matrix, and the marker may include the CLS marker and the SEP marker.

[0138] After determining the representation matrix, the representation matrix can be input into the trained neural network model. After processing the representation matrix, the neural network model can generate and output a first feature matrix for representing the semantic association relationship between the answer to be reviewed and the reference answer.

[0139] In a possible implementation, the neural network model may be an ALBERT model, which integrates the above-mentioned preprocessing process, that is, the above-mentioned preprocessing function may be implemented in the ALBERT model.

[0140] Specifically, the answer information to be reviewed and the reference answer information can be directly input into the trained ALBERT model. After the ALBERT model processes the answer information to be reviewed and the reference answer information, the first feature matrix representing the semantic association relationship between the answer to be reviewed and the reference answer can be obtained.

[0141] For S103,

[0142] When determining a second characteristic matrix for characterizing the statistical characteristic relationship between the answer to be reviewed and the reference answer based on the answer information to be reviewed and the reference answer information, the statistical characteristic information between the information to be reviewed and the reference answer information can be first determined based on the answer information to be reviewed and the reference answer information; and then, the second characteristic matrix can be determined based on the statistical characteristic information between the information to be reviewed and the reference answer information.

[0143] The statistical feature information includes multiple types of the following information:

[0144] The editing distance between the information to be reviewed and the reference answer information, the number of identical words and the number of different words in the information to be reviewed and the reference answer information, the word set similarity, the N-gram similarity, and the difference in text length between the information to be reviewed and the reference answer information.

[0145] The edit distance refers to a quantitative measurement of the difference between two strings, and the measurement method is to see how many times a string is processed to transform one string into another string, that is, how many times the reference answer information is processed to transform the information to be reviewed. Here, according to the existing edit distance algorithm, the information to be reviewed and the reference answer information are input to generate the corresponding edit distance A. A is a natural number.

[0146] Still taking the subjective question "Old House" as an example, there are 5 words in the information to be reviewed after word segmentation, and 14 words in the reference answer information. Among them, the 3 words "old house, attachment, and emotion" are the same words, and the rest of the words are different words. Among them, the same words are the number of words in the information to be reviewed that hit the reference answer information, for example, the information to be reviewed includes the 3 words "old house, attachment, and emotion" in the reference answer information, and the different words are the opposite, which will not be repeated here.

[0147] The number of identical words and the number of different words in the information to be reviewed and the reference answer information are "3, 2, 3, 11", where "3" means there are 3 identical words; "2" means there are 2 different words in the information to be reviewed; and "11" means there are 11 different words in the reference answer information.

[0148] The word set similarity refers to the similarity between each word in the information to be reviewed and the reference answer information. Based on the existing Jaccard similarity algorithm, the information to be reviewed and the reference answer information are input to generate the corresponding word set similarity B. B is a number in the range of [0, 1].

[0149] The N-gram similarity of the N-gram model is also based on each word to calculate the similarity between two documents. Based on the existing N-gram algorithm, the information to be reviewed and the reference answer information can be input to generate the corresponding N-gram similarity C. The C is a number in the range of [0, 1].

[0150] The text length difference between the information to be reviewed and the reference answer information refers to the text length difference between the two original documents without word segmentation processing. The text length of the reference answer information is 52, and the text length of the information to be reviewed is 24, and the corresponding text length difference is 28.

[0151] When determining the second feature matrix based on the statistical feature information between the information to be reviewed and the reference answer information, the above statistical feature information may be concatenated to generate the second feature matrix, for example: [A 3 23 11 BC 24].

[0152] In a possible implementation, when the statistical feature information is concatenated to generate the second feature matrix, the statistical feature information may be normalized, and the range of the normalized statistical feature information is compressed to [0, 1].

[0153] For example, the values ​​of the statistical feature information are 1, 1, 1, 1, 1, 1, 1, 1. Correspondingly, the generated normalized second feature matrix is: [0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125]

[0155] For S104,

[0156] When determining the review result corresponding to the answer information to be reviewed based on the first feature matrix, the second feature matrix and the pre-trained feature extraction matrix, the first feature matrix and the second feature matrix can be firstly vertically spliced ​​to obtain a full feature matrix; then, the full feature matrix is ​​subjected to feature extraction based on the feature extraction matrix to obtain a similarity matrix; finally, based on the similarity matrix and the activation function, a similarity score between the answer to be reviewed and the reference answer is generated.

[0157] Exemplarily, taking the generated first feature matrix as 768*8 and the generated second feature matrix as 1*8 as an example, the two matrices are vertically spliced ​​to obtain a 769*8 full feature matrix.

[0158] Furthermore, based on the feature extraction matrix, the full feature matrix is ​​subjected to feature extraction, that is, the 769*8 full feature matrix is ​​converted into a 1*2 similarity matrix that expresses the positive and negative correlations of samples. For example, based on the matrix operation rules, a 1*769 feature extraction matrix can be pre-multiplied to the full feature matrix, and then a 8*2 feature extraction matrix can be post-multiplied to generate the 1*2 similarity matrix. For example, [0.8, 0.2], where 0.8 represents the positive correlation between the answer information to be reviewed and the reference answer information; 0.2 represents the negative correlation between the answer information to be reviewed and the reference answer information.

[0159] After obtaining the similarity matrix, the similarity score between the answer to be reviewed and the reference answer can be generated based on the activation function, and the value range of the similarity score is [0, 1].

[0160] Here, the activation function is a function running on the neurons of the neural network model, responsible for mapping the input of the neurons to the output. Exemplarily, the 1*2 similarity matrix can be mapped to [0, 1] based on the commonly used Sigmoid function to generate the similarity score with a value range of [0, 1].

[0161] In a possible application scenario, after the similarity score is generated, a corresponding score can be generated based on the full score corresponding to the subjective question and the similarity score. For example, if the similarity score is 0.8 and the full score corresponding to the subjective question is 20 points, then the corresponding score is 16 points.

[0162] Furthermore, the corresponding score and the reference answer information may be displayed at a preset location on the client.

[0163] In a possible implementation, the neural network model may be an ALBERT model, and the training process of the neural network may be as follows: Figure 2 As shown, it includes the following steps:

[0164] S201. Obtain the sample answer information to be reviewed, the sample reference answer information corresponding to the sample answer information to be reviewed, and the annotation information of the sample answer information to be reviewed, wherein the annotation information is used to indicate the correctness of the sample answer information to be reviewed.

[0165] S202: Determine a first sample feature matrix based on the sample answer information to be reviewed, the sample reference answer information, and the neural network model to be trained.

[0166] S203: Determine a second sample feature matrix based on statistical feature information between the sample answer information to be reviewed and the sample reference answer information.

[0167] S204: Determine a similarity matrix between the answer to be reviewed and the reference answer based on the first sample feature matrix, the second sample feature matrix, and the feature extraction matrix to be trained.

[0168] S205. Based on the annotation information and the similarity matrix, determine the loss value in this training process. When the loss value is greater than a set threshold, adjust the model parameters of the neural network model and the feature extraction matrix, and re-train the neural network model.

[0169] The review method provided by the embodiment of the present disclosure uses a trained neural network model to extract the semantic association relationship between the answers to be reviewed and the reference answers to obtain a first feature matrix, and then combines the statistical features between the answers to be reviewed and the reference answers to generate the review results of the answers to be reviewed. Based on this method, the review results of the answers to be reviewed can be automatically generated based on the semantic association relationship and statistical features between the answers to be reviewed and the reference answers. Therefore, while improving the review speed, it avoids the influence of subjective color on the review results and improves the accuracy of the review results.

[0170] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0171] Based on the same inventive concept, a review device corresponding to the review method is also provided in the embodiment of the present disclosure. Since the principle of solving the problem by the device in the embodiment of the present disclosure is similar to the above-mentioned review method in the embodiment of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0172] Reference Figure 3 , which is a schematic diagram of a review device provided by an embodiment of the present disclosure, the device comprises: an acquisition module 301, a first determination module 302, a second determination module 303, and a third determination module 304; wherein,

[0173] The acquisition module 301 is used to acquire the answer information to be reviewed and the reference answer information;

[0174] A first determination module 302 is used to determine a first feature matrix for representing the semantic association relationship between the answer to be reviewed and the reference answer based on the trained neural network model, the answer information to be reviewed and the reference answer information;

[0175] A second determination module 303 is used to determine a second characteristic matrix for characterizing the statistical characteristic relationship between the answer to be reviewed and the reference answer based on the answer to be reviewed information and the reference answer information;

[0176] The third determination module 304 is used to determine the review result corresponding to the answer information to be reviewed based on the first feature matrix, the second feature matrix and a pre-trained feature extraction matrix.

[0177] In a possible implementation manner, the first determination module 302, when determining the first feature matrix for characterizing the semantic association relationship between the answer to be reviewed and the reference answer based on the trained neural network model, the answer information to be reviewed and the reference answer information, is used to:

[0178] Preprocessing the answer information to be reviewed and the reference answer information to obtain a representation matrix for representing the answer information to be reviewed and the reference answer information;

[0179] The representation matrix is ​​input into the neural network model to obtain the first feature matrix.

[0180] In a possible implementation manner, the first determining module 302, when preprocessing the answer information to be reviewed and the reference answer information to obtain a representation matrix for representing the answer information to be reviewed and the reference answer information, is used to:

[0181] Performing word segmentation processing on the answer information to be reviewed to obtain at least one first keyword to be screened corresponding to the answer to be reviewed, and performing word segmentation processing on the reference answer information to obtain at least one second keyword to be screened corresponding to the reference answer information;

[0182] Based on a plurality of stop words stored in a stop word library, determining a target keyword from the first keywords to be screened and the second keywords to be screened;

[0183] The word vector corresponding to each target keyword is determined, and the representation matrix is ​​determined according to the order in which the target keywords appear in the answer information to be reviewed and the reference answer information.

[0184] In a possible implementation manner, the first determination module 302, when determining the word vector corresponding to each target keyword, is used to:

[0185] Based on a pre-set keyword dictionary storing the correspondence between keywords and word vectors, the word vector corresponding to each target keyword is searched.

[0186] In a possible implementation manner, the first determination module 302, when determining the target keyword from the first keywords to be screened and the second keywords to be screened based on the multiple stop words stored in the stop word library, is configured to:

[0187] Matching the first keyword to be screened and the second keyword to be screened with a plurality of stop words stored in a stop word library respectively, and determining the keyword to be screened that is unsuccessfully matched;

[0188] In the case that the number of keywords included in the keywords to be screened that failed to match exceeds the preset number, based on the order in which the keywords to be screened that failed to match appear in the answer information to be reviewed, the first N keywords included in the keywords to be screened are used as the target keywords, where N is the preset number and N is a positive integer.

[0189] In a possible implementation manner, the first determination module 302, when determining the word vector corresponding to each target keyword and determining the representation matrix according to the order in which the target keywords appear in the answer information to be reviewed and the reference answer information, is used to:

[0190] Determine the word vector corresponding to the token;

[0191] The word vectors corresponding to the target keywords belonging to the answer information to be reviewed are concatenated in the order in which the target keywords appear in the answer information to be reviewed to obtain a first concatenated matrix; and the word vectors corresponding to the target keywords belonging to the reference answer information are concatenated in the order in which the target keywords appear in the reference answer information to obtain a second concatenated matrix;

[0192] The character vector corresponding to the marker, the first splicing matrix and the second splicing matrix are spliced ​​to obtain the representation matrix, and the marker is used to distinguish the first splicing matrix from the second splicing matrix.

[0193] In a possible implementation manner, the second determination module 303, when determining the second feature matrix for characterizing the statistical feature relationship between the answer to be reviewed and the reference answer based on the answer to be reviewed information and the reference answer information, is used to:

[0194] Based on the answer information to be reviewed and the reference answer information, determining statistical feature information between the information to be reviewed and the reference answer information;

[0195] The second feature matrix is ​​determined based on statistical feature information between the information to be reviewed and the reference answer information.

[0196] In a possible implementation manner, the statistical feature information includes multiple types of the following information:

[0197] The editing distance between the information to be reviewed and the reference answer information, the number of identical words and the number of different words in the information to be reviewed and the reference answer information, the word set similarity, the N-gram similarity, and the difference in text length between the information to be reviewed and the reference answer information.

[0198] In a possible implementation manner, the review result corresponding to the answer information to be reviewed includes a similarity score between the answer information to be reviewed and the reference answer information;

[0199] The third determination module 304, when determining the review result corresponding to the answer information to be reviewed based on the first feature matrix, the second feature matrix and the pre-trained feature extraction matrix, is used to:

[0200] Vertically concatenate the first characteristic matrix and the second characteristic matrix to obtain a full characteristic matrix;

[0201] Performing feature extraction on the full feature matrix based on the feature extraction matrix to obtain a similarity matrix;

[0202] Based on the similarity matrix and the activation function, a similarity score between the answer to be reviewed and the reference answer is generated.

[0203] In a possible implementation manner, the review device further includes a training module 305, and the training module 305 is used to:

[0204] Acquire the sample answer information to be reviewed, sample reference answer information corresponding to the sample answer information to be reviewed, and annotation information of the sample answer information to be reviewed, wherein the annotation information is used to indicate the correctness of the sample answer information to be reviewed;

[0205] Determine a first sample feature matrix based on the sample answer information to be reviewed, the sample reference answer information, and the neural network model to be trained;

[0206] Determining a second sample feature matrix based on statistical feature information between the sample answer information to be reviewed and the sample reference answer information;

[0207] Determine a similarity matrix between the answer to be reviewed and the reference answer based on the first sample feature matrix, the second sample feature matrix, and the feature extraction matrix to be trained;

[0208] Based on the annotation information and the similarity matrix, the loss value in this training process is determined. When the loss value is greater than a set threshold, the model parameters of the neural network model and the feature extraction matrix are adjusted, and the neural network model is retrained.

[0209] The review device provided by the embodiment of the present disclosure uses a trained neural network model to extract the semantic association relationship between the answers to be reviewed and the reference answers to obtain a first feature matrix, and then combines the statistical features between the answers to be reviewed and the reference answers to generate the review results of the answers to be reviewed. Based on this device, the review results of the answers to be reviewed can be automatically generated based on the semantic association relationship and statistical features between the answers to be reviewed and the reference answers. Therefore, while improving the review speed, it avoids the influence of subjective color on the review results and improves the accuracy of the review results.

[0210] Based on the same technical concept, the embodiment of the present disclosure also provides a computer device. Figure 4As shown, it is a schematic diagram of the structure of a computer device 400 provided in an embodiment of the present disclosure, including a processor 401, a memory 402, and a bus 403. Among them, the memory 402 is used to store execution instructions, including a memory 4021 and an external memory 4022; the memory 4021 here is also called an internal memory, which is used to temporarily store the operation data in the processor 401, and the data exchanged with the external memory 4022 such as a hard disk. The processor 401 exchanges data with the external memory 4022 through the memory 4021. When the computer device 400 is running, the processor 401 communicates with the memory 402 through the bus 403, so that the processor 401 executes the following instructions:

[0211] Obtain information on answers to be reviewed and reference answers;

[0212] Based on the trained neural network model, the answer information to be reviewed and the reference answer information, determining a first feature matrix for characterizing the semantic association relationship between the answer to be reviewed and the reference answer;

[0213] Based on the answer information to be reviewed and the reference answer information, determining a second feature matrix for characterizing the statistical feature relationship between the answer to be reviewed and the reference answer;

[0214] Based on the first feature matrix, the second feature matrix and a pre-trained feature extraction matrix, a review result corresponding to the answer information to be reviewed is determined.

[0215] In a possible implementation manner, the instructions executed by the processor 401 include determining a first feature matrix for characterizing the semantic association relationship between the answer to be reviewed and the reference answer based on the trained neural network model, the answer information to be reviewed, and the reference answer information, including:

[0216] Preprocessing the answer information to be reviewed and the reference answer information to obtain a representation matrix for representing the answer information to be reviewed and the reference answer information;

[0217] The representation matrix is ​​input into the neural network model to obtain the first feature matrix.

[0218] In a possible implementation manner, in the instructions executed by the processor 401, the answer information to be reviewed and the reference answer information are preprocessed to obtain a representation matrix for representing the answer information to be reviewed and the reference answer information, including:

[0219] Performing word segmentation processing on the answer information to be reviewed to obtain at least one first keyword to be screened corresponding to the answer to be reviewed, and performing word segmentation processing on the reference answer information to obtain at least one second keyword to be screened corresponding to the reference answer information;

[0220] Based on a plurality of stop words stored in a stop word library, determining a target keyword from the first keywords to be screened and the second keywords to be screened;

[0221] The word vector corresponding to each target keyword is determined, and the representation matrix is ​​determined according to the order in which the target keywords appear in the answer information to be reviewed and the reference answer information.

[0222] In a possible implementation, the instructions executed by the processor 401 to determine the word vector corresponding to each target keyword include:

[0223] Based on a pre-set keyword dictionary storing the correspondence between keywords and word vectors, the word vector corresponding to each target keyword is searched.

[0224] In a possible implementation, the instructions executed by the processor 401 include determining a target keyword from the first keywords to be screened and the second keywords to be screened based on a plurality of stop words stored in a stop word library, including:

[0225] Matching the first keyword to be screened and the second keyword to be screened with a plurality of stop words stored in a stop word library respectively, and determining the keyword to be screened that is unsuccessfully matched;

[0226] In the case that the number of keywords included in the keywords to be screened that failed to match exceeds the preset number, based on the order in which the keywords to be screened that failed to match appear in the answer information to be reviewed, the first N keywords included in the keywords to be screened are used as the target keywords, where N is the preset number and N is a positive integer.

[0227] In a possible implementation, the instructions executed by the processor 401 include determining a word vector corresponding to each target keyword, and determining the representation matrix according to the order in which the target keywords appear in the answer information to be reviewed and the reference answer information, including:

[0228] Determine the word vector corresponding to the token;

[0229] The word vectors corresponding to the target keywords belonging to the answer information to be reviewed are concatenated in the order in which the target keywords appear in the answer information to be reviewed to obtain a first concatenated matrix; and the word vectors corresponding to the target keywords belonging to the reference answer information are concatenated in the order in which the target keywords appear in the reference answer information to obtain a second concatenated matrix;

[0230] The character vector corresponding to the marker, the first splicing matrix and the second splicing matrix are spliced ​​to obtain the representation matrix, and the marker is used to distinguish the first splicing matrix from the second splicing matrix.

[0231] In a possible implementation manner, in the instructions executed by the processor 401, based on the answer information to be reviewed and the reference answer information, determining a second feature matrix for characterizing the statistical feature relationship between the answer to be reviewed and the reference answer includes:

[0232] Based on the answer information to be reviewed and the reference answer information, determining statistical feature information between the information to be reviewed and the reference answer information;

[0233] The second feature matrix is ​​determined based on statistical feature information between the information to be reviewed and the reference answer information.

[0234] In a possible implementation manner, in the instruction executed by the processor 401, the statistical feature information includes multiple types of the following information:

[0235] The editing distance between the information to be reviewed and the reference answer information, the number of identical words and the number of different words in the information to be reviewed and the reference answer information, the word set similarity, the N-gram similarity, and the difference in text length between the information to be reviewed and the reference answer information.

[0236] In a possible implementation manner, in the instruction executed by the processor 401, the review result corresponding to the answer information to be reviewed includes a similarity score between the answer information to be reviewed and the reference answer information;

[0237] The determining, based on the first feature matrix, the second feature matrix and a pre-trained feature extraction matrix, a review result corresponding to the answer information to be reviewed includes:

[0238] Vertically concatenate the first characteristic matrix and the second characteristic matrix to obtain a full characteristic matrix;

[0239] Performing feature extraction on the full feature matrix based on the feature extraction matrix to obtain a similarity matrix;

[0240] Based on the similarity matrix and the activation function, a similarity score between the answer to be reviewed and the reference answer is generated.

[0241] In a possible implementation manner, the instructions executed by the processor 401 further include training the neural network model according to the following method:

[0242] Acquire the sample answer information to be reviewed, sample reference answer information corresponding to the sample answer information to be reviewed, and annotation information of the sample answer information to be reviewed, wherein the annotation information is used to indicate the correctness of the sample answer information to be reviewed;

[0243] Determine a first sample feature matrix based on the sample answer information to be reviewed, the sample reference answer information, and the neural network model to be trained;

[0244] Determining a second sample feature matrix based on statistical feature information between the sample answer information to be reviewed and the sample reference answer information;

[0245] Determine a similarity matrix between the answer to be reviewed and the reference answer based on the first sample feature matrix, the second sample feature matrix, and the feature extraction matrix to be trained;

[0246] Based on the annotation information and the similarity matrix, the loss value in this training process is determined. When the loss value is greater than a set threshold, the model parameters of the neural network model and the feature extraction matrix are adjusted, and the neural network model is retrained.

[0247] The present disclosure also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the review method described in the above method embodiment are executed. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0248] The computer program product of the review method provided in the embodiment of the present disclosure includes a computer-readable storage medium storing a program code, and the instructions included in the program code can be used to execute the steps of the review method described in the above method embodiment. For details, please refer to the above method embodiment, which will not be repeated here. The computer program product can be implemented in hardware, software or a combination thereof. In an optional embodiment, the computer program product is specifically embodied as a computer storage medium. In another optional embodiment, the computer program product is specifically embodied as a software product, such as a software development kit (Software Development Kit, SDK) and the like.

[0249] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the device or unit can be electrical, mechanical or other forms.

[0250] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0251] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0252] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0253] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The protection scope of the present disclosure is not limited thereto. Although the present disclosure is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present disclosure, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A review method, characterized in that: include: Obtain information on answers to be reviewed and reference answers; Based on the trained neural network model, the answer information to be reviewed and the reference answer information, determining a first feature matrix for characterizing the semantic association relationship between the answer to be reviewed and the reference answer; Based on the answer information to be reviewed and the reference answer information, determine a second feature matrix for characterizing the statistical feature relationship between the answer information to be reviewed and the reference answer; the second feature matrix is ​​determined based on the statistical feature information between the answer information to be reviewed and the reference answer information, and the statistical feature information includes multiple types of the following information: the edit distance between the answer information to be reviewed and the reference answer information, the number of identical words and the number of different words in the answer information to be reviewed and the reference answer information, word set similarity, N-gram similarity, and text length difference between the answer information to be reviewed and the reference answer information; The first feature matrix and the second feature matrix are vertically spliced ​​to obtain a full feature matrix; features are extracted from the full feature matrix based on a pre-trained feature extraction matrix to obtain a similarity matrix; Based on the similarity matrix and the activation function, a review result corresponding to the answer information to be reviewed is generated, and the review result corresponding to the answer information to be reviewed includes a similarity score between the answer information to be reviewed and the reference answer information.

2. The method according to claim 1, characterized in that The determining, based on the trained neural network model, the answer information to be reviewed and the reference answer information, of a first feature matrix for characterizing the semantic association relationship between the answer to be reviewed and the reference answer comprises: Preprocessing the answer information to be reviewed and the reference answer information to obtain a representation matrix for representing the answer information to be reviewed and the reference answer information; The representation matrix is ​​input into the neural network model to obtain the first feature matrix.

3. The method according to claim 2, characterized in that The preprocessing of the answer information to be reviewed and the reference answer information to obtain a representation matrix for representing the answer information to be reviewed and the reference answer information includes: Performing word segmentation processing on the answer information to be reviewed to obtain at least one first keyword to be screened corresponding to the answer to be reviewed, and performing word segmentation processing on the reference answer information to obtain at least one second keyword to be screened corresponding to the reference answer information; Based on a plurality of stop words stored in a stop word library, determining a target keyword from the first keywords to be screened and the second keywords to be screened; The word vector corresponding to each target keyword is determined, and the representation matrix is ​​determined according to the order in which the target keywords appear in the answer information to be reviewed and the reference answer information.

4. The method according to claim 3, characterized in that: Determining the word vector corresponding to each target keyword includes: Based on a pre-set keyword dictionary storing the correspondence between keywords and word vectors, the word vector corresponding to each target keyword is searched.

5. The method according to claim 3, characterized in that: The step of determining a target keyword from the first keywords to be screened and the second keywords to be screened based on the multiple stop words stored in the stop word library includes: Matching the first keyword to be screened and the second keyword to be screened with a plurality of stop words stored in a stop word library respectively, and determining keywords to be screened that are unsuccessfully matched; In the case that the number of keywords included in the keywords to be screened that failed to match exceeds the preset number, based on the order in which the keywords to be screened that failed to match appear in the answer information to be reviewed, the first N keywords included in the keywords to be screened are used as the target keywords, where N is the preset number and N is a positive integer.

6. The method according to claim 3, characterized in that: The step of determining the word vector corresponding to each target keyword and determining the representation matrix according to the order in which the target keywords appear in the answer information to be reviewed and the reference answer information includes: Determine the word vector corresponding to the token; The word vectors corresponding to the target keywords belonging to the answer information to be reviewed are concatenated in the order in which the target keywords appear in the answer information to be reviewed to obtain a first concatenated matrix; and the word vectors corresponding to the target keywords belonging to the reference answer information are concatenated in the order in which the target keywords appear in the reference answer information to obtain a second concatenated matrix; The character vector corresponding to the marker, the first splicing matrix and the second splicing matrix are spliced ​​to obtain the representation matrix, and the marker is used to distinguish the first splicing matrix from the second splicing matrix.

7. The method according to claim 1, characterized in that The determining, based on the answer information to be reviewed and the reference answer information, of a second characteristic matrix for characterizing the statistical characteristic relationship between the answer to be reviewed and the reference answer comprises: Based on the answer information to be reviewed and the reference answer information, determining statistical feature information between the answer information to be reviewed and the reference answer information; The second feature matrix is ​​determined based on statistical feature information between the answer information to be reviewed and the reference answer information.

8. The method according to any one of claims 1 to 7, characterized in that: The method also includes training the neural network model according to the following method: Acquire sample answer information to be reviewed, sample reference answer information corresponding to the sample answer information to be reviewed, and annotation information of the sample answer information to be reviewed, wherein the annotation information is used to indicate the correctness of the sample answer information to be reviewed; Determine a first sample feature matrix based on the sample answer information to be reviewed, the sample reference answer information, and the neural network model to be trained; Determining a second sample feature matrix based on statistical feature information between the sample answer information to be reviewed and the sample reference answer information; Determine a similarity matrix between the answer to be reviewed and the reference answer based on the first sample feature matrix, the second sample feature matrix, and the feature extraction matrix to be trained; Based on the annotation information and the similarity matrix, the loss value in this training process is determined. When the loss value is greater than a set threshold, the model parameters of the neural network model and the feature extraction matrix are adjusted, and the neural network model is retrained.

9. A review device, characterized in that: include: The acquisition module is used to obtain the answer information to be reviewed and the reference answer information; A first determination module is used to determine a first feature matrix for characterizing the semantic association relationship between the answer to be reviewed and the reference answer based on the trained neural network model, the answer information to be reviewed and the reference answer information; A second determination module is used to determine a second feature matrix for characterizing the statistical feature relationship between the answer to be reviewed and the reference answer based on the answer information to be reviewed and the reference answer information; the second feature matrix is ​​determined based on the statistical feature information between the answer information to be reviewed and the reference answer information, and the statistical feature information includes multiple types of the following information: the edit distance between the answer information to be reviewed and the reference answer information, the number of identical words and the number of different words in the answer information to be reviewed and the reference answer information, word set similarity, N-gram similarity, and text length difference between the answer information to be reviewed and the reference answer information; A third determination module is used to vertically splice the first feature matrix and the second feature matrix to obtain a full feature matrix; extract features from the full feature matrix based on a pre-trained feature extraction matrix to obtain a similarity matrix; Based on the similarity matrix and the activation function, a review result corresponding to the answer information to be reviewed is generated, and the review result corresponding to the answer information to be reviewed includes a similarity score between the answer information to be reviewed and the reference answer information.

10. A computer device, characterized in that: include: A processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the processor is used to execute the machine-readable instructions stored in the memory. When the machine-readable instructions are executed by the processor, the processor executes the steps of the review method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program. When the computer program is executed by a computer device, the computer device executes the steps of the review method according to any one of claims 1 to 8.

Images