Method, device and equipment for evaluating quality of title answer and storage medium

Abstract

The application relates to the computer field, in particular to the artificial intelligence field, and provides a question answering quality evaluation method and device, equipment and a storage medium. The method comprises the following steps: obtaining answering results input by target objects for a same subjective question; obtaining target similarity vectors of the answering results based on the intermediate similarity vectors of each answering result and the intermediate similarity vectors of the answering results, wherein each target similarity vector represents a first similarity degree distribution of the answering result and a second similarity degree distribution between the first similarity degree distributions of the answering results; determining answering quality evaluation features of the corresponding answering results based on the obtained target similarity vectors, and inputting the answering quality evaluation features into a target evaluation model to obtain answering quality evaluation results of the corresponding answering results. The feature extraction is performed in the form of vector operation, the operation amount is small, the delay is low, and the application scenarios are wide.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computers, in particular to the field of artificial intelligence, and provides a question answer quality evaluation method and device, equipment and a storage medium. BACKGROUND

[0002] In recent years, with the development of computer technology, automatic marking technology has been gradually applied to examinations, which not only reduces the workload of teachers and improves the marking efficiency, but also ensures the fairness and impartiality of examinations. The automatic marking technology can realize automatic marking for objective questions such as multiple-choice questions, fill-in-the-blank questions and judgment questions, but subjective questions have always been a technical problem that needs to be overcome in automatic marking technology due to the diversity of answers and the flexibility of evaluation criteria. Therefore, the automatic marking technology for subjective questions has become a research focus in this field.

[0003] At the beginning, a scheme combining subject word matching and off-topic detection model is used to determine whether the subjective questions in the test paper are off-topic. However, in the application stage, the rule-based subject word matching cannot cover all subject words, and the recall effect is not good, which further affects the detection accuracy of the model.

[0004] In order to improve the detection accuracy of the model, the current mainstream off-topic detection method is to use an off-topic detection model based on natural language processing (NLP) technology to extract the correlation features between the test questions and the students' answers, and then based on the correlation features, the off-topic detection of the answer results of each student is performed.

[0005] Although this detection method improves the detection accuracy of the model to some extent, the huge neural network also brings new problems of large computation and high delay, which cannot meet the detection needs of application scenarios with high real-time performance. In addition, due to the immaturity of the current NLP technology, the off-topic detection model cannot accurately extract the corresponding correlation features from the application scenarios not involved in the training stage, which further reduces the detection accuracy and robustness of the model. SUMMARY

[0006] The embodiments of the present application provide a question answer quality evaluation method, device, equipment and storage medium to solve the problems of large computation and high delay in feature extraction, and only extracting features of specific scenarios.

[0007] In a first aspect, the embodiments of the present application provide a question answer quality evaluation method, comprising:

[0008] obtain the answer results input by each target object for the same subjective question respectively, and obtain the intermediate similarity vectors corresponding to each answer result respectively, wherein each intermediate similarity vector represents a first similarity degree distribution between the corresponding answer result and the answer results;

[0009] For each intermediate similarity vector, the following operations are performed respectively: based on the intermediate similarity vector of one answer result and the intermediate similarity vectors of the answer results, a corresponding target similarity vector is obtained, wherein each target similarity vector represents a second similarity degree distribution between the first similarity degree distribution of the one answer result and the first similarity degree distributions of the answer results;

[0010] Based on each target similarity vector obtained respectively, the answer quality evaluation feature of the corresponding answer result is determined, and each answer quality evaluation feature is input into a target evaluation model to obtain the answer quality evaluation result of the corresponding answer result output by the target evaluation model.

[0011] In a second aspect, the embodiments of the present application further provide an evaluation device for question answer quality, comprising:

[0012] The vectorization processing unit is configured to obtain the answer results input by each target object for the same subjective question respectively, and obtain the intermediate similarity vectors corresponding to each answer result respectively, wherein each intermediate similarity vector represents a first similarity degree distribution between the corresponding answer result and the answer results;

[0013] The feature extraction unit is configured to, for each intermediate similarity vector obtained, perform the following operations respectively: based on the intermediate similarity vector of one answer result and the intermediate similarity vectors of the answer results, a corresponding target similarity vector is obtained, wherein each target similarity vector represents a second similarity degree distribution between the first similarity degree distribution of the one answer result and the first similarity degree distributions of the answer results;

[0014] The answer quality evaluation unit is configured to, based on each target similarity vector obtained respectively, determine the answer quality evaluation feature of the corresponding answer result, and input each answer quality evaluation feature into a target evaluation model to obtain the answer quality evaluation result of the corresponding answer result output by the target evaluation model.

[0015] Optionally, the answer quality evaluation unit is configured to:

[0016] The j1-dimensional feature vector of the target similarity vector is obtained in sequence, and each time the j1-dimensional feature vector is obtained, the following operations are performed: the j1-dimensional feature vector obtained currently is compared with the neighbor threshold value, and based on the association relationship represented by the comparison result, the j1-dimensional feature vector of the neighbor vector is determined.11 a comparison result representing whether there is a correlation between the corresponding answer result and the one answer result, 1≤j1≤N, 1≤j 11 ≤N, N being the total number of the answer results;

[0017] Based on the obtained N-dimensional feature vector, a neighbor vector of the one answer result is obtained.

[0018] Optionally, the answer quality assessment unit is configured to:

[0019] When the currently obtained j1th-dimensional feature vector is less than the neighbor threshold value, based on the correlation represented by the comparison result, the j 11 The N-dimensional feature vector is a first feature vector value;

[0020] When the currently obtained j1th-dimensional feature vector exceeds the neighbor threshold value, based on the correlation represented by the comparison result, the j 11 The N-dimensional feature vector is a second feature vector value.

[0021] Optionally, the answer quality assessment unit is configured to:

[0022] The dimensional feature vectors of the one neighbor vector are added, and the sum of the feature vectors is determined as the neighbor number.

[0023] Optionally, the answer quality assessment unit is configured to:

[0024] Using the neighbor vector of the one answer result, a target similarity vector of the one answer result is masked to obtain a neighbor weight vector, wherein the one neighbor weight vector contains multiple dimensional feature vectors, each dimensional feature vector corresponds to one of the multiple answer results, and each dimensional feature vector represents the correlation degree between the corresponding answer result and the one answer result.

[0025] Based on the one neighbor weight vector and the neighbor number, a corresponding average neighbor weight is obtained.

[0026] Optionally, the vectorization processing unit is configured to:

[0027] Respectively vectorizing the answer results to obtain corresponding answer result vectors;

[0028] For each obtained answer result vector, the following operations are performed: based on the answer result vector of the one answer result and the answer result vectors of the answer results, a corresponding intermediate similarity vector is obtained.

[0029] Optionally, the vectorization processing unit is configured to:

[0030] The answer result vectors of the answer results are sequentially traversed, and when the answer result vector of the jth answer result is traversed, the jth dimension of the intermediate similarity vector is determined based on the answer result vector of the one answer result and the answer result vector of the jth answer result. 111 dimensional feature vector, where 1≤j≤N, 1≤j 111 ≤N, N is the total number of the answer results;

[0031] The intermediate similarity vector of the one answer result is obtained based on the obtained N-dimensional feature vector.

[0032] In a third aspect, an embodiment of the present application further provides a computer device, including a processor and a memory, wherein the memory stores program code, when the program code is executed by the processor, the processor executes the steps of the question answer quality evaluation method.

[0033] In a fourth aspect, an embodiment of the present application further provides a computer readable storage medium, including program code, when the program product is run on a computer device, the program code is used to make the computer device execute the steps of the question answer quality evaluation method.

[0034] In a fifth aspect, an embodiment of the present application further provides a computer program product, including computer instructions, when the computer instructions are executed by a processor, the steps of the question answer quality evaluation method are executed.

[0035] The present application has the following beneficial effects:

[0036] The embodiments of the present application provide a question answer quality evaluation method, device, equipment and storage medium, the method includes: obtaining answer results input by each target object for a same subjective question, and obtaining intermediate similarity vectors corresponding to each answer result, wherein each intermediate similarity vector represents a first similarity distribution between the corresponding answer result and each answer result; for each obtained intermediate similarity vector, the following operations are performed: obtaining a target similarity vector based on the intermediate similarity vector of one answer result and the intermediate similarity vectors of each answer result, wherein each target similarity vector represents: the first similarity degree distribution of the answer result, and the second similarity degree distribution between the first similarity degree distribution of each answer result; then, based on each obtained target similarity vector, determining an answer quality evaluation feature of the corresponding answer result, and inputting each answer quality evaluation feature into a target evaluation model to obtain an answer quality evaluation result of the corresponding answer result output by the target evaluation model.

[0037] The traditional off-topic detection model is based on NLP technology, extracts the correlation features between the test questions and the student answers, and has the problems of large operation amount, high delay, and only being able to extract features in specific scenarios.

[0038] To solve the above problems, the embodiments of the present application use several steps of vector operation to obtain the answer quality evaluation features of each answer result, which has small operation amount, low delay, is more suitable for application in application scenarios with high real-time requirements, and this feature extraction method is suitable for different application scenarios, reduces the limitation of application scenarios on feature extraction, and is beneficial to improve the detection accuracy and robustness of the model.

[0039] Other features and advantages of the present application will be set forth in the following description, and in part will become apparent from the description, or can be learned by practice of the present application. The objects and other advantages of the present application will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings. BRIEF DESCRIPTION OF DRAWINGS

[0040] The accompanying drawings, which are included to provide a further understanding of the present application and are incorporated in and constitute a part of this application, illustrate embodiments of the present application and serve to explain the present application. In the drawings:

[0041] Figure 1 An optional schematic diagram for one application scenario in the embodiments of the present application;

[0042] Figure 2A A flowchart of the question answer quality evaluation method provided by the embodiments of the present application;

[0043] Figure 2B A logic diagram of the question answer quality evaluation method provided by the embodiments of the present application;

[0044] Figure 2C A flowchart of obtaining the intermediate similarity vector corresponding to each answer result provided by the embodiments of the present application;

[0045] Figure 2D A flowchart of determining the answer quality evaluation features of one answer result provided by the embodiments of the present application;

[0046] Figure 2E A node association diagram provided by the embodiments of the present application;

[0047] Figure 3A A flowchart of performing answer quality evaluation on the answer results given by each student in a spoken language test provided by the embodiments of the present application;

[0048] Figure 3BA logical diagram for evaluating the answer quality of the answer results given by each student in a spoken language test is provided in the embodiments of the present application.

[0049] Figure 4A A flowchart for evaluating the answer quality of the compositions written by each student in a Chinese language test is provided in the embodiments of the present application.

[0050] Figure 4B A logical diagram for evaluating the answer quality of the compositions written by each student in a Chinese language test is provided in the embodiments of the present application.

[0051] Figure 5 A structural diagram of an evaluation device for the answer quality of a question is provided in the embodiments of the present application.

[0052] Figure 6 A structural diagram of a computer device is provided in the embodiments of the present application.

[0053] Figure 7 A structural diagram of a computer device is provided in the embodiments of the present application. DETAILED DESCRIPTION

[0054] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions of the present application will be described below in connection with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments described in the present application, all other embodiments obtained by those of ordinary skill in the art without any creative work fall within the scope of protection of the present application.

[0055] Some terms in the embodiments of the present application are explained below to facilitate the understanding of those of ordinary skill in the art.

[0056] 1. Artificial Intelligence (AI):

[0057] Artificial intelligence is to use digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use the knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which aims to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to design and implement principles and methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0058] Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, both hardware and software level technology. Artificial intelligence basic technology generally includes, such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics and other technologies; Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology and machine learning / deep learning and other fields.

[0059] With the research and progress of artificial intelligence technology, artificial intelligence has been studied and applied in many fields, such as common smart home, intelligent customer service, virtual assistants, smart speakers, intelligent marketing, driverless vehicles, autonomous vehicles, robots, intelligent medical care, etc. It is believed that with the development of technology, artificial intelligence will be applied in more fields and play an increasingly important role.

[0060] 2、Machine learning:

[0061] Machine learning is a multi-disciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It is a discipline that studies how computers simulate or implement human learning behavior to acquire new knowledge or skills, and reorganize existing knowledge structure to continuously improve their performance.

[0062] Machine learning is the core of artificial intelligence and the fundamental approach to making computers intelligent. Its applications are widespread in various fields of artificial intelligence, including deep learning, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0063] 3、Vector Space Model (VSM):

[0064] Text is a string composed of characters and punctuation marks, including words, phrases, sentences, paragraphs, and chapters. However, the string itself is unstructured data and cannot be directly applied to statistical machine learning models. Therefore, text needs to be converted into a vector that can express the semantic meaning of the text, and VSM is the simplest way to represent text.

[0065] 4、Mask:

[0066] In the field of image processing, selected images, graphics or objects are commonly used to block part or all of the image to be processed to control the image processing area. The specific image, graphic or object used for covering is called "mask". When processing digital images, the mask can also be a two-dimensional matrix array or a multi-value image.

[0067] 5、Decision tree:

[0068] A decision tree is a tree structure similar to a flowchart, in which each non-leaf node inside the tree represents a test on an attribute (e.g., a flipped coin shows heads, a flipped coin shows tails), each branch represents a test result, each leaf node represents a class label, and each path from the root node to a leaf node represents a classification rule.

[0069] The design idea of the embodiments of the present application will be briefly introduced as follows:

[0070] In recent years, with the development of computer technology, automatic marking technology has been gradually applied to examinations, which not only reduces the workload of teachers and improves the marking efficiency, but also ensures the fairness and impartiality of examinations. The automatic marking technology can realize automatic marking for objective questions such as multiple-choice questions, fill-in-the-blank questions and true-or-false questions, but subjective questions have been a technical difficulty to be overcome in the automatic marking technology due to the diversity of answers and the flexibility of evaluation criteria. Therefore, the automatic marking technology for subjective questions has become a research focus in the field.

[0071] At the beginning, a scheme combining subject word matching and off-topic detection model is used to determine whether the subjective questions in the test paper are off-topic. However, in the application stage, the rule-based subject word matching cannot cover all subject words, and the recall effect is not good, which further affects the detection accuracy of the model.

[0072] In order to improve the detection accuracy of the model, the current mainstream off-topic detection method is to use an off-topic detection model based on natural language processing (NLP) technology to extract the relevance features between the test questions and the students' answers, and then based on the relevance features, to detect the off-topic of each student's answer.

[0073] Although this detection method improves the detection accuracy of the model to some extent, the huge neural network also brings new problems of large computation and high delay, which cannot meet the detection needs of application scenarios with high real-time requirements. In addition, due to the immaturity of the current NLP technology, the off-topic detection model cannot accurately extract the corresponding relevance features from the application scenarios not involved in the training stage, which reduces the detection accuracy and robustness of the model.

[0074] In view of this, the embodiment of the present application proposes a question answer quality evaluation method, device, equipment and storage medium. The method comprises: obtaining the answer results input by each target object for the same subjective question, and obtaining the intermediate similarity vectors corresponding to each answer result, wherein each intermediate similarity vector represents the first similarity distribution between the corresponding answer result and each answer result; for each obtained intermediate similarity vector, the following operations are performed: based on the intermediate similarity vector of one answer result and the intermediate similarity vectors of each answer result, the corresponding target similarity vector is obtained, wherein each target similarity vector represents the second similarity distribution between the first similarity degree distribution of the answer result and the first similarity degree distribution of each answer result; then, based on each obtained target similarity vector, the answer quality evaluation feature of the corresponding answer result is determined, and each answer quality evaluation feature is input into a target evaluation model to obtain the answer quality evaluation result of the corresponding answer result output by the target evaluation model.

[0075] The traditional off-topic detection model is based on NLP technology, extracts the correlation features between the test questions and the student answers, and has the problems of large calculation amount, high delay and only being able to extract features in specific scenarios when extracting features.

[0076] In order to solve the above problems, the embodiment of the present application uses several steps of vector operation to obtain the answer quality evaluation features of each answer result, which has small calculation amount, low delay, is more suitable for application in application scenarios with high real-time requirements, and this feature extraction method is suitable for different application scenarios, reduces the limitation of application scenarios on feature extraction, and is beneficial to improve the detection accuracy and robustness of the model.

[0077] The preferred embodiments of the present application are described below in conjunction with the drawings of the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application, and the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0078] Figure 1 One of the application scenarios is shown, which includes two physical terminal devices 110 and a server 130, and each physical terminal device 110 establishes a communication connection with the server 130 through a wired network or a wireless network.

[0079] Among them, the physical terminal device 110 of the embodiment of the present application is a computer device used by a user. The computer device includes but is not limited to a mobile phone, a computer, a smart voice interaction device, a smart home appliance, a vehicle-mounted terminal, an aircraft, etc.

[0080] The server 130 in the embodiments of the present application can be a stand-alone physical server, a server cluster or a distributed system composed of multiple physical servers, a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and basic cloud computing services such as big data and artificial intelligence platforms, and the like, which are not limited in the present application.

[0081] Each physical terminal device 110 presents an electronic test paper 120 to the corresponding target object, the electronic test paper including multiple objective questions and multiple subjective questions, and each target object inputs an answer to a subjective question, and the corresponding physical terminal device 110 sends the answer result of each target object to the same subjective question to the server 130.

[0082] The obtained answer results are vectorized using the answer quality evaluation system deployed on the server 130 to obtain corresponding answer result vectors, and then the answer quality evaluation features of each answer result are determined by performing feature extraction on each answer result vector, and each answer quality evaluation feature is input into the target evaluation model inside the system to obtain the answer quality evaluation result of the corresponding answer result.

[0083] Referring to Figure 2A the flowchart and Figure 2B the logical diagram shown, the process of evaluating the answer quality of each target object inputting an answer to the same subjective question using the evaluation method of the answer quality of the question proposed in the embodiments of the present application is as follows:

[0084] S201: The answer quality evaluation system obtains the answer results of each target object inputting an answer to the same subjective question, and obtains the intermediate similarity vectors corresponding to each answer result, respectively, wherein each intermediate similarity vector represents the first similarity distribution between the corresponding answer result and each answer result.

[0085] The examination includes various forms such as written test and oral test, when the subjective question is a written test question, the answer results input by each target object are in text format, and the corresponding intermediate similarity vectors are obtained by performing vectorization processing and other operations on each answer result; when the subjective question is an oral test question, the answer results input by each target object are in audio format, and the automatic speech recognition (ASR) technology is used to convert the audio format answer results into text format answer results (if not specified, the answer results mentioned below are all in text format), and then the corresponding intermediate similarity vectors are obtained by performing vectorization processing and other operations on each answer result.

[0086] Next, refer to Figure 2C the flowchart shown, and continue to introduce the process of obtaining the intermediate similarity vector corresponding to each answer result.

[0087] S2011: The answer quality evaluation system respectively vectorizes each answer result to obtain the corresponding answer result vector.

[0088] Text is a string composed of characters and punctuation marks, including words, phrases, sentences, paragraphs, and chapters. However, the string itself is unstructured data and cannot be directly used in statistical machine learning models, so each answer result needs to be vectorized to convert the text into a vector that can express the semantic meaning of the text.

[0089] As Figure 2B shown, the feature items of each word appearing in all answer results are counted, the number of feature items is determined as the dimension of the answer result vector, and the corresponding VSM is constructed; then, the VSM is used to count the number of occurrences of each feature item in each answer result to obtain the corresponding feature dimension.

[0090] Since repeated words are likely to appear in answer results (for example, students often repeat themselves during oral exams, resulting in the same feature item appearing multiple times in this answer), the following rules are established to simplify the vector: if the number of occurrences is 0, the corresponding dimension is set to 0, indicating that the answer result does not contain the feature item; if the number of occurrences is greater than 0, the corresponding dimension is set to 1, indicating that the answer result contains the feature item.

[0091] Using the above rules, the obtained feature dimensions are vectorized to obtain the corresponding answer result vectors; then, by concatenating each answer result vector, the corresponding answer result matrix A is obtained. The size of the answer result matrix A is N x M, N is the total number of answer results, M is the total number of feature items, and A i represents the answer result vector of the i-th answer result, and A[i,j 1111 ] represents the j 1111 th feature vector of the i-th answer result vector.

[0092] For example, in an oral exam, one student's answer result is "yes, yes, I have", and the word stem of the entire exam is "yes no i do have be to like". The vectorization of this answer result is shown in Table 1:

[0093] Table 1

[0094] the answer result of one of the students yes, yes, I have word stems of the entire exam yes no i do have be to like a characteristic dimension of the answer result 2 0 2 0 1 0 0 0 an answer result vector of the answer result 1 0 1 0 1 0 0

[0095] S2012: For each obtained answer result, the answer quality evaluation system performs the following operations: based on the answer result vector of one answer result and the answer result vectors of each answer result, the corresponding intermediate similarity vector is obtained.

[0096] As shown in formula 1, for each obtained answer result, the answer quality evaluation system performs the following operations: sequentially traversing the answer result vectors of each answer result, when traversing the answer result vector of the jth answer result, based on the answer result vector of the answer result and the answer result vector of the jth answer result, the jth feature vector of the intermediate similarity vector is determined, where 1≤j≤N, 1≤j 111 ≤N, N is the total number of the answer results; 111

[0097] Based on the obtained N-dimensional feature vector, the intermediate similarity vector of the answer result is obtained, and by splicing each intermediate similarity vector, the corresponding intermediate similarity matrix B is obtained.

[0098]

[0099] Where, A normal is the normalized answer result matrix, A normal T is the transpose matrix of A normal ; A[i,j 1111 ] represents the j 1111 dimensional feature vector of the ith answer result vector, A[i,k] represents the k dimensional feature vector of the ith answer result vector, and N is the total number of the answer results.

[0100] The size of the intermediate similarity matrix B is N×N, B i represents the intermediate similarity vector of the ith answer result, B[i,j 111 ] represents the j 111 dimensional feature vector based on the ith intermediate similarity vector, which is determined based on the cosine distance, Euclidean distance, etc. between the answer result vector of the ith answer result and the answer result vector of the jth answer result.

[0101] S202: For each obtained intermediate similarity vector, the answer quality evaluation system performs the following operations: based on the intermediate similarity vector of one answer result and the intermediate similarity vectors of each answer result, the corresponding target similarity vector is obtained, wherein each target similarity vector represents: the first similarity degree distribution of the answer result, and the second similarity degree distribution between the first similarity degree distribution of each answer result.

[0102] ​As shown in formula 2, the intermediate similarity vectors of each answer result are sequentially obtained, and each time the answer quality evaluation system performs the following operation: based on the intermediate similarity vector of the jth obtained answer result, the jth1 dimensional feature vector of the target similarity vector is determined in combination with the intermediate similarity vector of one answer result, where 1≤j≤N, 1≤j1≤N, and N is the total number of answer results.

[0103] Based on the obtained N-dimensional feature vector, the target similarity vector of the answer result is obtained, and the corresponding target similarity matrix T is obtained by splicing each target similarity vector.

[0104]

[0105] Where B normal is the normalized intermediate similarity matrix, B normal T is the transpose matrix of B normal . B i represents the intermediate similarity vector of the ith answer result, then B[i,j 111 ] represents the j 111 th dimensional feature vector based on the ith intermediate similarity vector, and the cosine distance, Euclidean distance, etc. between the answer result vector based on the ith answer result and the answer result vector of the jth answer result is determined. Similarly, B[i,k] represents the kth dimensional feature vector based on the ith intermediate similarity vector.

[0106] The size of the target similarity matrix T is N×N, N is the total number of answer results, and T i represents the target similarity vector of the ith answer result, then T[i,j1] represents the jth1 dimensional feature vector of the ith answer result, which is determined based on the first similarity degree distribution of the ith answer result and the first similarity degree distribution of the jth answer result.

[0107] The target similarity vector is used to measure the connection weight between two answer results. The connection weight can originally be determined using the answer result vector, but the vector similarity of VSM is not robust enough, and off-topic answers may be misjudged as on-topic answers. Therefore, in order to overcome the above problems, the embodiments of the present application refer to the idea of collaborative filtering, and when determining a target similarity vector, whether the distribution between the first similarity degree distribution of the ith answer result and the first similarity degree distribution of each answer result is similar is determined.

[0108] S203: The answer quality evaluation system determines the answer quality evaluation features of the corresponding answer results based on the obtained target similarity vectors respectively, and inputs each answer quality evaluation feature into the target evaluation model to obtain the answer quality evaluation results of the corresponding answer results output by the target evaluation model.

[0109] As shown in Figure 2D , for each target similarity vector, the answer quality evaluation features of the corresponding answer results are obtained by performing the following operations:

[0110] S2031: The answer quality evaluation system determines a neighbor vector based on the target similarity vector of one answer result and the set neighbor threshold value; one neighbor vector contains multiple dimensional feature vectors, each dimensional feature vector corresponds to one of the multiple answer results, and each dimensional feature vector represents whether there is an association relationship between the corresponding answer result and the answer result.

[0111] As shown in formula 3, the feature vectors of each dimension of the target similarity vector are obtained in turn, and each time the answer quality evaluation system performs the following operations: compares the j1th feature vector obtained at the current time with the neighbor threshold value, and determines the j1th feature vector of the neighbor vector based on the association relationship represented by the comparison result, where 1≤j1≤N, 1≤j 11 ≤N, and N is the total number of answer results. 11

[0112] Based on the obtained N-dimensional feature vectors, the neighbor vector of the one answer result is obtained, and by splicing each neighbor vector, the corresponding neighbor matrix M is obtained.

[0113]

[0114]

[0115]

[0116] where D i is the number of neighbors of the ith answer result, the size of the neighbor matrix M is N×N, N is the total number of answer results, M[i,j 11 ] is the j 11 th feature vector of the ith neighbor vector, and a represents the neighbor threshold value.

[0117] Specifically, based on the association relationship represented by the comparison result, the j 11 th feature vector of the neighbor vector is determined as follows:

[0118] ​When the jth dimension of the currently obtained feature vector is less than the neighbor threshold value, the jth dimension of the neighbor vector is determined based on the correlation represented by the comparison result. 11 The jth dimension of the feature vector is the first feature vector value.

[0119] When the jth dimension of the currently obtained feature vector exceeds the neighbor threshold value, the jth dimension of the neighbor vector is determined based on the correlation represented by the comparison result. 11 The jth dimension of the feature vector is the second feature vector value.

[0120] Suppose that there are 2000 students participating in the current exam, and the target similarity matrix generated based on the answer results given by each student is The neighbor threshold value a is 2, and the obtained neighbor matrix is

[0121] For an exam, the proportion of off-topic answer results is about 2%. By using the statistical characteristics that the semantic similarity between off-topic answer results and on-topic answer results is low, and the semantic similarity between on-topic answer results is high, and by taking the answer results as nodes and the connection weights as edges between the nodes, the neighbor matrix D is converted into a node association graph as shown in Figure 2E , so as to more intuitively and visually show the correlation between the answer results, which is conducive to the determination of the answer quality evaluation features of the answer quality evaluation system.

[0122] Figure 2E The numbers in the above table represent student numbers, and it can be seen from Figure 2E that basically each on-topic answer result has a neighbor edge and is distributed in the overlapping area at the center of the graph, while the off-topic answer results are independently distributed around the graph and have no adjacent answer results. Therefore, the number of neighbors and the average neighbor weight of each answer result can be determined as the corresponding answer quality evaluation features, so that the trained target evaluation model can machine score each answer result and output the corresponding answer quality evaluation result.

[0123] S2032: The answer quality evaluation system determines the corresponding neighbor number based on a neighbor vector, and determines the corresponding average neighbor weight based on the target similarity vector of the answer result and the neighbor vector, wherein the average neighbor weight represents the average value of the correlation between each answer result and the answer result.

[0124] For each neighbor vector in the neighbor matrix D, the answer quality evaluation system performs the following operations to obtain the corresponding neighbor number: adding each dimension of the feature vector of a neighbor vector, and determining the sum of the feature vectors as the neighbor number.

[0125] According to the foregoing introduction, the neighbor matrix D is a matrix composed of 0 and 1, and therefore, the sum of each feature vector of each neighbor vector in the matrix is an integer, and thus the sum of each feature vector can be determined as the corresponding neighbor quantity.

[0126] Continuing the above example, when the neighbor matrix D is , the neighbor quantities of each answer result are 2, 3, …, 3, and 4 in turn.

[0127] For the target similarity vector of each answer result and the neighbor vector, the answer quality evaluation system performs the following operations to determine the corresponding average neighbor weight:

[0128] As shown in formula 4, the target similarity vector of an answer result is masked using the neighbor vector of the answer result to obtain a neighbor weight vector; wherein the neighbor weight vector contains multiple dimensional feature vectors, each dimensional feature vector corresponds to one of the multiple answer results, and each dimensional feature vector represents the degree of association between the corresponding answer result and the answer result.

[0129] T zero = T ⊙ M formula 4

[0130] Wherein, T zero is a neighbor weight matrix, T is a target similarity matrix, M is a neighbor matrix, and ⊙ represents matrix point multiplication.

[0131] In the field of image processing, a selected image, figure or object is often used to shield part or all of the image to be processed to achieve the purpose of controlling the image processing area, and the specific image, figure or object used for covering is called “mask”. When processing digital images, the mask can also be a two-dimensional matrix array or a multi-value image.

[0132] However, the “masking processing” mentioned in the embodiments of the present application refers to matrix point multiplication of the target similarity matrix and the neighbor matrix, and the weight of the answer result without an association relationship is set to 0 to achieve the purpose of masking the answer result without an association relationship.

[0133] Continuing the above example, when the target similarity matrix is , and the neighbor matrix D is , the obtained neighbor weight matrix T is

[0134] As shown in formula 5, based on the one neighbor weight vector and the neighbor quantity, the corresponding average neighbor weight is obtained.

[0135]

[0136] Wherein, Si is the average neighbor weight of the i-th answer result, D i is the neighbor number of the i-th answer result, T zero_i is the neighbor weight vector of the i-th answer result, T zero [i, j1'] is the j1'-th eigenvector of the i-th neighbor weight vector, N is the total number of answer results.

[0137] Continuing the above example, when the neighbor weight matrix and the neighbor numbers of each answer result are 2, 3, …, 3, 4 in turn, the average neighbor weights of each answer result are 2.5, 3, …, 2.3, 3.5 in turn.

[0138] S2033: The neighbor number and the average neighbor weight are taken as the answer quality evaluation features of the answer result.

[0139] When the subjective question is a written test question, the neighbor number and the average neighbor weight can be taken as the answer quality evaluation features of the answer result and input into the target evaluation model for answer quality evaluation. However, in order to further improve the detection accuracy of the model, the answer quality evaluation features can also be taken as one of the text features, and other text features (such as part-of-speech statistics, keyword hit rate, sentence number, etc.) are input into the target evaluation model for answer quality evaluation.

[0140] When the subjective question is an oral test question, the answer quality evaluation features and the acoustic features (such as the sound energy, peak value, duration, pause number, word number, pronunciation score, etc. of the speaker) of the answer result can be input into the target evaluation model for answer quality evaluation. Similarly, in order to further improve the detection accuracy of the model, the answer quality evaluation features can also be taken as one of the text features, and other text features (such as part-of-speech statistics, keyword hit rate, sentence number, etc.), acoustic features, etc. multiple features are input into the target evaluation model for answer quality evaluation.

[0141] The traditional off-topic detection model is based on NLP technology, extracts the correlation features between the test questions and the student answers, and has the problems of large computation amount, high delay, and only being able to extract features in specific scenarios when extracting features.

[0142] To solve the above problems, the embodiments of the present application use several steps of vector operation to obtain the answer quality evaluation features of each answer result, which has small operation amount, low delay, is more suitable for application in application scenarios with high real-time requirement, and this feature extraction method is suitable for different application scenarios, reduces the limitation of application scenarios on feature extraction, and is beneficial to improve the detection accuracy and robustness of the model. In addition, since no model is used to extract features, a large number of labeled sample data is not needed to train the model to ensure the feature extraction accuracy and stability of the evaluation model, which reduces the training cost of the model.

[0143] Finally, a brief introduction is given to the process of training the to-be-trained evaluation model to obtain the target evaluation model: the to-be-trained evaluation model used in the embodiments of the present application is a regression model based on a decision tree. In order to reduce the training cost, in each round of training, the answer results of 10-20% of the students in each exam are extracted, and the extracted part of the answer results is manually scored by professional teachers, and is determined as the labeled data. The to-be-trained evaluation model uses the obtained labeled data to perform model training, fits the scoring rule of the teachers, and performs machine scoring on the answer results of the remaining students in each exam. After multiple rounds of training, the trained target evaluation model is obtained.

[0144] Since the exam includes multiple forms such as written test and oral test, the following two specific embodiments are taken as examples to introduce the process of applying the question answer quality evaluation method proposed in the embodiments of the present application to different forms of exam scenarios to perform answer quality evaluation on the answer results given by each student.

[0145] Referring to Figure 3A the flowchart shown in Figure 3B the logical diagram shown in, the process of performing answer quality evaluation on the answer results given by each student in the oral test is as follows:

[0146] S301: The answer quality evaluation system collects the audio format answer results input by each student in the same oral test for the same oral question;

[0147] S302: The answer quality evaluation system uses ASR technology to perform speech-to-text on the audio format answer results to obtain the text format answer results;

[0148] S303: The answer quality evaluation system performs feature extraction on the audio format answer results to obtain the corresponding acoustic features;

[0149] S304: The answer quality evaluation system performs vectorization processing on the text format answer results to obtain the corresponding answer result vectors;

[0150] S305: The answer quality evaluation system obtains an intermediate similarity vector of each answer result based on the obtained answer result vector;

[0151] S306: The answer quality evaluation system obtains a target similarity vector of each answer result based on the obtained intermediate similarity vector;

[0152] S307: The answer quality evaluation system obtains an answer quality evaluation feature of each answer result based on the obtained target similarity vector;

[0153] S308: The answer quality evaluation system inputs the obtained answer quality evaluation feature and corresponding acoustic feature into an internal target evaluation model to obtain an answer quality evaluation result of the corresponding answer result output by the model.

[0154] Referring to Figure 4A the flowchart shown in Figure 4B the logical diagram shown in, the process of evaluating the answer quality of the composition of each student in a Chinese exam is as follows:

[0155] S401: The answer quality evaluation system collects the compositions written by each student in the exam;

[0156] S402: The answer quality evaluation system performs vectorization processing on each composition to obtain a corresponding answer result vector;

[0157] S403: The answer quality evaluation system obtains an intermediate similarity vector of each composition based on the obtained answer result vector;

[0158] S404: The answer quality evaluation system obtains a target similarity vector of each composition based on the obtained intermediate similarity vector;

[0159] S405: The answer quality evaluation system obtains an answer quality evaluation feature of each composition based on the obtained target similarity vector;

[0160] S406: The answer quality evaluation system inputs the obtained answer quality evaluation feature into an internal target evaluation model to obtain an answer quality evaluation result of the corresponding composition output by the model.

[0161] Based on the same inventive concept as the above method embodiment, the embodiments of the present application also provide an evaluation device for the answer quality of a question. As Figure 5 shown, the evaluation device 500 for the answer quality of a question can include:

[0162] The vectorization processing unit 501 is configured to obtain an answer result input by each target object for a same subjective question, and obtain an intermediate similarity vector corresponding to each answer result, where each intermediate similarity vector represents a first similarity degree distribution between the corresponding answer result and each answer result.

[0163] The feature extraction unit 502 is configured to, for each intermediate similarity vector, obtain a target similarity vector based on the intermediate similarity vector of one answer result and the intermediate similarity vectors of each answer result, where each target similarity vector represents a second similarity degree distribution between the first similarity degree distribution of one answer result and the first similarity degree distributions of each answer result.

[0164] The answer quality evaluation unit 503 is configured to determine an answer quality evaluation feature of each answer result based on the obtained target similarity vector, and input each off-topic feature into the target evaluation model to obtain an answer quality evaluation result of the corresponding answer result output by the target evaluation model.

[0165] Optionally, the feature extraction unit 502 is configured to:

[0166] The intermediate similarity vectors of each answer result are sequentially obtained, and the following operations are performed each time the intermediate similarity vector of the jth answer result is obtained: a j1-dimensional feature vector of the target similarity vector is determined based on the intermediate similarity vector of the jth answer result and the intermediate similarity vector of one answer result, where 1≤j≤N and 1≤j1≤N, and N is the total number of answer results.

[0167] The target similarity vector of one answer result is obtained based on the obtained N-dimensional feature vector.

[0168] Optionally, the answer quality evaluation unit 503 is configured to:

[0169] For each target similarity vector, the answer quality evaluation feature of the corresponding answer result is obtained by performing the following operations:

[0170] A neighbor vector is determined based on the target similarity vector of one answer result and a set neighbor threshold value, where each neighbor vector includes multiple-dimensional feature vectors, each dimensional feature vector corresponds to one of the multiple answer results, and each dimensional feature vector represents whether there is an association relationship between the corresponding answer result and one answer result.

[0171] The number of neighbors is determined based on one neighbor vector, and the average neighbor weight is determined based on the target similarity vector of one answer result and one neighbor vector, where the average neighbor weight represents an average value of the association degree between each answer result and one answer result.

[0172] The number of neighbors and the average neighbor weight are used as the answer quality evaluation features of the answer.

[0173] Optionally, the answer quality evaluation unit 503 is configured to:

[0174] The j1th feature vector of the target similarity vector is obtained in sequence, and each time the j1th feature vector is obtained, the following operations are performed: comparing the j1th feature vector obtained at present with the neighbor threshold value, and determining the j1th feature vector of the neighbor vector based on the association relationship represented by the comparison result, wherein one comparison result represents whether there is an association relationship between the corresponding answer and one answer, 1≤j1≤N, 1≤j 11 ≤N, and N is the total number of answers; 11

[0175] Based on the obtained N-dimensional feature vector, the neighbor vector of one answer is obtained.

[0176] Optionally, the answer quality evaluation unit 503 is configured to:

[0177] When the j1th feature vector obtained at present is less than the neighbor threshold value, the j1th feature vector of the neighbor vector is determined as the first feature vector value based on the association relationship represented by the comparison result. 11

[0178] When the j1th feature vector obtained at present exceeds the neighbor threshold value, the j1th feature vector of the neighbor vector is determined as the second feature vector value based on the association relationship represented by the comparison result. 11

[0179] Optionally, the answer quality evaluation unit 503 is configured to:

[0180] The feature vectors of the neighbor vector are added, and the sum of the feature vectors is determined as the number of neighbors.

[0181] Optionally, the answer quality evaluation unit 503 is configured to:

[0182] The target similarity vector of one answer is masked using the neighbor vector of one answer to obtain a neighbor weight vector, wherein the neighbor weight vector contains multiple-dimensional feature vectors, each dimensional feature vector corresponds to one of the multiple answers, and each dimensional feature vector represents the association degree between the corresponding answer and one answer.

[0183] Based on the neighbor weight vector and the number of neighbors, the average neighbor weight is obtained.

[0184] Optionally, the vectorization processing unit 501 is configured to:​​​

[0185] vectorize each answer result respectively to obtain corresponding answer result vectors;

[0186] For each obtained answer result vector, the following operation is performed: based on the answer result vector of one answer result and the answer result vectors of each answer result, a corresponding intermediate similarity vector is obtained.

[0187] Optionally, the vectorization processing unit 501 is configured to:

[0188] The answer result vectors of each answer result are sequentially traversed, and when the answer result vector of the jth answer result is traversed, based on the answer result vector of one answer result and the answer result vector of the jth answer result, the jth feature vector of the intermediate similarity vector is determined, where 1≤j≤N, 1≤j 111 N, N is the total number of each answer result; 111

[0189] Based on the obtained N-dimensional feature vector, the intermediate similarity vector of one answer result is obtained.

[0190] For the convenience of description, each part is divided into modules (or units) according to the function and described respectively. Of course, in the implementation of the present application, the functions of each module (or unit) can be realized in the same or multiple software or hardware.

[0191] After introducing the subject answer quality evaluation method and device of the example embodiment of the present application, next, the computer device according to another example embodiment of the present application is introduced.

[0192] Those skilled in the art can understand that each aspect of the present application can be implemented as a system, a method or a program product. Therefore, each aspect of the present application can be specifically implemented as follows: a complete hardware embodiment, a complete software embodiment (including firmware, microcode, etc.), or an embodiment combined with hardware and software, which can be collectively referred to as "circuit", "module" or "system".

[0193] Based on the same inventive concept as the above method embodiment, the present embodiment also provides a computer device, which is shown in Figure 6 The computer device 600 can at least include a processor 601 and a memory 602. The memory 602 stores program code, and when the program code is executed by the processor 601, the processor 601 performs the steps of any one of the above-mentioned subject answer quality evaluation methods.

[0194] ​In some possible implementation, the computing device according to the present application can comprise at least one processor, and at least one memory. The memory stores program codes which, when executed by the processor, cause the processor to perform the steps in the method of evaluating the quality of the answer to the question according to various exemplary embodiments of the present application described above. For example, the processor can perform the steps as shown in Figure 2A

[0195] The computing device 700 according to this embodiment of the present application will be described below with reference to Figure 7 Figure 7 The computing device 700 is merely an example, and should not bring any limitation to the function and use range of the embodiments of the present application.

[0196] As shown in Figure 7 , the computing device 700 is in the form of a general-purpose computing device. The components of the computing device 700 can include, but are not limited to, the at least one processing unit 701 described above, the at least one storage unit 702 described above, and a bus 703 connecting different system components, including the storage unit 702 and the processing unit 701.

[0197] The bus 703 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a processor or local bus using any of a variety of bus architectures.

[0198] The storage unit 702 can include a readable medium in the form of volatile memory, such as a random access memory (RAM) 7021 and / or a cache memory unit 7022, and can further include a read-only memory (ROM) 7023.

[0199] The storage unit 702 can also include a program / utility 7025 having a set of program modules 7024, including but not limited to an operating system, one or more application programs, other program modules, and program data, each of which or a combination thereof can include implementation of a network environment.

[0200] ​​Computing device 700 can also communicate with one or more external devices 704 such as a keyboard or pointing device, through I / O interface 705. Note that same or similar input devices can also be used in combination with the computing device 700. In general, the computing device 700 will communicate with one or more other computing devices 704 in a network, or the Internet 706, through network adapter 707. As depicted, network adapter 707 is in communication with the other components of the computing device 700 through bus 703. It should be appreciated that although not shown, other hardware and / or software components could be used in combination with the computing device 700. These components, as is well known in the art, will be

[0201] According to the same inventive concept as the method embodiments described above, the various aspects of the method for evaluating the quality of answers to a question provided by the present application can also be implemented in the form of a program product, which includes program code, when the program product is run on a computer device, the program code is used to make the computer device execute the steps of the method for evaluating the quality of answers to a question according to various exemplary embodiments of the present application described above in the specification, for example, the computer device can execute the steps as shown in Figure 2A

[0202] The program product can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium, for example, can be, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples (a non-exhaustive list) of the readable storage medium include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0203] Although the preferred embodiments of the present application have been described, those skilled in the art will be able to make additional changes and modifications to these embodiments once they have the benefit of the foregoing description. Therefore, it is intended to include all such changes and modifications in the scope of the present application as set forth in the appended claims.

[0204] ​Obviously, many modifications and variations of the present application are possible in light of the above teachings. It is, therefore, to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. A method of evaluating the quality of answers to a question, characterized in that, The method comprises the following steps: obtaining the answer results of each target object for the same subjective question, and obtaining the intermediate similarity vectors corresponding to each answer result, wherein each intermediate similarity vector represents the first similarity degree distribution between the corresponding answer result and the answer results; for each intermediate similarity vector, the following operations are performed: based on the intermediate similarity vector of one answer result and the intermediate similarity vectors of the answer results, the corresponding target similarity vector is obtained, wherein each target similarity vector represents the second similarity degree distribution between the first similarity degree distribution of the one answer result and the first similarity degree distribution of the answer results; for each target similarity vector, the answer quality evaluation feature of the corresponding answer result is obtained by performing the following operations: based on the target similarity vector of the one answer result and the set neighbor threshold value, a neighbor vector is determined; the one neighbor vector comprises a plurality of dimensional feature vectors, each dimensional feature vector corresponds to one of the plurality of answer results, and each dimensional feature vector represents whether there is an association relationship between the corresponding answer result and the one answer result; based on the one neighbor vector, the corresponding neighbor number is determined, and based on the target similarity vector of the one answer result and the one neighbor vector, the corresponding average neighbor weight is determined, wherein the average neighbor weight represents the average value of the association degree between the answer results and the one answer result; the neighbor number and the average neighbor weight are used as the answer quality evaluation feature of the one answer result; inputting each answer quality evaluation feature into a target evaluation model to obtain the answer quality evaluation result of the corresponding answer result output by the target evaluation model.

2. The method of claim 1, wherein, The method comprises the following steps: obtaining the answer results of each target object for the same subjective question, and obtaining the intermediate similarity vectors corresponding to each answer result, wherein each intermediate similarity vector represents the first similarity degree distribution between the corresponding answer result and the answer results; for each intermediate similarity vector, the following operations are performed: based on the intermediate similarity vector of one answer result and the intermediate similarity vectors of the answer results, the corresponding target similarity vector is obtained, wherein each target similarity vector represents the second similarity degree distribution between the first similarity degree distribution of the one answer result and the first similarity degree distribution of the answer results; 3. The method of claim 1, wherein, for each target similarity vector, the answer quality evaluation feature of the corresponding answer result is obtained by performing the following operations: based on the target similarity vector of the one answer result and the set neighbor threshold value, a neighbor vector is determined; the one neighbor vector comprises a plurality of dimensional feature vectors, each dimensional feature vector corresponds to one of the plurality of answer results, and each dimensional feature vector represents whether there is an association relationship between the corresponding answer result and the one answer result; based on the one neighbor vector, the corresponding neighbor number is determined, and based on the target similarity vector of the one answer result and the one neighbor vector, the corresponding average neighbor weight is determined, wherein the average neighbor weight represents the average value of the association degree between the answer results and the one answer result; the neighbor number and the average neighbor weight are used as the answer quality evaluation feature of the one answer result; obtaining each dimension feature vector of the target similarity vector in sequence, and performing the following operation each time: comparing the j1th dimension feature vector obtained currently with the neighbor threshold value, and determining the j1th dimension feature vector of the neighbor vector based on the correlation relationship represented by the comparison result 11 wherein one comparison result represents whether there is a correlation relationship between the corresponding answer result and the one answer result, 1≤j1≤N, 1≤j 11 ≤N, and N is the total number of the answer results; and obtaining each dimension feature vector of the target similarity vector in sequence, and performing the following operation each time: comparing the j1th dimension feature vector obtained currently with the neighbor threshold value, and determining the j1th dimension feature vector of the neighbor vector based on the correlation relationship represented by the comparison result 11 wherein one comparison result represents whether there is a correlation relationship between the corresponding answer result and the one answer result, 1≤j1≤N, 1≤j 11 ≤N, and N is the total number of the answer results; and inputting each answer quality evaluation feature into a target evaluation model to obtain the answer quality evaluation result of the corresponding answer result output by the target evaluation model.

4. The method of claim 3, wherein, comparing the jth dimension feature vector currently acquired with the neighbor threshold value, and determining the jth dimension feature vector of the neighbor vector based on the relationship between the jth dimension feature vector currently acquired and the neighbor threshold value 11 dimension feature vector In the current acquisition of the j1 dimensional feature vector is less than the neighbor threshold value, based on the comparison result of the relationship between the representation of the association, the j 11 dimensional feature vector is the first feature vector value; When the currently acquired j1-th dimension feature vector exceeds the neighbor threshold, the j-th dimension of the neighbor vector is determined based on the association relationship represented by the comparison result. 11 The eigenvector of dimension 1 is the value of the second eigenvector.

5. The method of claim 4, wherein, The method comprises the following steps: obtaining the answer results of each target object for the same subjective question, and obtaining the intermediate similarity vectors corresponding to each answer result, wherein each intermediate similarity vector represents the first similarity degree distribution between the corresponding answer result and the answer results; 6. The method of claim 1, wherein, for each intermediate similarity vector, the following operations are performed: based on the intermediate similarity vector of one answer result and the intermediate similarity vectors of the answer results, the corresponding target similarity vector is obtained, wherein each target similarity vector represents the second similarity degree distribution between the first similarity degree distribution of the one answer result and the first similarity degree distribution of the answer results; for each target similarity vector, the answer quality evaluation feature of the corresponding answer result is obtained by performing the following operations: based on the target similarity vector of the one answer result and the set neighbor threshold value, a neighbor vector is determined; the one neighbor vector comprises a plurality of dimensional feature vectors, each dimensional feature vector corresponds to one of the plurality of answer results, and each dimensional feature vector represents whether there is an association relationship between the corresponding answer result and the one answer result; based on the one neighbor vector, the corresponding neighbor number is determined, and based on the target similarity vector of the one answer result and the one neighbor vector, the corresponding average neighbor weight is determined, wherein the average neighbor weight represents the average value of the association degree between the answer results and the one answer result; the neighbor number and the average neighbor weight are used as the answer quality evaluation feature of the one answer result; inputting each answer quality evaluation feature into a target evaluation model to obtain the answer quality evaluation result of the corresponding answer result output by the target evaluation model. The method comprises the following steps: obtaining the answer results of each target object for the same subjective question, and obtaining the intermediate similarity vectors corresponding to each answer result, wherein each intermediate similarity vector represents the first similarity degree distribution between the corresponding answer result and the answer results; for each intermediate similarity vector, the following operations are performed: based on the intermediate similarity vector of one answer result and the intermediate similarity vectors of the answer results, the corresponding target similarity vector is obtained, wherein each target similarity vector represents the second similarity degree distribution between the first similarity degree distribution of the one answer result and the first similarity degree distribution of the answer results; for each target similarity vector, the answer quality evaluation feature of the corresponding answer result is obtained by performing the following operations: based on the target similarity vector of the one answer result and the set neighbor threshold value, a neighbor vector is determined; the one neighbor vector comprises a plurality of dimensional feature vectors, each dimensional feature vector corresponds to one of the plurality of answer results, and each dimensional feature vector represents whether there is an association relationship between the corresponding answer result and the one answer result; based on the one neighbor vector, the corresponding neighbor number is determined, and based on the target similarity vector of the one answer result and the one neighbor vector, the corresponding average neighbor weight is determined, wherein the average neighbor weight represents the average value of the association degree between the answer results and the one answer result; the neighbor number and the average neighbor weight are used as the answer quality evaluation feature of the one answer result; inputting each answer quality evaluation feature into a target evaluation model to obtain the answer quality evaluation result of the corresponding answer result output by the target evaluation model. mask the target similarity vector of the one answer result by using the neighbor vector of the one answer result, to obtain one neighbor weight vector, wherein the one neighbor weight vector contains a plurality of dimensional feature vectors, each dimensional feature vector corresponds to one of the plurality of answer results, and each dimensional feature vector represents the correlation degree between the corresponding answer result and the one answer result; determine a corresponding average neighbor weight based on the one neighbor weight vector and the neighbor number.

7. The method of claim 2, wherein, The respective intermediate similarity vectors corresponding to the answer results are obtained by: vectorizing the respective answer results to obtain respective answer result vectors; for each answer result vector, obtaining a corresponding intermediate similarity vector based on the answer result vector of the one answer result and the answer result vectors of the respective answer results.

8. The method of claim 1, wherein, The respective intermediate similarity vectors corresponding to the answer results are obtained by: The answer result vectors of the answer results are sequentially traversed, when the answer result vector of the jth answer result is traversed, the jth element of the intermediate similarity vector is determined based on the answer result vector of the one answer result and the answer result vector of the jth answer result, wherein 1≤j≤N, 1≤j 111 N, N is the total number of the answer results. 111 N, N is the total number of the answer results. obtaining the intermediate similarity vector of the one answer result based on the obtained N-dimensional feature vector.

9. An evaluation device for the quality of an answer to a question, characterized in that It includes: a vectorization processing unit configured to obtain answer results input by respective target objects for the same subjective question, and obtain respective intermediate similarity vectors corresponding to the answer results, wherein each intermediate similarity vector represents a first similarity degree distribution between the corresponding answer result and the respective answer results; a feature extraction unit configured to, for each intermediate similarity vector, obtain a corresponding target similarity vector based on the intermediate similarity vector of the one answer result and the intermediate similarity vectors of the respective answer results, wherein each target similarity vector represents a second similarity degree distribution between the first similarity degree distribution of the one answer result and the first similarity degree distribution of the respective answer results; an answer quality evaluation unit configured to, for each target similarity vector, obtain an answer quality evaluation feature of the corresponding answer result by performing the following operations: determining a neighbor vector based on the target similarity vector of the one answer result and a set neighbor threshold value, wherein the neighbor vector contains a plurality of dimensional feature vectors, each dimensional feature vector corresponds to one of the plurality of answer results, and each dimensional feature vector represents whether there is an association relationship between the corresponding answer result and the one answer result; determining a corresponding neighbor number based on the one neighbor vector, and determining a corresponding average neighbor weight based on the target similarity vector of the one answer result and the one neighbor vector, wherein the average neighbor weight represents the average value of the correlation degree between the respective answer results and the one answer result; and taking the neighbor number and the average neighbor weight as the answer quality evaluation feature of the one answer result; inputting each answer quality evaluation feature into a target evaluation model to obtain an answer quality evaluation result of the corresponding answer result output by the target evaluation model.

10. The apparatus of claim 9, wherein, The feature extraction unit is configured to: obtaining intermediate similarity vectors of the respective answer results in sequence, and performing the following operation each time: determining a j1-dimensional feature vector of the target similarity vector based on the intermediate similarity vector of the jth answer result obtained currently and in combination with the intermediate similarity vector of the one answer result, where 1≤j≤N and 1≤j1≤N, and N is the total number of the respective answer results; obtaining the target similarity vector of the one answer result based on the obtained N-dimensional feature vector.

11. A computer device, comprising: The device comprises a processor and a memory, wherein the memory stores program code, and when the program code is executed by the processor, the processor executes the steps of the method in any one of claims 1-8.

12. A computer-readable storage medium, characterized in that, The device comprises program code, and when the program code is executed on the computer device, the program code is used to make the computer device execute the steps of the method in any one of claims 1-8.

13. A computer program product, characterised in that, The device comprises computer instructions, and when the computer instructions are executed by a processor, the steps of the method in any one of claims 1-8 are implemented.

Images