Sample evaluation

Abstract

According to an implementation of the present disclosure, a scheme for sample evaluation is proposed. In this scheme, a set of comparison results is generated based on an input sample and a reference sample set. Each comparison result indicates the probability that a target attribute of the input sample is superior to the target attribute of a corresponding reference sample in the reference sample set. Based on the set of comparison results, at least one reference sample is determined from the reference sample set that matches the input sample according to the target attribute. Based on the corresponding scores of the at least one reference sample with respect to the target attribute, the score of the input sample with respect to the target attribute is determined. In this way, the score of the input sample with respect to the target input can be determined efficiently and accurately, thereby achieving efficient, accurate, and robust sample evaluation.

Description

Background Art

[0001] In various data processing scenarios, it is often necessary to evaluate a specific attribute of a sample, and the evaluation results are usually presented as numerical values or grades. Therefore, it is necessary to score samples. For example, in computer-assisted language learning, it may be necessary to evaluate user speech samples. Score the speech samples for things like oral fluency and pronunciation accuracy. Similar sample evaluation requirements also exist in many other application scenarios, such as text evaluation, age estimation, and credit assessment. Summary of the Invention

[0002] According to an implementation of the present disclosure, a solution for sample evaluation is proposed. In this solution, a set of comparison results is generated based on an input sample and a reference sample set. Each comparison result indicates the probability that the target attribute of the input sample is superior to the target attribute of a corresponding reference sample in the reference sample set. Then, based on the set of comparison results, at least one reference sample is determined from the reference sample set that matches the input sample according to the target attribute. The target attribute of the input sample is close to or similar to the target attribute of the at least one determined reference sample. Based on the corresponding score of the at least one reference sample with respect to the target attribute, the score of the input sample with respect to the target attribute is determined. This solution transforms the score prediction problem into a series of relatively simple binary comparisons or binary classifications, that is, a comparison of the input sample's quality with the reference samples. This series of binary comparisons helps to accurately identify reference samples whose target attributes are close to those of the input sample. This ensures the accuracy of the input sample score determined based on the scores of the reference samples. In this way, more efficient, accurate, and robust sample evaluation can be achieved.

[0003] This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. It is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Figure 1 A block diagram illustrating a computing device capable of implementing various implementations of the present disclosure is shown;

[0005] Figure 2 An architectural diagram illustrating a system for sample evaluation according to an implementation of the present disclosure is shown;

[0006] Figure 3 A block diagram illustrating a training phase according to some implementations of the present disclosure is shown;

[0007] Figure 4 A block diagram illustrating an inference phase according to some implementations of the present disclosure; and

[0008] Figure 5A flow chart of a method for sample evaluation according to an implementation of the present disclosure is shown.

[0009] In these drawings, the same or similar reference symbols are used to designate the same or similar elements. DETAILED DESCRIPTION

[0010] The present disclosure will now be discussed with reference to several example implementations. It should be understood that these implementations are discussed only to enable those skilled in the art to better understand and implement the present disclosure, rather than to imply any limitation on the scope of the present disclosure.

[0011] As used herein, the term "including" and its variations are to be interpreted as open-ended terms meaning "including but not limited to." The term "based on" is to be interpreted as "based, at least in part, on." The terms "an implementation" and "an implementation" are to be interpreted as "at least one implementation." The term "another implementation" is to be interpreted as "at least one other implementation." The terms "first," "second," and so on may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0012] As used herein, a "neural network" is capable of processing inputs and providing corresponding outputs, and typically includes an input layer and an output layer and one or more hidden layers between the input layer and the output layer. The neural networks used in deep learning applications typically include many hidden layers, thereby extending the depth of the network. The various layers of the neural network are connected in sequence so that the output of the previous layer is provided as the input of the next layer, wherein the input layer receives the input of the neural network, and the output of the output layer serves as the final output of the neural network. Each layer of the neural network includes one or more nodes (also referred to as processing nodes or neurons), each of which processes the input from the previous layer. In this article, the terms "neural network", "network" and "neural network model" are used interchangeably.

[0013] As used herein, the “attribute” of a sample refers to the properties or characteristics that the sample possesses in one or more specific aspects. The “target attribute” of a sample may be one or more attributes or a combination of attributes selected for the purpose of sample evaluation. Therefore, the “target attribute” of a sample may vary depending on the purpose of the sample evaluation. For example, in the evaluation of language learning samples, the “target attribute” may include but is not limited to oral fluency, pronunciation accuracy, language proficiency, etc. In age estimation, the “target attribute” may refer to the age of the user corresponding to the sample. In credit evaluation, the “target attribute” may refer to the credit rating of the user corresponding to the sample.

[0014] As briefly described above, there is a need for sample evaluation in many application scenarios, that is, to score one or more attributes of the sample. Taking the language learning application scenario as an example, it may be necessary to evaluate the user's language learning sample. For example, it may be necessary to score a speech sample from the user (such as a recording) for oral fluency. A common scoring scheme for oral fluency is a 5-point mean opinion score (MOS), where a score of 1 represents "very poor", a score of 2 represents "poor", a score of 3 represents "average", a score of 4 represents "good", and a score of 5 represents "excellent". For another example, it may be necessary to score a text sample from the user (such as an essay). This scoring of samples in language learning application scenarios usually involves subjective evaluation.

[0015] The results of subjective evaluations are limited by the evaluator, such as an expert. Different evaluators may provide different, even significantly different, evaluations of the same sample. The same evaluator may also provide different evaluations of the same sample at different times. This type of subjective evaluation typically requires the participation of multiple evaluators, resulting in significant time and labor costs. In online voice learning scenarios, this subjective evaluation also fails to provide users with real-time evaluation results.

[0016] In order to perform automatic evaluation, a quantitative and objectively measurable scoring method is required. Different scoring models have been proposed, such as multi-classifiers and Gaussian models. In these existing schemes, the different scores of samples on the target attribute (e.g., oral fluency) are independent, and the inherent order between these different scores is ignored. In addition, regression methods that treat scores as numerical values have been proposed. However, in this method, training unstable kernels for regression is challenging and is prone to overfitting during training.

[0017] Ordinal regression aims to classify or predict numerical values based on labeled samples, where the labels of the samples exhibit a natural ordering. Training an ordinal regression model requires labeled samples, and the sample labels, i.e., the scores of the samples with respect to the target attribute, require subjective evaluation by human evaluators. Therefore, the number of such labeled samples is typically small, which is not conducive to obtaining a robust and accurate ordinal regression model.

[0018] In view of this, according to an implementation of the present disclosure, a solution for sample evaluation is provided, which aims to solve one or more of the above-mentioned problems and other potential problems. In this solution, a set of comparison results is generated based on an input sample and a reference sample set. Each comparison result indicates the probability that the target attribute of the input sample is better than the target attribute of the corresponding reference sample in the reference sample set. Then, based on the set of comparison results, at least one reference sample is determined from the reference sample set that matches the input sample according to the target attribute. The target attribute of the input sample is close to or similar to the target attribute of the at least one reference sample determined. Next, based on the corresponding score of the at least one reference sample with respect to the target attribute, a score of the input sample with respect to the target attribute is determined.

[0019] The process of scoring input samples typically involves complex multi-classification or ordinal regression. The sample evaluation scheme proposed here can be viewed as ordinal regression based on reference samples. It transforms the originally complex multi-classification or ordinal regression into a series of simpler binary comparisons or classifications: that is, comparisons between the input sample and the reference sample. This series of binary comparisons helps accurately identify reference samples whose target attributes are similar to those of the input sample. This ensures the accuracy of the input sample score determined based on the reference sample score. In this way, more efficient, accurate, and robust sample evaluation can be achieved.

[0020] Various example implementations of this solution are described in detail below with reference to the accompanying drawings.

[0021] Sample Environment

[0022] Figure 1 FIG1 shows a block diagram of a computing device 100 capable of implementing various implementations of the present disclosure. It should be understood that Figure 1 The computing device 100 shown is merely exemplary and should not be construed as limiting the functionality and scope of the implementations described herein. Figure 1 As shown, computing device 100 comprises a computing device in the form of a general-purpose computing device 100. Components of computing device 100 may include, but are not limited to, one or more processors or processing units 110, memory 120, storage device 130, one or more communication units 140, one or more input devices 150, and one or more output devices 160.

[0023] In some implementations, the computing device 100 can be implemented as various user terminals or service terminals with computing capabilities. The service terminal can be a server, a large computing device, etc. provided by various service providers. The user terminal is such as a mobile terminal, a fixed terminal, or a portable terminal of any type, including a mobile phone, a site, a unit, a device, a multimedia computer, a multimedia tablet, an Internet node, a communicator, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio broadcast receiver, an e-book device, a gaming device, or any combination thereof, including accessories and peripherals of these devices, or any combination thereof. It is also foreseeable that the computing device 100 can support any type of interface for the user (such as a "wearable" circuit, etc.).

[0024] Processing unit 110 may be a real or virtual processor and is capable of performing various processes according to a program stored in memory 120. In a multi-processor system, multiple processing units execute computer-executable instructions in parallel to increase the parallel processing capabilities of computing device 100. Processing unit 110 may also be referred to as a central processing unit (CPU), a microprocessor, a controller, or a microcontroller.

[0025] The computing device 100 typically includes a plurality of computer storage media. Such media can be any available media accessible to the computing device 100, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 120 can be a volatile memory (e.g., registers, cache, random access memory (RAM)), a non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The memory 120 can include sample processing modules 122, which are program modules configured to perform the functions of the various implementations described herein. The sample processing modules 122 can be accessed and executed by the processing unit 110 to implement the corresponding functions.

[0026] Storage device 130 may be removable or non-removable media and may include machine-readable media that can be used to store information and / or data and can be accessed within computing device 100. Computing device 100 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not described in Figure 1 As shown in FIG, a magnetic disk drive for reading or writing from a removable, nonvolatile magnetic disk and an optical disk drive for reading or writing from a removable, nonvolatile optical disk can be provided. In these cases, each drive can be connected to a bus (not shown) by one or more data media interfaces.

[0027] The communication unit 140 enables communication with other computing devices via a communication medium. Additionally, the functionality of the components of the computing device 100 can be implemented as a single computing cluster or multiple computing machines that can communicate via a communication connection. Thus, the computing device 100 can operate in a networked environment using logical connections to one or more other servers, personal computers (PCs), or another general network node.

[0028] Input device 150 may be one or more of various input devices, such as a mouse, keyboard, trackball, voice input device, etc. Output device 160 may be one or more output devices, such as a display, speaker, printer, etc. Computing device 100 may also communicate with one or more external devices (not shown) via communication unit 140 as needed, such as storage devices, display devices, etc., with one or more devices that allow a user to interact with computing device 100, or with any device that allows computing device 100 to communicate with one or more other computing devices (e.g., a network card, modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0029] In some implementations, in addition to being integrated on a single device, some or all of the various components of computing device 100 may be configured in the form of a cloud computing architecture. In a cloud computing architecture, these components may be remotely located and work together to implement the functionality described herein. In some implementations, cloud computing provides computing, software, data access, and storage services that do not require the end user to be aware of the physical location or configuration of the systems or hardware providing these services. In various implementations, cloud computing provides services over a wide area network (such as the Internet) using appropriate protocols. For example, a cloud computing provider provides applications over a wide area network, and these applications can be accessed through a web browser or any other computing component. The software or components of the cloud computing architecture and the corresponding data may be stored on servers at remote locations. Computing resources in a cloud computing environment may be consolidated at remote data center locations or they may be dispersed. Cloud computing infrastructure can provide services through shared data centers, even though they appear to be a single access point for users. Therefore, the components and functionality described herein can be provided by a service provider at a remote location using a cloud computing architecture. Alternatively, they can be provided from a conventional server, or they can be installed directly or otherwise on the client device.

[0030] The computing device 100 may be used to implement sample evaluations in various implementations of the present disclosure. Figure 1As shown, computing device 100 may receive input sample 170 via input device 150. Input sample 170 is typically associated with a user. For example, in an age estimation application scenario, input sample 170 may include a facial image of the user; in a language learning application scenario, input sample 170 may include a speech segment spoken by the user, or a text segment written by the user.

[0031] In some implementations, the input sample 170 may be raw data, such as raw voice data or text data. In this implementation, the computing device 100 may preprocess the raw data to extract features of the input sample 170 for the target attribute. In some implementations, the input sample 170 may be preprocessed data. The input sample 170 may include one or more features extracted for the target attribute based on the raw data. For example, if an article or a paragraph is to be comprehensively evaluated, the extracted features may include, but are not limited to, average sentence length, shortest sentence length, longest sentence length, average paragraph length, shortest paragraph length, longest paragraph length, number of words, number of sentences, number of paragraphs, grammatical error ratio, spelling error ratio, unigram / bigram / trigram probability, etc.

[0032] A reference sample set 190 may be stored at the computing device 100. Alternatively or additionally, the computing device 100 may receive the reference sample set 190 from the input device 150 or via the communication unit 140. The reference sample set 190 includes multiple reference samples of the same type as the input sample 170. Each reference sample may have a score related to the target attribute, which is also referred to herein as a "predetermined score." The predetermined score of the reference sample may be assigned by an evaluator or determined based on the evaluator's rating. Therefore, the reference sample may be considered a labeled sample. Similar to the input sample, the reference sample may be raw data or may include features extracted for the target attribute.

[0033] The computing device 100 can implement the scheme for sample evaluation described herein to determine an evaluation result 180 for the input sample 170. The evaluation result 180 can be presented in the form of a score of the input sample 170 with respect to the target attribute. For example, in the oral fluency example mentioned above, the evaluation result 180 can include a score of the input sample 170 with respect to oral fluency, such as one of the five scores of "1", "2", "3", "4", and "5". Alternatively or additionally, the evaluation result can also be presented in the form of a grade of the input sample 170 with respect to the target attribute. For example, the evaluation result 180 can include one of the grades of "very poor", "poor", "average", "good", and "excellent" corresponding to the five scores of "1", "2", "3", "4", and "5".

[0034] although Figure 1Only one input sample 170 is shown in FIG, but it should be understood that the computing device 100 can process multiple such input samples. Figure 1 In the example shown in FIG, computing device 100 receives input samples 170 from input device 150 and provides evaluation results 180 via output device 160, but this is merely illustrative and not intended to be limiting. Computing device 100 may also receive input samples 170 from other devices (not shown) via communication unit 140 and / or provide evaluation results 180 externally via communication unit 140.

[0035] System Architecture

[0036] Figure 2 FIG2 shows an architecture diagram of a system 200 for sample evaluation according to an implementation of the present disclosure. The system 200 may be implemented in Figure 1 For example, in some implementations, the system 200 may be implemented as Figure 1 At least a portion of the sample processing module 122 of the computing device 100 is implemented as a computer program module. Figure 2 As shown, system 200 may generally include a training subsystem 210 and an inference subsystem 220. It should be understood that the structure and functionality of system 200 are described for exemplary purposes only and do not imply any limitation on the scope of the present disclosure. Implementations of the present disclosure may also be implemented in different structures and / or functions.

[0037] In general, the sample evaluation scheme according to the implementation of the present disclosure can be divided into two phases: a training phase and an inference phase. In the training phase, the training subsystem 210 can receive a training sample set 230 and train a classifier 250 for sample evaluation based on the training sample set 230.

[0038] The training sample set 230 may include multiple training samples of the same type as the input sample 170. Each training sample may have a score related to the target attribute. The score of the training sample may be pre-given by an evaluator or determined based on the evaluator's rating. Therefore, the training sample may also be considered a labeled sample.

[0039] In some implementations, each training sample may include raw data, such as speech data or text data. In such implementations, the training subsystem 210 may include a feature extraction module for extracting one or more features of the training sample for a target attribute. In some implementations, each training sample may be processed data that includes one or more features extracted for a target attribute based on the raw data.

[0040] The trained classifier 250 can predict the relative performance of two samples in terms of a target attribute. In other words, the trained classifier 250 can determine the probability that one of the two samples outperforms the target attribute of the other. For example, the trained classifier 250 can determine the probability that one speech sample outperforms another speech sample in terms of spoken fluency.

[0041] Classifier 250 can be implemented using any suitable model. For example, classifier 250 can be a binary classifier based on a neural network. The neural network can adopt any suitable network structure. For another example, classifier 250 can be a classifier based on a support vector machine. The scope of the present disclosure is not limited in this respect.

[0042] In the inference phase, the inference subsystem 220 may receive the trained classifier 250, the reference sample set 190, and the input sample 170. The inference subsystem 220 may utilize the trained classifier 250 to determine a score of the input sample 170 with respect to a target attribute based on the reference sample set 190.

[0043] The reference sample set 190 can be determined based on the training sample set 230. In this implementation, the labeled samples can be fully utilized. In some implementations, the reference sample set 190 can include all training samples in the training sample set 230. In this implementation, the distribution of the reference samples is the distribution of the training samples, which is closer to the true score distribution. Therefore, in the inference phase, the input sample 170 is compared with each reference sample that is closer to the true score distribution. This helps to locate reference samples that are closer to the input sample 170 in terms of the target attribute, thereby more accurately determining the score of the input sample. In other implementations, the reference sample set 190 can include a portion of the training samples in the training sample set 230. How to determine the reference sample from the training sample set 230 will be described in detail below.

[0044] Alternatively, the reference sample set 190 may be independent of the training sample set 230. Although the reference samples and the training samples are of the same type, the reference samples may be obtained independently of the training samples. For example, the reference samples and the training samples may be associated with different users.

[0045] The following will refer to Figure 3 and Figure 4 Describes example procedures for the training and inference phases.

[0046] Training phase

[0047] The above describes the overall architecture of the system 200 for sample evaluation. Some implementations of the training phase performed by the training subsystem 210 will be described below. Figure 3A block diagram 300 is shown of a training phase according to some implementations of the present disclosure. The following describes the training of the classifier 250 using a neural network as an example, but this is only exemplary and is not intended to limit the scope of the present disclosure.

[0048] During the training phase, the training subsystem 210 generates multiple training sample pairs 311, 312, ..., 31N based on the training sample set 230, where N is a natural number. The multiple training sample pairs 311, 312, ..., 31N may constitute a set of training sample pairs, hereinafter referred to as "training pair set 310". Each training sample pair includes two training samples. Figure 3 As shown, the training sample pair 311 includes training samples 311-1 and 311-2, the training sample pair 312 includes training samples 312-1 and 312-2, and the training sample pair 31N includes training samples 31N-1 and 31N-2.

[0049] Training samples 311-1, 312-1, ..., 31N-1 may be collectively or individually referred to as first training samples, and training samples 311-2, 312-2, ..., 31N-2 may be collectively or individually referred to as second training samples. The first training sample may have a first score with respect to the target attribute, and the second training sample may have a second score with respect to the target attribute. It should be understood that the first score and the second score may be the same or different.

[0050] Then, the training subsystem 210 can use the plurality of training sample pairs 311, 312, ..., 31N to train the classifier 250. The comparison result generated by the trained classifier 250 based on the first training sample and the second training sample is consistent with the relative level of the first score and the second score. In other words, the judgment of the trained classifier 250 on whether the target attribute of the first training sample is better than the target attribute of the second training sample is consistent with the relative level of the first score and the second score.

[0051] By combining training samples into training sample pairs, the training sample set used to train classifier 250 is actually expanded. Furthermore, the scores of training samples are typically subjective assessments given by evaluators. As mentioned above, this subjective assessment is limited by the evaluator. Therefore, the scores of training samples are likely to be unevenly distributed, which negatively impacts modeling for sample evaluation. The combined training sample pairs disclosed herein can mitigate this negative impact.

[0052] The following describes in detail an example process of training the classifier 250 using the training sample set 230. Assume that the vector x i ∈X represents the features of the i-th training sample in the training sample set 230, and uses y i ∈Y={r1,r2,...,r k} to represent the score of the i-th training sample, where r K >r K-1 >···>r1 and the symbol > indicates different scores r k The relative levels between To represent N X The training sample set 230 includes training samples.

[0053] In some implementations, the training subsystem 210 may combine any two training samples in the training sample set 230 into a training sample pair in the training pair set 310. In such an implementation, the training pair set 310 may be represented as The first training sample (x i ,y i ) and the second training sample (x j ,y j ) belong to the training sample set X. The label z used to train the classifier 250 j represents y i Is it greater than y j , and can be expressed by the following formula:

[0054]

[0055] Next, the training subsystem 210 can use the training set The classifier 250 is trained using each training sample pair in the training sample pair to compare which training sample in the training sample pair is better with respect to the target attribute.

[0056] In some other implementations, the training subsystem 210 may first select multiple anchored samples from the training sample set 230 to form an anchored sample set. For example, the training sample set 230 may be divided into multiple groups of training samples (e.g., buckets) according to the scores of the training samples in the training sample set 230 with respect to the target attribute. Each group of training samples has the same score with respect to the target attribute. Then, a certain number of training samples may be selected from each group of training samples as anchor samples. For example, in the example of oral fluency, the training sample set 230 may be divided into five groups or five buckets corresponding to scores of "1", "2", "3", "4", and "5", respectively. Then, a certain number of training samples may be selected (e.g., randomly selected) from each bucket as anchor samples.

[0057] Next, the training subsystem 210 may combine the anchor samples in the anchor sample set with the training samples in the training sample set 230 into training sample pairs. In this way, a training pair set 310 for training the classifier 250 may be generated.

[0058] Given a training sample set For each score rk (k=1, .., K), the training subsystem 210 can randomly select y=r k N A training samples (x i ,y i ) as anchor samples to form anchor sample set A. The training subsystem 210 may determine a set of N D The training sample subset D includes the training samples of the anchor sample set 230. The training sample subset D may not have any intersection with the anchor sample set A. For example, the training sample subset D may include the remaining training samples in the training sample set 230 excluding the anchor sample. Alternatively, the training sample subset D may have an intersection with the anchor sample set A. For example, the training sample subset D may include all training samples in the training sample set 230.

[0059] In this case, with N D The training sample subset D of training samples can be expressed as And the anchor sample set A can be expressed as where N A is the number of anchor samples for each score, and K is the total number of scores. Thus, the training set can be generated based on the Cartesian product between the training sample subset D and the anchor sample set A. In the training set P, the first training sample (x j ,y j ) belongs to the training sample subset D, and the second training sample Belongs to the anchor sample set A. Labels used to train the classifier 250 represents y j Is it greater than r k , and can be expressed by the following formula:

[0060]

[0061] Next, the training subsystem 210 can use the training set The classifier 250 is trained using each training sample pair in the training sample pair to compare which training sample in the training sample pair is better with respect to the target attribute.

[0062] Inference stage

[0063] Some implementations of the training phase are described above. The trained classifier 250 can be used by the inference subsystem 220 to determine the score of the input sample 170. Figure 4 A block diagram 400 is shown of the inference phase according to some implementations of the present disclosure.

[0064] In the inference phase, the inference subsystem 220 generates a set of comparison results based on the input sample 170 and the reference sample set 190. In the following, B is used to represent the reference sample set 190. Figure 2 As mentioned, in some implementations, the reference sample set 190 may include all training samples in the training sample set 230. That is, in such an implementation, B=X.

[0065] In other implementations, reference sample set 190 may include a portion of the training samples in training sample set 230. That is, in this implementation, B is a subset of X. As an example, in an online evaluation scenario, it may be necessary to provide evaluation results 180 in real time and quickly. If the number of training samples in training sample set 230 is large and the computing power of computing device 100 is limited, a portion of training sample set 230 may be selected as reference sample set 190. For example, reference sample set 190 may be anchor sample set A described above. It should be understood that even if anchor sample set A is not used during the training phase, reference sample set 190 may be generated from training sample set 230 using the method for generating anchor sample set A described above.

[0066] exist Figure 4 In the example of FIG, reference sample set 190 may include multiple reference samples 410-1, 410-2, ..., 410-M, which may be collectively or individually referred to as reference samples 410, where M is a natural number. Inference subsystem 220 may combine input sample 170 with each reference sample 410 to form a corresponding input-reference sample pair. Figure 4 Input-reference sample pairs 411, 412, ..., 41M are shown.

[0067] The inference subsystem 220 inputs the input-reference sample pairs 411, 412, ..., 41M to the trained classifier 250. The classifier 250 may generate corresponding comparison results 420-1, 420-2, ..., 420-M, which may be collectively or individually referred to as comparison results 420.

[0068] Each comparison result 420 indicates a probability that the target attribute of the input sample 170 is superior to the target attribute of the corresponding reference sample 410 in the reference sample set 190. The comparison result 420 may indicate the probability that the target attribute of the input sample 170 is superior to the target attribute of the corresponding reference sample 410 in any suitable manner. For example, the comparison result 420 may include a probability that the target attribute of the input sample 170 is superior to the target attribute of the corresponding reference sample 410. For another example, the comparison result may include a probability that the target attribute of the input sample 170 is not superior to (or inferior to) the target attribute of the corresponding reference sample 410.

[0069] In some implementations, the probability indicated by the comparison result 420 may be a binary-valued hard decision output by the classifier 250. For example, in such an implementation, a probability of 1 may indicate that the target attribute of the input sample 170 is better than the target attribute of the corresponding reference sample 410, while a probability of 0 may indicate that the target attribute of the input sample 170 is not better than the target attribute of the corresponding reference sample 410.

[0070] In some implementations, the probability indicated by the comparison result 420 can be a soft decision output by the classifier 250, such as a posterior probability estimated by the classifier 250. In such an implementation, the magnitude of the probability indicated by the comparison result 420 can represent the confidence that the target attribute of the input sample 170 is superior to the target attribute of the corresponding reference sample 410. In implementations that use a hard decision as the comparison result 420, cumulative quantization error may occur, which in turn may lead to a decrease in the accuracy of the final evaluation result. Utilizing the posterior probability estimated by the classifier 250 can effectively mitigate this cumulative quantization error, thereby facilitating accurate sample evaluation.

[0071] Use x′ to represent the features of the input sample 170, use y′ to represent the predicted score of the input sample 170 with respect to the target attribute, and use a i To represent the characteristics of the i-th reference sample in the reference sample set 190, use To represent the score of the i-th reference sample on the target attribute. Then, using the trained classifier 250, based on the feature x′ of the input sample and the feature a of the i-th reference sample i , the i-th comparison result can be generated:

[0072]

[0073] where a i ∈B. Therefore, in this implementation, F(x′, a i ) can be defined as the softmax transfer function in the last layer of the neural network used to implement the classifier 250.

[0074] Continue to refer Figure 4 Based on the set of comparison results 420-1, 420-2, ..., 420-M, the inference subsystem 220 determines at least one reference sample from the reference sample set 190 that matches the input sample 170 according to the target attribute. The determined at least one reference sample is also referred to herein as a "target reference sample." Figure 4Target reference sample 430 is shown in . Target reference sample 430 is close to or similar to the input sample in the target attribute space. For example, target reference sample 430 may be the reference sample closest to the input sample in the target attribute space. For example, in a language learning application scenario, target reference sample 430 may be closest to input sample 170 in terms of oral fluency compared to other reference samples in reference sample set 190. Next, inference subsystem 220 may determine a score for input sample 170 with respect to the target attribute based on the score for target reference sample 430 with respect to the target attribute, thereby providing evaluation result 180.

[0075] A variety of methods can be used to determine the target reference sample 430 based on the comparison result 420. In some implementations, the target reference sample 430 can be determined from the reference sample set 190 based on the probability value indicated by the comparison result 420. For example, in an implementation where the comparison result 420 includes a posterior probability output by the classifier 250 (e.g., Equation (3)), a comparison result with a posterior probability equal to or close to a predetermined threshold can be determined, and the reference sample 410 corresponding to such a comparison result can be determined as the target reference sample 430. The predetermined threshold can be determined based on the specific implementation of the classifier 250, such as 0.5. If the posterior probability included in the comparison result 420-2 is equal to the predetermined threshold or the difference from the predetermined threshold is within a certain range, the reference sample 410-2 can be determined as the target reference sample 430. In this implementation, if there is only one target reference sample 430, the inference subsystem 220 can determine the score of the target reference sample 430 with respect to the target attribute as the score of the input sample 170 with respect to the target attribute. If there are multiple target reference samples 430 , the inference subsystem 220 may determine a weighted average of the scores of the multiple target reference samples 430 with respect to the target attribute as the score of the input sample 170 with respect to the target attribute.

[0076] In some implementations, a ranking method can be used to determine the target reference sample 430 from the reference sample set 190. For example, the inference subsystem 220 can rank the input sample 170 and the reference samples 410 in the reference sample set 190 according to the target attribute based on the comparison result 420. The inference subsystem 220 can then determine the target reference sample 430 based on the ranking result.

[0077] Because reference samples 410 in reference sample set 190 have predetermined scores for the target attribute, such as scores assigned by an evaluator, reference samples 410 in reference sample set 190 may have predetermined rankings based on the predetermined scores. In this implementation, ranking input samples 170 and reference samples 410 may be implemented as predicting the ranking position of input sample 170 relative to the ordered reference samples 410 according to the target attribute. For example, the probabilities indicated by comparison results 420-1, 420-2, ..., 420-M may be added together. The result of the addition may correspond to the predicted ranking of input sample 170. The reference sample in reference sample set 190 whose predetermined ranking corresponds to the predicted ranking may be determined as target reference sample 430.

[0078] Continuing with the example of formula (3) above, the predicted ranking r(x′) of the input sample with feature x′ relative to the ordered reference sample 410 can be expressed as follows:

[0079]

[0080] In some implementations, the predicted ranking r(x′) may be rounded. The reference sample in the reference sample set 190 whose predetermined ranking corresponds to the rounded predicted ranking r(x′) (also referred to as the “first reference sample”) may be determined as the target reference sample 430. Accordingly, the score of the target reference sample 430 with respect to the target attribute may be determined as the score of the input sample 170 with respect to the target attribute. That is, the score s(x′) of the input sample with the feature representation x′ with respect to the target attribute may be expressed as follows:

[0081] s(x′)=s(a r(x′) ) (5).

[0082] In some implementations, a reference sample predetermined to be ranked before the predicted rank r(x′) (also referred to as a "second reference sample") and a reference sample predetermined to be ranked after the predicted rank r(x′) (also referred to as a "third reference sample") in the reference sample set 190 can be determined as the target reference sample 430. The score of the input sample 170 with respect to the target attribute can then be determined by weighting the respective scores of the second and third reference samples with respect to the target attribute. For example, the weighting coefficient can be determined based on the value of the predicted rank r(x′). In this implementation, by weighting the scores of multiple reference samples, a more fine-grained score for the input sample 170 can be obtained.

[0083] The above describes the inference process according to some implementations of the present disclosure. In these implementations, a series of comparison results between the input sample and the reference sample can be used to determine the reference sample in the reference sample set whose target attribute is close to or similar to the input sample. Then, the score of the input sample can be determined based on the score of such reference sample. It can be seen that the reference sample in the reference sample set actually plays the role of an anchor point, and such an anchor point is used to locate the input sample according to the target attribute. This makes the determined score of the input sample with respect to the target attribute accurate and robust. In this way, the solution of the present disclosure can achieve a higher correlation and a smaller mean absolute error close to that of manual labeling with fewer training samples.

[0084] After determining the score of the input sample 170 with respect to the target attribute, further processing can be performed based on the score in different application scenarios. For example, in a language learning scenario, courses that match the score can be recommended to the user based on the score. In a credit rating scenario, services that match the score can be recommended to the user based on the score.

[0085] Furthermore, the sample evaluation scheme of the present disclosure can also be used to determine multiple scores for input samples 170 with respect to different attributes (which can be considered different dimensions). Multiple scores for multiple input samples associated with different users with respect to target attribute(s) can also be determined. The multiple input samples can then be clustered based on these multiple scores. In this way, users associated with multiple input samples in the same cluster can be considered to be closest in terms of the target attribute(s). For example, in language learning or other teaching scenarios, such users can be grouped into the same group or class.

[0086] In some cases, the input sample 170 and the score of the input sample 170 determined by the inference subsystem 220 can be added to the training sample set 230 to update the classifier 250. For example, if the score of the input sample 170 determined by the inference subsystem 220 is manually assessed to be accurate, the input sample 170 and its score can be used to further train the classifier 250. Alternatively or additionally, the input sample 170 and its score can also be added to the reference sample set 190.

[0087] Example method and example implementation

[0088] Figure 5 Flowchart of a method 500 for sample evaluation according to some implementations of the present disclosure is shown. The method 500 may be implemented by the computing device 100, for example, at the sample processing module 122 in the memory 120 of the computing device 100.

[0089] like Figure 5As shown, at block 510, the computing device 100 generates a set of comparison results based on the input sample and the reference sample set. Each comparison result indicates a probability that the target attribute of the input sample is superior to the target attribute of a corresponding reference sample in the reference sample set. At block 520, the computing device 100 determines, based on the set of comparison results, at least one reference sample from the reference sample set that matches the input sample in terms of the target attribute. The target attribute of the input sample is close to the target attribute of the at least one reference sample. At block 530, the computing device 100 determines a score for the input sample with respect to the target attribute based on the corresponding score of the at least one reference sample with respect to the target attribute.

[0090] In some implementations, determining at least one reference sample from the reference sample set includes: ranking the input sample and reference samples in the reference sample set according to the target attribute based on a set of comparison results; and determining the at least one reference sample based on the ranking results.

[0091] In some implementations, determining at least one reference sample based on the ranking results includes determining a first reference sample in the reference sample set that has the same ranking as the input sample as the at least one reference sample. Determining a score of the input sample with respect to the target attribute includes determining a score of the input sample with respect to the target attribute based on the score of the first reference sample with respect to the target attribute.

[0092] In some implementations, determining at least one reference sample based on the ranking results includes determining a second reference sample ranked before the input sample and a third reference sample ranked after the input sample in the reference sample set as the at least one reference sample. Determining a score of the input sample with respect to the target attribute includes determining the score of the input sample with respect to the target attribute by weighting the respective scores of the second reference sample and the third reference sample with respect to the target attribute.

[0093] In some implementations, generating a set of comparison results includes: using a trained classifier, based on the input sample and a fourth reference sample in the reference sample set, generating a comparison result for the fourth reference sample, the comparison result indicating a probability that a target attribute of the input sample is better than a target attribute of the fourth reference sample.

[0094] In some implementations, method 500 further includes: generating a plurality of training sample pairs based on the training sample set, one of the plurality of training sample pairs including a first training sample having a first score with respect to the target attribute and a second training sample having a second score with respect to the target attribute; and training a classifier using the plurality of training sample pairs so that a comparison result generated by the trained classifier based on the first training sample and the second training sample conforms to the relative levels of the first score and the second score.

[0095] In some implementations, method 500 further includes determining a reference sample set based on the training sample set.

[0096] In some implementations, determining a reference sample set based on a training sample set includes: dividing the training sample set into multiple groups of training samples according to scores of the training samples in the training sample set with respect to a target attribute; and selecting a reference sample in the reference sample set from each group of training samples in the multiple groups of training samples.

[0097] Based on the above description, it can be seen that the sample evaluation solution implemented according to the present disclosure transforms the score prediction problem into a series of relatively simple binary comparisons or binary classifications: that is, a comparison of the quality of the input sample against a reference sample. This series of binary comparisons helps accurately identify reference samples that are similar to the input sample in terms of target attributes. This ensures the accuracy of the input sample score determined based on the reference sample score. In this way, more efficient, accurate, and robust sample evaluation can be achieved.

[0098] Some example implementations of the present disclosure are listed below.

[0099] In one aspect, the present disclosure provides a computer-implemented method. The method includes: generating a set of comparison results based on an input sample and a reference sample set, each comparison result indicative of a probability that a target attribute of the input sample is superior to the target attribute of a corresponding reference sample in the reference sample set; determining, based on the set of comparison results, at least one reference sample from the reference sample set that matches the input sample according to the target attribute; and determining a score for the input sample with respect to the target attribute based on the corresponding score of the at least one reference sample with respect to the target attribute.

[0100] In some implementations, determining the at least one reference sample from the reference sample set includes: ranking the input sample and reference samples in the reference sample set according to the target attribute based on the set of comparison results; and determining the at least one reference sample based on a result of the ranking.

[0101] In some implementations, determining the at least one reference sample based on the ranking result includes determining a first reference sample in the reference sample set that has the same ranking as the input sample as the at least one reference sample. Determining the score of the input sample with respect to the target attribute includes determining the score of the input sample with respect to the target attribute based on the score of the first reference sample with respect to the target attribute.

[0102] In some implementations, determining the at least one reference sample based on the ranking result includes determining a second reference sample ranked before the input sample and a third reference sample ranked after the input sample in the reference sample set as the at least one reference sample. Determining the score of the input sample with respect to the target attribute includes determining the score of the input sample with respect to the target attribute by weighting the respective scores of the second reference sample and the third reference sample with respect to the target attribute.

[0103] In some implementations, generating the set of comparison results includes: using a trained classifier to generate a comparison result for the fourth reference sample based on the input sample and a fourth reference sample in the reference sample set, the comparison result indicating the probability that the target attribute of the input sample is better than the target attribute of the fourth reference sample.

[0104] In some implementations, the method further includes: generating a plurality of training sample pairs based on a training sample set, one of the plurality of training sample pairs including a first training sample having a first score with respect to the target attribute and a second training sample having a second score with respect to the target attribute; and training the classifier using the plurality of training sample pairs so that a comparison result generated by the trained classifier based on the first training sample and the second training sample conforms to the relative levels of the first score and the second score.

[0105] In some implementations, the method further includes determining the reference sample set based on the training sample set.

[0106] In some implementations, determining the reference sample set based on the training sample set includes: dividing the training sample set into multiple groups of training samples according to the scores of the training samples in the training sample set with respect to the target attribute; and selecting a reference sample in the reference sample set from each group of training samples in the multiple groups of training samples.

[0107] In another aspect, the present disclosure provides an electronic device. The electronic device includes: a processing unit; and a memory coupled to the processing unit and containing instructions stored thereon, wherein when executed by the processing unit, the instructions cause the device to perform actions, the actions comprising: generating a set of comparison results based on an input sample and a reference sample set, each comparison result in the set of comparison results indicating a probability that a target attribute of the input sample is better than the target attribute of a corresponding reference sample in the reference sample set; determining, based on the set of comparison results, at least one reference sample from the reference sample set that matches the input sample according to the target attribute; and determining a score for the input sample with respect to the target attribute based on the corresponding score of the at least one reference sample with respect to the target attribute.

[0108] In some implementations, determining the at least one reference sample from the reference sample set includes: ranking the input sample and reference samples in the reference sample set according to the target attribute based on the set of comparison results; and determining the at least one reference sample based on a result of the ranking.

[0109] In some implementations, determining the at least one reference sample based on the ranking result includes determining a first reference sample in the reference sample set that has the same ranking as the input sample as the at least one reference sample. Determining the score of the input sample with respect to the target attribute includes determining the score of the input sample with respect to the target attribute based on the score of the first reference sample with respect to the target attribute.

[0110] In some implementations, determining the at least one reference sample based on the ranking result includes determining a second reference sample ranked before the input sample and a third reference sample ranked after the input sample in the reference sample set as the at least one reference sample. Determining the score of the input sample with respect to the target attribute includes determining the score of the input sample with respect to the target attribute by weighting the respective scores of the second reference sample and the third reference sample with respect to the target attribute.

[0111] In some implementations, generating the set of comparison results includes: using a trained classifier to generate a comparison result for the fourth reference sample based on the input sample and a fourth reference sample in the reference sample set, the comparison result indicating the probability that the target attribute of the input sample is better than the target attribute of the fourth reference sample.

[0112] In some implementations, the method further includes: generating a plurality of training sample pairs based on a training sample set, one of the plurality of training sample pairs including a first training sample having a first score with respect to the target attribute and a second training sample having a second score with respect to the target attribute; and training the classifier using the plurality of training sample pairs so that a comparison result generated by the trained classifier based on the first training sample and the second training sample conforms to the relative levels of the first score and the second score.

[0113] In some implementations, the method further includes determining the reference sample set based on the training sample set.

[0114] In some implementations, determining the reference sample set based on the training sample set includes: dividing the training sample set into multiple groups of training samples according to the scores of the training samples in the training sample set with respect to the target attribute; and selecting a reference sample in the reference sample set from each group of training samples in the multiple groups of training samples.

[0115] In yet another aspect, the present disclosure provides a computer program product tangibly stored in a non-transitory computer storage medium and comprising machine-executable instructions that, when executed by a device, cause the device to perform the method of the above aspect.

[0116] In yet another aspect, the present disclosure provides a computer-readable medium having machine-executable instructions stored thereon. When the machine-executable instructions are executed by a device, the device is caused to perform the method of the above aspect.

[0117] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), and the like.

[0118] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0119] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0120] In addition, although each operation is described in a specific order, this should be understood as requiring such operation to be performed in the specific order shown or in a sequential order, or requiring that all illustrated operations should be performed to obtain the desired result. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate implementation can also be implemented in a single implementation in combination. On the contrary, the various features described in the context of a single implementation can also be implemented in multiple implementations individually or in any suitable sub-combination mode.

[0121] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

1. A computer-implemented method comprising: generating a set of comparison results based on an input sample and a reference sample set, each comparison result in the set of comparison results indicating a probability that a target attribute of the input sample is better than the target attribute of a corresponding reference sample in the reference sample set, the input sample comprising at least one of the following: a speech sample, an image sample, or a text sample, each of the comparison results being determined based on a feature of the input sample, a predicted score of the input sample with respect to the target attribute, a feature of the corresponding reference sample, and a score of the corresponding reference sample with respect to the target attribute; Based on the set of comparison results, determining at least one reference sample from the reference sample set that matches the input sample according to the target attribute, Wherein determining the at least one reference sample from the reference sample set comprises: Ranking the input sample and reference samples in the reference sample set according to the target attribute based on the set of comparison results; Based on the ranking result, determining a first reference sample, a second reference sample, and a third reference sample, wherein the first reference sample is ranked the same as the input sample, the second reference sample is ranked before the input sample, and the third reference sample is ranked after the input sample; and Determining a score of the input sample with respect to the target attribute based on a corresponding score of the at least one reference sample with respect to the target attribute comprises: The score of the input sample with respect to the target attribute is determined by weighting the respective scores of the first reference sample, the second reference sample, and the third reference sample with respect to the target attribute.

2. The method of claim 1 , wherein generating the set of comparison results comprises: Using the trained classifier, based on the input sample and a fourth reference sample in the reference sample set, a comparison result for the fourth reference sample is generated, wherein the comparison result indicates a probability that the target attribute of the input sample is better than the target attribute of the fourth reference sample.

3. The method according to claim 2, further comprising: Based on the training sample set, a plurality of training sample pairs are generated, one of the plurality of training sample pairs includes a first training sample having a first score with respect to the target attribute and a second training sample having a second score with respect to the target attribute; as well as The classifier is trained using the plurality of training sample pairs, so that a comparison result generated by the trained classifier based on the first training sample and the second training sample conforms to the relative levels of the first score and the second score.

4. The method according to claim 3, further comprising: The reference sample set is determined based on the training sample set.

5. The method according to claim 4, wherein determining the reference sample set based on the training sample set comprises: Dividing the training sample set into multiple groups of training samples according to scores of the training samples in the training sample set with respect to the target attribute; as well as From each group of training samples in the multiple groups of training samples, a reference sample in the reference sample set is selected.

6. An electronic device comprising: processing unit; as well as a memory coupled to the processing unit and containing instructions stored thereon, the instructions, when executed by the processing unit, causing the apparatus to perform actions, the actions comprising: generating a set of comparison results based on an input sample and a reference sample set, each comparison result in the set of comparison results indicating a probability that a target attribute of the input sample is better than the target attribute of a corresponding reference sample in the reference sample set, the input sample comprising at least one of the following: a speech sample, an image sample, or a text sample, each of the comparison results being determined based on a feature of the input sample, a predicted score of the input sample with respect to the target attribute, a feature of the corresponding reference sample, and a score of the corresponding reference sample with respect to the target attribute; Based on the set of comparison results, determining at least one reference sample from the reference sample set that matches the input sample according to the target attribute, Wherein determining the at least one reference sample from the reference sample set comprises: Ranking the input sample and reference samples in the reference sample set according to the target attribute based on the set of comparison results; Based on the ranking result, determining a first reference sample, a second reference sample, and a third reference sample, wherein the first reference sample is ranked the same as the input sample, the second reference sample is ranked before the input sample, and the third reference sample is ranked after the input sample; and Determining a score of the input sample with respect to the target attribute based on a corresponding score of the at least one reference sample with respect to the target attribute comprises: The score of the input sample with respect to the target attribute is determined by weighting the respective scores of the first reference sample, the second reference sample, and the third reference sample with respect to the target attribute.

7. The apparatus of claim 6, wherein generating the set of comparison results comprises: Using the trained classifier, based on the input sample and a fourth reference sample in the reference sample set, a comparison result for the fourth reference sample is generated, wherein the comparison result indicates a probability that the target attribute of the input sample is better than the target attribute of the fourth reference sample.

8. The apparatus according to claim 7, further comprising: Based on the training sample set, a plurality of training sample pairs are generated, one of the plurality of training sample pairs includes a first training sample having a first score with respect to the target attribute and a second training sample having a second score with respect to the target attribute; as well as The classifier is trained using the plurality of training sample pairs, so that a comparison result generated by the trained classifier based on the first training sample and the second training sample conforms to the relative levels of the first score and the second score.

9. The apparatus according to claim 8, further comprising: The reference sample set is determined based on the training sample set.

10. The apparatus according to claim 9, wherein determining the reference sample set based on the training sample set comprises: Dividing the training sample set into multiple groups of training samples according to scores of the training samples in the training sample set with respect to the target attribute; as well as From each group of training samples in the multiple groups of training samples, a reference sample in the reference sample set is selected.

11. A computer program product tangibly stored in a non-transitory computer storage medium and comprising machine-executable instructions that, when executed by a device, cause the device to perform actions comprising: generating a set of comparison results based on an input sample and a reference sample set, each comparison result in the set of comparison results indicating a probability that a target attribute of the input sample is better than the target attribute of a corresponding reference sample in the reference sample set, the input sample comprising at least one of the following: a speech sample, an image sample, or a text sample, each of the comparison results being determined based on a feature of the input sample, a predicted score of the input sample with respect to the target attribute, a feature of the corresponding reference sample, and a score of the corresponding reference sample with respect to the target attribute; Based on the set of comparison results, determining at least one reference sample from the reference sample set that matches the input sample according to the target attribute, Wherein determining the at least one reference sample from the reference sample set comprises: Ranking the input sample and reference samples in the reference sample set according to the target attribute based on the set of comparison results; Based on the ranking result, determining a first reference sample, a second reference sample, and a third reference sample, wherein the first reference sample is ranked the same as the input sample, the second reference sample is ranked before the input sample, and the third reference sample is ranked after the input sample; and Determining a score of the input sample with respect to the target attribute based on a corresponding score of the at least one reference sample with respect to the target attribute comprises: The score of the input sample with respect to the target attribute is determined by weighting the respective scores of the first reference sample, the second reference sample, and the third reference sample with respect to the target attribute.

Images