Scoring method and device for annotation data and related equipment

Abstract

The invention provides an annotation data scoring method and device and related equipment, and relates to the technical field of artificial intelligence. Original data and annotation data in to-be-scored data are input into a first deep learning model at the same time, and the original data and the annotation data are scored through the first deep learning model; in this way, in the scoring process of the annotation data, the relevance between the annotation data and the original data is considered, and meanwhile the original data and the annotation data are input into the first deep learning model, so that the first deep learning model comprehensively understands the context; and the first deep learning model captures a detail problem and a context mismatching problem in the annotation data, so that the accuracy of annotation data scoring is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a method and device for scoring annotated data and related equipment. BACKGROUND

[0002] For the quality score of annotated data, the existing technology usually adopts the way of artificial intelligence (AI) pre-annotation and manual quality inspection to score the annotation quality. Or, according to the annotated data set, the large model is fine-tuned and trained, and based on the annotated data predicted by the large model, the annotated data predicted by the large model is scored based on the result of manual annotation. The above scoring methods are only for scoring annotated data, without considering the relationship between annotated data and original data, thereby resulting in the problem of low accuracy of the scoring value obtained when scoring annotated data in the existing technology. SUMMARY

[0003] The present application provides a method and device for scoring annotated data and related equipment, which solves the problem of low accuracy of the scoring value obtained when scoring annotated data in the existing technology.

[0004] To solve the above problems, the present application is implemented as follows:

[0005] In a first aspect, the present application provides a method for scoring annotated data, which comprises:

[0006] Obtaining data to be scored, the data to be scored including original data and annotated data, the annotated data including annotated content of the original data;

[0007] Inputting the data to be scored into a first deep learning model for scoring to obtain a scoring value corresponding to the annotated data, the first deep learning model being used to score the accuracy of the annotated data based on the original data.

[0008] Optionally, before the data to be scored is input into the first deep learning model for scoring to obtain the scoring value, the method further comprises:

[0009] Obtaining a plurality of training data sets, wherein each training data set includes a plurality of training samples, the training sample including sample original data and sample annotated data, the sample annotated data including annotated content of the sample original data, the scoring values of the plurality of sample annotated data included in the training data set being the same; the scoring values of the sample annotated data included in different training data sets in the plurality of training data sets being different;

[0010] training a second deep learning model based on the plurality of training data sets, and obtaining the first deep learning model after the training is completed.

[0011] Optionally, the inputting the to-be-scored data into the first deep learning model for scoring to obtain a score value corresponding to the annotated data comprises:

[0012] performing word segmentation on the original data based on the first deep learning model to obtain a first word segmentation sequence, and performing word segmentation on the annotated data based on the first deep learning model to obtain a second word segmentation sequence;

[0013] performing vector conversion on the first word segmentation sequence and the second word segmentation sequence based on the first deep learning model to obtain a plurality of feature vectors, the plurality of feature vectors being semantic representations of the first word segmentation sequence and the second word segmentation sequence, and one of the feature vectors being used to represent semantics of one word in the first word segmentation sequence or one word in the second word segmentation sequence;

[0014] scoring the plurality of feature vectors based on the first deep learning model to obtain the score value corresponding to the annotated data.

[0015] Optionally, the first deep learning model comprises an input layer, an encoding layer, a quality grading layer, a linear weighting layer and an output layer connected in sequence; the input layer is used to perform word segmentation on the original data to obtain the first word segmentation sequence, and perform word segmentation on the annotated data to obtain the second word segmentation sequence; and the encoding layer is used to perform vector conversion on the first word segmentation sequence and the second word segmentation sequence to obtain the plurality of feature vectors.

[0016] The scoring the plurality of feature vectors based on the first deep learning model to obtain the score value corresponding to the annotated data comprises:

[0017] scoring the plurality of feature vectors based on the quality grading layer of the first deep learning model to obtain a plurality of score probabilities, the plurality of score probabilities corresponding to a plurality of preset score values one by one, and the score probability being used to represent a probability that the score value of the annotated data is the corresponding preset score value;

[0018] performing weighted calculation on the plurality of score probabilities based on the linear weighting layer of the first deep learning model to obtain the score value corresponding to the annotated data;

[0019] outputting the score value corresponding to the annotated data based on the output layer.

[0020] Optionally, the scoring the plurality of feature vectors based on the quality grading layer of the first deep learning model to obtain a plurality of score probabilities comprises:

[0021] generate a bias matrix b and a weight matrix W based on the plurality of feature vectors and the plurality of preset score values, wherein the weight matrix W is a d*x matrix, d is the number of the feature vectors, x is the number of the plurality of preset score values, the value of the i-th row and the j-th column in the W is the product of the i-th feature value in the feature vector and the j-th preset score value in the plurality of preset score values, the bias matrix b is a 1 row x column matrix;

[0022] calculate an original score vector y of the labeled data based on the bias matrix and the weight matrix, wherein the original score vector y = σ (W*d+b), σ is an activation parameter, and the original score vector y includes x first score values, which correspond to the x preset score values one by one;

[0023] calculate a plurality of score probabilities p based on the plurality of preset score values and the original score vector y i , wherein, the p i is a probability that the score value of the labeled data corresponds to the i-th preset score value in the plurality of preset score values, the value of the y i is the i-th first score value in the x first score values, i is a corresponding number value of the i-th preset score value in the plurality of preset score values, and e is a natural constant.

[0024] Optionally, the first deep learning model includes a linear weighting layer, and the linear weighting layer is configured to perform weighting calculation on the plurality of score probabilities to obtain the score value corresponding to the labeled data.

[0025] Optionally, the first deep learning model includes a linear weighting layer, and the linear weighting layer is configured to perform weighting calculation on the plurality of score probabilities to obtain the score value corresponding to the labeled data.

[0026] Optionally, after the inputting the to-be-scored data into the first deep learning model to obtain the score value corresponding to the labeled data, the method further includes:

[0027] In a case where the score value corresponding to the to-be-scored data satisfies a preset condition, inputting the to-be-scored data into the first deep learning model to train the first deep learning model, and obtaining a third deep learning model after the training is completed, wherein the preset condition is that the score value corresponding to the to-be-scored data is greater than or equal to a preset threshold.

[0028] In a second aspect, an embodiment of the present application provides a scoring device for labeled data. The device includes:

[0029] The acquisition module is configured to acquire to-be-scored data, wherein the to-be-scored data includes original data and labeled data, and the labeled data includes labeled content of the original data.

[0030] The scoring module is configured to input the to-be-scored data into a first deep learning model for scoring to obtain a score value corresponding to the labeled data, wherein the first deep learning model is configured to score accuracy of the labeled data based on the original data.

[0031] In a third aspect, the present application also provides an electronic device, which includes a processor, a memory, and a computer program stored in the memory and executable on the processor, and the computer program, when executed by the processor, implements the steps in the method according to the first aspect.

[0032] In a fourth aspect, the present application also provides a computer-readable storage medium, which stores a computer program, and the computer program, when executed by a processor, implements the steps in the method according to the first aspect.

[0033] In a fifth aspect, the present application also provides a computer program product, which includes computer instructions, and the computer instructions, when executed by a processor, implement the steps in the method according to the first aspect.

[0034] The present application provides a scoring method and device for labeled data and related equipment, and the method includes: acquiring to-be-scored data, wherein the to-be-scored data includes original data and labeled data, and the labeled data includes labeled content of the original data; inputting the to-be-scored data into a first deep learning model for scoring to obtain a score value corresponding to the labeled data, wherein the first deep learning model is configured to score accuracy of the labeled data based on the original data. The present application inputs the original data and the labeled data in the to-be-recognized and scored data into the trained first deep learning model at the same time, scores the original data and the labeled data through the first deep learning model, and thus obtains the score value corresponding to the labeled data. Therefore, in the scoring process of the labeled data, the correlation between the labeled data and the original data is considered, and the original data and the labeled data are input into the first deep learning model, so that the first deep learning model comprehensively understands the context, captures the detail problems and context mismatch problems in the labeled data, and thus improves the accuracy of the labeled data scoring. BRIEF DESCRIPTION OF DRAWINGS

[0035] In order to more clearly illustrate the technical solutions of the present application, the drawings needed to be used in the description of the present application will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0036] Figure 1 A flowchart of a scoring method for labeled data provided by an embodiment of the present application is shown in FIG. 1.

[0037] Figure 2 A structural diagram of a first deep learning model provided by an embodiment of the present application is shown in FIG. 2.

[0038] Figure 3 A structural diagram of a scoring device for labeled data provided by an embodiment of the present application is shown in FIG. 3.

[0039] Figure 4 A structural diagram of an electronic device provided by an embodiment of the present application is shown in FIG. 4. DETAILED DESCRIPTION

[0040] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0041] The terms "first", "second", and the like in the embodiments of the present application are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or device including a series of steps or units does not necessarily have to include only those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to the process, method, product, or device. In addition, "and / or" is used in the present application to represent at least one of the connected objects, for example, A and / or B and / or C represents 7 cases including A alone, B alone, C alone, A and B both exist, B and C both exist, A and C both exist, and A, B and C all exist.

[0042] Referring to Figure 1 , Figure 1 A flowchart of a scoring method for labeled data provided by an embodiment of the present application is shown in FIG. 1. Figure 1 The scoring method for labeled data can include the following steps as shown in FIG. 1.

[0043] Step 101, obtaining to-be-scored data, the to-be-scored data including original data and annotation data, the annotation data including annotation content of the original data.

[0044] In the embodiment, the to-be-scored data includes original data and annotation data, wherein the original data can be text, picture, video, etc., and the annotation data is annotation or explanation content of the original data. For example, the original data is a text "This product is really great, I feel very satisfied", and the annotation data corresponds to the sentiment label of each text data, which can be "positive" for example. Or the original data is "I am disappointed with this service, it is far from my expectation", and the annotation data can be "negative". For example, the original data is a text "Machine learning is changing the way we work, especially in the field of data analysis", and the annotation data corresponds to the technical field classification label of each text data, which can be "artificial intelligence" for example. Or the original data is "Virtual reality technology is bringing a new experience to the game industry", and the annotation data can be "virtual reality".

[0045] Step 102, inputting the to-be-scored data into a first deep learning model for scoring to obtain a score value corresponding to the annotation data, the first deep learning model being used for scoring the accuracy of the annotation data based on the original data.

[0046] In the embodiment, the first deep learning model is a trained model, and its function is to score the accuracy of the annotation data based on the original data. Specifically, the first deep learning model can be a data annotation quality evaluation model, which is a model used to measure and analyze the accuracy and consistency of data annotation results. In the field of machine learning and artificial intelligence, data annotation is the basis for training models, and high-quality annotation data directly affects the performance of the model.

[0047] The accuracy of the annotation data refers to the accuracy of the annotation content of the original data by the annotation data. Specifically, the higher the accuracy of the annotation data, the more accurate the annotation of the original data by the annotation data, and the lower the accuracy of the annotation data, the less accurate the annotation of the original data by the annotation data.

[0048] In the training process of the first deep learning model, a plurality of score levels can be set in advance to score the annotation data. For example, the score levels can include five levels, i.e. perfect, good, general, poor and unqualified, and the output score value can be any one of the five levels, for example, the perfect corresponds to a score value of 5, the good corresponds to a score value of 4, the general corresponds to a score value of 3, the poor corresponds to a score value of 2, and the unqualified corresponds to a score value of 1.

[0049] It should be noted that during the training process of the first deep learning model, it is also necessary to use the original data and the labeled data to train the first deep learning model at the same time, so that the first deep learning model can score the correlation between the original data and the labeled data, allowing the model to capture the details and context mismatch problems in the annotations, improve the judgment of the annotation accuracy and applicability, and thereby improve the accuracy of the scoring of the labeled data.

[0050] In this application, the original data and the labeled data in the data to be scored are simultaneously input into the first deep learning model, and the original data and the labeled data are scored by the first deep learning model to obtain the corresponding scoring value of the labeled data. Therefore, in the scoring process of the labeled data, the correlation between the labeled data and the original data is taken into account. At the same time, the original data and the labeled data are input into the first deep learning model, so that the first deep learning model fully understands the context, and allows the first deep learning model to capture the details and context mismatch problems in the labeled data, thereby improving the accuracy of the labeled data scoring.

[0051] In some feasible implementations, optionally, before inputting the data to be scored into the first deep learning model for scoring and obtaining a scoring value, the method further includes:

[0052] Acquire multiple training data sets, wherein each of the training data sets includes multiple training samples, the training samples include sample original data and sample annotated data, the sample annotated data includes annotated content of the sample original data, and the multiple sample annotated data included in the training data sets have the same score value; and the sample annotated data included in different training data sets have different score values;

[0053] The second deep learning model is trained based on the multiple training data sets, and after the training is completed, the first deep learning model is obtained.

[0054] In this embodiment, before using the first deep learning model, it is necessary to use multiple training data sets to train the second deep learning model to obtain the first deep learning model. Specifically, the second deep learning model is an untrained data annotation quality assessment model.

[0055] In the embodiment, multiple training data sets are included, each of which includes multiple training samples, each of which includes a set of sample original data and sample annotation data, and the sample annotation data includes the annotation content of the sample original data. It needs to be noted that the score values of the multiple sample annotation data included in each training data set are the same, that is, the score values of the sample annotation data in each training data set are the same score, for example, 5 points. In multiple training data sets, the score values of the sample annotation data included in different training data sets are different, for example, when there are 5 training data sets, the score values of the sample annotation data in each training data set can correspond to 1 point, 2 points, 3 points, 4 points and 5 points respectively.

[0056] It needs to be noted that a validation set and a test set can also be set, and the allocation ratio of the training data set, the validation set and the test set can be 8:1:1, so that the training results of the multiple training data sets are verified and tested through the validation set and the test set, thereby ensuring the accuracy of the model training.

[0057] Optionally, the method further comprises:

[0058] performing word segmentation processing on the original data based on the first deep learning model to obtain a first word segmentation sequence, and performing word segmentation processing on the annotation data to obtain a second word segmentation sequence;

[0059] performing vector conversion on the first word segmentation sequence and the second word segmentation sequence based on the first deep learning model to obtain multiple feature vectors, the multiple feature vectors being semantic representations of the first word segmentation sequence and the second word segmentation sequence, and one of the feature vectors being used to represent the semantic of one word in the first word segmentation sequence or one word in the second word segmentation sequence;

[0060] performing scoring on the multiple feature vectors based on the first deep learning model to obtain the score value corresponding to the annotation data.

[0061] In the embodiment, the score value corresponding to the annotation data is scored according to the first deep learning model. Specifically, after the data to be scored is input into the first deep learning model, the original data and the annotation data need to be first processed. Specifically, the original data is processed to obtain a first word segmentation sequence, and the annotation data is processed to obtain a second word segmentation sequence.

[0062] After the word segmentation processing is completed, the first word segmentation sequence and the second word segmentation sequence are vectorized to obtain multiple feature vectors. Exemplarily, the first word segmentation sequence and the second word segmentation sequence are vectorized by the encoder of the first deep learning model to generate multiple feature vectors, wherein the multiple feature vectors are semantic representations of the first word segmentation sequence and the second word segmentation sequence, and each feature vector is used to represent: the semantics of a word in the first word segmentation sequence or a word in the second word segmentation sequence.

[0063] After obtaining multiple feature vectors, scoring is performed based on the multiple feature vectors to obtain the score value corresponding to the labeled data, where the scoring results corresponding to the score value can be divided into five levels of 1-5. Therefore, this application provides more detailed quality feedback and better reflects the differences between different labeled data.

[0064] Optionally, the first deep learning model includes an input layer, an encoding layer, a quality grading layer, a linear weighting layer, and an output layer connected in sequence; the input layer is used to perform word segmentation processing on the original data to obtain the first word segmentation sequence, and to perform word segmentation processing on the labeled data to obtain the second word segmentation sequence; the encoding layer is used to perform vector conversion on the first word segmentation sequence and the second word segmentation sequence to obtain the multiple feature vectors;

[0065] Scoring the plurality of feature vectors based on the first deep learning model to obtain score values ​​corresponding to the labeled data includes:

[0066] Scoring the multiple feature vectors based on the quality grading layer of the first deep learning model to obtain multiple scoring probabilities, where the multiple scoring probabilities correspond one-to-one to multiple preset scoring values, and the scoring probabilities are used to represent the probability that the scoring value of the labeled data is the corresponding preset scoring value;

[0067] Based on the linear weighted layer of the first deep learning model, performing weighted calculation on the multiple scoring probabilities to obtain a scoring value corresponding to the labeled data;

[0068] Based on the output layer, the score value corresponding to the labeled data is output.

[0069] In this embodiment, if Figure 2 As shown, Figure 2 This is a structural diagram of the first deep learning model provided in this embodiment, wherein the first deep learning model includes an input layer, a coding layer, a quality grading layer, a linear weighted layer and an output layer connected in sequence.

[0070] Specifically, the input layer includes a data preprocessing process, which performs word segmentation on the original data to obtain a first word segmentation sequence, and performs word segmentation on the labeled data to obtain a second word segmentation sequence. Specifically, the WordPiece tokenizer is used to perform word segmentation on the text, and the text content is split into a token sequence through word segmentation. A [CLS] mark is added at the beginning of the original content, and a [SEP] mark is added at the end of the original content. The [SEP] mark is used to mark the end of the original content and the beginning of the labeled content.

[0071] The encoding layer is an encoder stage of the deep learning model. Specifically, data is passed up from bottom to top, self-attention is used in each layer, and the result is passed through a feedforward neural network to the next encoder. Each position of the encoder outputs a vector representing the semantic representation of the position, and finally a plurality of feature vectors are obtained.

[0072] The quality grading layer is used to score the plurality of feature vectors to obtain a plurality of scoring probabilities. For example, if the scoring is divided into 1-5 levels, five scoring probabilities (w1-w5) will be generated, and the five scoring probabilities correspond to the 1-5 levels, i.e., the probability of scoring 1, the probability of scoring 2, …, the probability of scoring 5.

[0073] The linear weighting layer is used to weight the five scoring probabilities based on five weights (q1-q5) to obtain a final score value Σ corresponding to the labeled data.

[0074] The output layer is used to output the score value corresponding to the labeled data. It needs to be noted that for the labeled data, the quality score is calculated by the quality evaluation model. If the quality score is within the set threshold, it is considered that the quality meets the requirements. The labeling is completed. If the quality is judged as unqualified, it needs to enter the expert review and labeling stage. The final labeling is performed by an expert. Optionally, after the first deep learning model is used to score the to-be-scored data to obtain the score value corresponding to the labeled data, the method further includes: in a case where the score value corresponding to the to-be-scored data meets a preset condition, inputting the to-be-scored data into the first deep learning model to train the first deep learning model, and obtaining a third deep learning model after the training is completed. The preset condition is that the score value corresponding to the to-be-scored data is greater than or equal to a preset threshold.

[0075] In the embodiment, the judgment is made in the first deep learning model, and the data meeting the requirements and the labeling data completed by the expert are fed back to the training to update the model, forming a closed loop and continuously optimizing the model. The preset condition includes that the score value corresponding to the to-be-scored data is greater than or equal to a preset threshold, which can be set according to actual conditions, for example, set to 4.5 points, which is not limited in the embodiment.

[0076] Optionally, the quality grading layer based on the first deep learning model scores the plurality of feature vectors to obtain a plurality of score probabilities, including:

[0077] According to the plurality of feature vectors and the plurality of preset score values, a bias matrix b and a weight matrix W are generated, wherein the weight matrix W is a matrix of d*x, d is the number of feature vectors, x is the number of the plurality of preset score values, the value of the i-th row and the j-th column in W is the product of the i-th feature value in the feature vector and the j-th preset score value in the plurality of preset score values, and the bias matrix b is a matrix of 1 row and x columns;

[0078] The original score vector y of the labeling data is calculated based on the bias matrix and the weight matrix, wherein y=σ(W*d+b), σ is an activation parameter, and the original score vector y includes x first score values, which correspond to x preset score values one by one;

[0079] Based on the plurality of preset score values and the original score vector y, a plurality of score probabilities p are calculated i , wherein, The p i is the probability that the score value of the labeling data corresponds to the i-th score value in the plurality of preset score values, and the value of y i is the i-th first score value in the x first score values, i is the corresponding number value of the i-th preset score value in the plurality of preset score values, and e is a natural constant.

[0080] In the embodiment, the quality grading layer includes x preset score values, that is, x quality score levels. Specifically, the output vector of the classification problem [CLS] token can be used to represent the semantics of the entire sentence, that is, the [CLS] token is located at the front end of the sentence, and after being encoded by multiple layers of the first deep learning model, it can comprehensively consider the global information of the sentence. The vector d at the [CLS] position is taken, and then it is transmitted to the full connection layer in the quality grading layer, so as to map the high-dimensional vector to the number of quality grades.

[0081] Wherein, the quality score is divided into 5 levels, the output dimension mapped here is x, the weight matrix W of the full connection layer is a matrix of d*x, d is the number of feature vectors, x is the number of a plurality of preset score values, the value of the i-th row and the j-th column in W is the product of the i-th feature value in the feature vector and the j-th preset score value in the plurality of preset score values, the bias matrix b of the full connection layer is a matrix of 1 row and x columns, and thus the original score vector y of the labeled data is calculated in the full connection layer based on the bias matrix and the weight matrix, wherein the original score vector y = σ(W*d+b), the σ is an activation parameter, and the original score vector y includes x first score values, which correspond to the x preset score values one by one;

[0082] The quality grading layer also includes a normalization exponential function (Softmax) layer, which is used to calculate a plurality of score probabilities p from the plurality of preset score values and the original score vector y i , wherein the score probability p i represents the probability that the score value of the labeled data corresponds to the i-th score value in the plurality of preset score values, and the value of y i is the i-th first score value in the x first score values, i is the corresponding number value of the i-th preset score value in the plurality of preset score values, and e is a natural constant. Thus, the probability corresponding to each preset score value is calculated, for example, if the score is divided into 1-5 levels, 5 score probabilities are generated, and the 5 score probabilities correspond to 1-5 levels, i.e., the probability of scoring 1, the probability of scoring 2, and the probability of scoring 5.

[0083] Optionally, the linear weighting layer of the first deep learning model is used to perform weighted calculation on the plurality of score probabilities to obtain the score value corresponding to the labeled data, including:

[0084] The linear weighting layer of the first deep learning model is used to perform weighted calculation on the plurality of score probabilities to obtain the score value corresponding to the labeled data, including:

[0085] In this embodiment, the linear weighting layer of the first deep learning model is used to perform weighted calculation on the plurality of score probabilities to obtain the score value corresponding to the labeled data, i.e., the plurality of score probabilities corresponding to the quality grading and the corresponding quality score are linearly weighted and calculated to obtain the score value S, specifically In this embodiment, the quality grading layer and the linear weighting layer are introduced into the model structure, so that the model automatically learns the quality score distribution of the labeled content and the final score. According to the score, different processing mechanisms can be matched, which helps to improve the quality and efficiency of the entire labeling system, thereby realizing the automatic output of the labeled data scoring.

[0086] The original data and the labeled data in the data to be scored are input into the first deep learning model at the same time, the original data and the labeled data are scored by the first deep learning model, and a score value corresponding to the labeled data is obtained, so that in the scoring process of the labeled data, the correlation between the labeled data and the original data is considered, and the original data and the labeled data are input into the first deep learning model, so that the first deep learning model comprehensively understands the context, the first deep learning model captures the detail problems and context mismatch problems in the labeled data, and the accuracy of scoring the labeled data is improved.

[0087] Referring to Figure 3 , Figure 3 is a structural diagram of a scoring device for labeled data provided by an embodiment of the present application. As shown in Figure 3 , the scoring device for labeled data 300 comprises:

[0088] The acquisition module 310 is configured to acquire data to be scored, wherein the data to be scored comprises original data and labeled data, and the labeled data comprises labeled content of the original data.

[0089] The scoring module 320 is configured to input the data to be scored into a first deep learning model for scoring to obtain a score value corresponding to the labeled data, and the first deep learning model is configured to score the accuracy of the labeled data based on the original data.

[0090] Optionally, the scoring device for labeled data 300 further comprises:

[0091] The sample acquisition module is configured to acquire a plurality of training data sets, wherein each of the training data sets comprises a plurality of training samples, each of the training samples comprises sample original data and sample labeled data, the sample labeled data comprises labeled content of the sample original data, and the score values of the sample labeled data comprised in different training data sets are different.

[0092] The model training module is configured to train a second deep learning model based on the plurality of training data sets, and obtain the first deep learning model after the training is completed.

[0093] Optionally, the scoring module 320 comprises:

[0094] The processing submodule is configured to perform word segmentation processing on the original data based on the first deep learning model to obtain a first word segmentation sequence, and perform word segmentation processing on the labeled data to obtain a second word segmentation sequence.

[0095] The conversion submodule is configured to perform vector conversion on the first segmented sequence and the second segmented sequence based on the first deep learning model to obtain a plurality of feature vectors, the plurality of feature vectors being semantic representations of the first segmented sequence and the second segmented sequence, and one feature vector being configured to represent semantics of one segment in the first segmented sequence or one segment in the second segmented sequence.

[0096] The scoring submodule is configured to score the plurality of feature vectors based on the first deep learning model to obtain a score value corresponding to the annotation data.

[0097] Optionally, the first deep learning model comprises an input layer, an encoding layer, a quality grading layer, a linear weighting layer, and an output layer connected in sequence; the input layer is configured to perform segmentation processing on the original data to obtain the first segmented sequence, and perform segmentation processing on the annotation data to obtain the second segmented sequence; and the encoding layer is configured to perform vector conversion on the first segmented sequence and the second segmented sequence to obtain the plurality of feature vectors.

[0098] The scoring submodule comprises:

[0099] The scoring unit is configured to score the plurality of feature vectors based on the quality grading layer of the first deep learning model to obtain a plurality of scoring probabilities, the plurality of scoring probabilities corresponding to a plurality of preset score values one by one, and the scoring probability being configured to represent a probability that the score value of the annotation data is the corresponding preset score value.

[0100] The computing unit is configured to perform weighted calculation on the plurality of scoring probabilities based on the linear weighting layer of the first deep learning model to obtain the score value corresponding to the annotation data.

[0101] The output unit is configured to output the score value corresponding to the annotation data based on the output layer.

[0102] Optionally, the computing unit comprises:

[0103] The generating subunit is configured to generate a bias matrix b and a weight matrix W according to the plurality of feature vectors and the plurality of preset score values, wherein the weight matrix W is a d*x matrix, d is the number of the feature vectors, x is the number of the plurality of preset score values, the value of the i-th row and the j-th column in the W is the product of the i-th feature value in the feature vector and the j-th preset score value in the plurality of preset score values, and the bias matrix b is a 1-row x-column matrix.

[0104] The first calculation subunit is configured to calculate an original score vector y of the labeled data based on the bias matrix and the weight matrix, where the original score vector y = σ(W*d+b), the σ is an activation parameter, and the original score vector y includes x first score values, which are in one-to-one correspondence with x preset score values respectively.

[0105] The second calculation subunit is configured to calculate a plurality of score probabilities p based on the plurality of preset score values and the original score value y. i wherein, The p i is a probability that the score value of the labeled data is an i-th score value in the plurality of preset score values, the y i is an i-th first score value in the x first score values, the i is a corresponding number value of an i-th preset score value in the plurality of preset score values, and the e is a natural constant.

[0106] Optionally, the output unit includes:

[0107] The third calculation subunit is configured to perform weighted calculation on the plurality of score probabilities based on the plurality of preset score values and the original score value y, to obtain a score value S corresponding to the labeled data, where,

[0108] Optionally, the application further includes an updating module configured to input the to-be-scored data into the first deep learning model to train the first deep learning model in a case where the score value corresponding to the to-be-scored data satisfies a preset condition, and obtain a third deep learning model after the training is completed, the preset condition being that the score value corresponding to the to-be-scored data is greater than or equal to a preset threshold.

[0109] In the application, the original data and the labeled data in the to-be-scored data are input into the first deep learning model, the original data and the labeled data are scored by the first deep learning model, and a score value corresponding to the labeled data is obtained, so that the correlation between the labeled data and the original data is considered in the scoring process of the labeled data, the original data and the labeled data are input into the first deep learning model, the first deep learning model comprehensively understands the context, the first deep learning model captures the details in the labeled data and the context mismatching problem, and the accuracy of the scoring of the labeled data is improved.

[0110] The application also provides an electronic device. Please refer to Figure 4 The electronic device can include a processor 401, a memory 402, and a program 4021 stored in the memory 402 and executable on the processor 401.

[0111] The program 4021, when executed by the processor 401, can implement Figure 1 Any step in the corresponding method embodiment:

[0112] Obtain to-be-scored data, the to-be-scored data including original data and labeled data, the labeled data including labeled content of the original data;

[0113] Input the to-be-scored data into a first deep learning model for scoring to obtain a score value corresponding to the labeled data, the first deep learning model being configured to score the accuracy of the labeled data based on the original data.

[0114] Optionally, before the to-be-scored data is input into the first deep learning model for scoring to obtain the score value, the method further includes:

[0115] Obtain a plurality of training data sets, wherein each training data set includes a plurality of training samples, the training sample including sample original data and sample labeled data, the sample labeled data including labeled content of the sample original data, the plurality of sample labeled data included in the training data set having the same score value, and the sample labeled data included in different training data sets in the plurality of training data sets having different score values.

[0116] Train a second deep learning model based on the plurality of training data sets, and obtain the first deep learning model after the training is completed.

[0117] Optionally, the inputting the to-be-scored data into the first deep learning model for scoring to obtain the score value corresponding to the labeled data includes:

[0118] Perform word segmentation processing on the original data based on the first deep learning model to obtain a first word segmentation sequence, and perform word segmentation processing on the labeled data based on the first deep learning model to obtain a second word segmentation sequence;

[0119] Perform vector conversion on the first word segmentation sequence and the second word segmentation sequence based on the first deep learning model to obtain a plurality of feature vectors, the plurality of feature vectors being semantic representations of the first word segmentation sequence and the second word segmentation sequence, and one feature vector being configured to represent the semantic of one word in the first word segmentation sequence or one word in the second word segmentation sequence.

[0120] Score the plurality of feature vectors based on the first deep learning model to obtain the score value corresponding to the labeled data.

[0121] Optionally, the first deep learning model comprises an input layer, an encoding layer, a quality grading layer, a linear weighting layer and an output layer connected in sequence; the input layer is configured to perform word segmentation on the original data to obtain the first word segmentation sequence, and perform word segmentation on the labeled data to obtain the second word segmentation sequence; the encoding layer is configured to perform vector conversion on the first word segmentation sequence and the second word segmentation sequence to obtain the plurality of feature vectors;

[0122] The scoring of the plurality of feature vectors based on the first deep learning model to obtain the score value corresponding to the labeled data comprises:

[0123] The scoring of the plurality of feature vectors based on the quality grading layer of the first deep learning model to obtain a plurality of score probabilities, the plurality of score probabilities corresponding to a plurality of preset score values one by one, the score probability being used to represent a probability that the score value of the labeled data is the corresponding preset score value;

[0124] The weighting calculation of the plurality of score probabilities based on the linear weighting layer of the first deep learning model to obtain the score value corresponding to the labeled data;

[0125] The output of the score value corresponding to the labeled data based on the output layer.

[0126] Optionally, the scoring of the plurality of feature vectors based on the quality grading layer of the first deep learning model to obtain a plurality of score probabilities comprises:

[0127] Generating a bias matrix b and a weight matrix W according to the plurality of feature vectors and the plurality of preset score values, wherein the weight matrix W is a d*x matrix, d is the number of feature vectors, x is the number of preset score values, the value of the i-th row and the j-th column in the weight matrix W is the product of the i-th feature value in the feature vector and the j-th preset score value in the plurality of preset score values, and the bias matrix b is a 1-row x-column matrix;

[0128] Calculating the original score vector y of the labeled data based on the bias matrix and the weight matrix, wherein the original score vector y = σ(W*d+b), σ is an activation parameter, and the original score vector y comprises x first score values, which correspond to x preset score values one by one;

[0129] Calculating a plurality of score probabilities p based on the plurality of preset score values and the original score vector y i wherein, The p ito represent the score value of the labeled data as a probability corresponding to the i-th score value in the plurality of preset score values, the y i The value of x is the i-th first score value in the x first score values, the i is a corresponding number value of the i-th preset score value in the plurality of preset score values, and the e is a natural constant.

[0130] Optionally, the first deep learning model based on the linear weighting layer, the plurality of score probabilities are weighted and calculated to obtain the score value corresponding to the labeled data, comprising:

[0131] Based on the plurality of preset score values and the original score value y, the plurality of score probabilities are weighted and calculated to obtain the score value S corresponding to the labeled data, wherein,

[0132] Optionally, after the method further comprises:

[0133] In the case where the score value corresponding to the to-be-scored data satisfies a preset condition, the first deep learning model is trained by inputting the to-be-scored data into the first deep learning model, and a third deep learning model is obtained after the training is completed, and the preset condition is that the score value corresponding to the to-be-scored data is greater than or equal to a preset threshold.

[0134] The present application simultaneously inputs the original data and the labeled data in the to-be-scored data into the first deep learning model, scores the original data and the labeled data through the first deep learning model, and obtains the score value corresponding to the labeled data, so that the correlation between the labeled data and the original data is considered in the scoring process of the labeled data, and the original data and the labeled data are input into the first deep learning model, so that the first deep learning model comprehensively understands the context, captures the details in the labeled data and the context mismatch problem, and improves the accuracy of the labeled data scoring.

[0135] The embodiment of the present application also provides a computer readable storage medium, and the computer readable storage medium stores a computer program. The computer program is executed by a processor to realize each process of the above-mentioned scoring method for labeled data, and achieves the same technical effect. To avoid repetition, details are not repeated here. The computer readable storage medium includes a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0136] The embodiment of the present application further provides a computer program product stored in a storage medium, which is executed by at least one processor to implement various processes of the above-mentioned scoring method for labeled data, and can achieve the same technical effects. To avoid repetition, details are not described herein.

[0137] It should be noted that in this document, the terms "comprising", "including", or any other variant thereof are intended to cover non-exclusive inclusions, so that processes, methods, articles or devices including a series of elements not only include those elements, but also include other elements not explicitly listed, or further include elements inherent to such processes, methods, articles or devices. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0138] From the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment method can be realized by software plus a general hardware platform, of course, it can also be realized by hardware, but in many cases the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a plurality of instructions for making a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) execute the methods described in various embodiments of the present application.

[0139] The embodiments of the present application are described above in combination with the drawings, but the present application is not limited to the above-mentioned specific embodiments, the above-mentioned specific embodiments are only illustrative, not restrictive, and those skilled in the art can make many forms under the inspiration of the present application without departing from the scope of the present application and the protection scope of the claims.

Claims

1. A scoring method for labeled data, characterized in that: The method comprises: Acquire data to be scored, wherein the data to be scored includes original data and annotated data, wherein the annotated data includes annotation content of the original data; The data to be scored is input into a first deep learning model for scoring to obtain a scoring value corresponding to the labeled data. The first deep learning model is used to score the accuracy of the labeled data based on the original data.

2. The method according to claim 1, characterized in that Before inputting the data to be scored into the first deep learning model for scoring and obtaining a scoring value, the method further includes: Acquire multiple training data sets, wherein each of the training data sets includes multiple training samples, the training samples include sample original data and sample annotated data, the sample annotated data includes annotated content of the sample original data, and the multiple sample annotated data included in the training data sets have the same score value; and the sample annotated data included in different training data sets have different score values; The second deep learning model is trained based on the multiple training data sets, and after the training is completed, the first deep learning model is obtained.

3. The method according to claim 1, characterized in that Inputting the data to be scored into the first deep learning model for scoring to obtain a scoring value corresponding to the labeled data includes: Based on the first deep learning model, performing word segmentation processing on the original data to obtain a first word segmentation sequence, and performing word segmentation processing on the labeled data to obtain a second word segmentation sequence; Based on the first deep learning model, performing vector conversion on the first word segmentation sequence and the second word segmentation sequence to obtain multiple feature vectors, where the multiple feature vectors are semantic representations of the first word segmentation sequence and the second word segmentation sequence, and one feature vector is used to represent the semantics of a word in the first word segmentation sequence or a word in the second word segmentation sequence; Based on the first deep learning model, the multiple feature vectors are scored to obtain score values ​​corresponding to the labeled data.

4. The method according to claim 3, characterized in that The first deep learning model includes an input layer, an encoding layer, a quality grading layer, a linear weighting layer, and an output layer connected in sequence; the input layer is used to perform word segmentation processing on the original data to obtain the first word segmentation sequence, and to perform word segmentation processing on the labeled data to obtain the second word segmentation sequence; the encoding layer is used to perform vector conversion on the first word segmentation sequence and the second word segmentation sequence to obtain the multiple feature vectors; Scoring the plurality of feature vectors based on the first deep learning model to obtain score values ​​corresponding to the labeled data includes: Scoring the multiple feature vectors based on the quality grading layer of the first deep learning model to obtain multiple scoring probabilities, where the multiple scoring probabilities correspond one-to-one to multiple preset scoring values, and the scoring probabilities are used to represent the probability that the scoring value of the labeled data is the corresponding preset scoring value; Based on the linear weighted layer of the first deep learning model, performing weighted calculation on the multiple scoring probabilities to obtain a scoring value corresponding to the labeled data; Based on the output layer, the score value corresponding to the labeled data is output.

5. The method according to claim 4, characterized in that The quality grading layer based on the first deep learning model scores the multiple feature vectors to obtain multiple scoring probabilities, including: Generate a bias matrix b and a weight matrix W based on the multiple eigenvectors and the multiple preset scoring values, wherein the weight matrix W is a d*x matrix, d is the number of the eigenvectors, x is the number of the multiple preset scoring values, the value of the i-th row and j-th column in W is the product of the i-th eigenvalue in the eigenvector and the j-th preset scoring value in the multiple preset scoring values, and the bias matrix b is a 1-row x-column matrix; An original rating vector y of the labeled data is calculated based on the bias matrix and the weight matrix, wherein the original rating vector y=σ(W*d+b), σ is an activation parameter, and the original rating vector y includes x first rating values, and the x first rating values ​​respectively correspond one-to-one to the x preset rating values; Based on the multiple preset scoring values ​​and the original scoring value y, multiple scoring probabilities p are calculated. i ,in, The p i represents the probability that the score value of the labeled data corresponds to the i-th score value among the multiple preset score values, and the y i The value of is the i-th first scoring value among the x first scoring values, i is the corresponding number value of the i-th preset scoring value among the multiple preset scoring values, and e is a natural constant.

6. The method according to claim 5, characterized in that The linear weighted layer based on the first deep learning model performs weighted calculation on the multiple scoring probabilities to obtain a scoring value corresponding to the labeled data, including: Based on the multiple preset scoring values ​​and the original scoring value y, the multiple scoring probabilities are weighted to obtain the scoring value S corresponding to the labeled data, where:

7. The method according to any one of claims 1 to 6, characterized in that After inputting the data to be scored into the first deep learning model for scoring and obtaining the scoring value corresponding to the labeled data, the method further includes: When the scoring value corresponding to the data to be scored meets the preset conditions, the data to be scored is input into the first deep learning model to train the first deep learning model. After the training is completed, a third deep learning model is obtained. The preset condition is that the scoring value corresponding to the data to be scored is greater than or equal to a preset threshold.

8. A scoring device for labeled data, characterized in that: The device comprises: An acquisition module is used to acquire data to be scored, wherein the data to be scored includes original data and annotated data, wherein the annotated data includes annotation content of the original data; A scoring module is used to input the data to be scored into a first deep learning model for scoring to obtain a scoring value corresponding to the labeled data, and the first deep learning model is used to score the accuracy of the labeled data based on the original data.

9. An electronic device, characterized in that: include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which implements the steps of the method according to any one of claims 1 to 7 when executed by a processor.

11. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 7.

Images