Training data evaluation method, device, electronic device and storage medium

Abstract

The present application proposes a training data evaluation method, device, electronic device and storage medium, which can calculate the similarity between pre-collected question data and training data. If there is at least one target training data in the training data whose similarity with the question data is greater than a set similarity threshold, it is determined that the question data has been successfully answered. If the question answering rate of all question data is less than the set question answering rate threshold, the first supplementary data indication information of the training data is generated and output based on all question data that failed to be answered, wherein the question answering rate represents the proportion of the number of question data that have been successfully answered in all question data. This allows the training data to be supplemented according to the first supplementary data indication information, so that the integrity of the training data meets the training requirements of the speech model, thereby improving the model training effect and training efficiency.

Description

Technical Field

[0001] The present application relates to the technical field of language model training, and in particular to a method, device, electronic device, and storage medium for evaluating training data. Background Art

[0002] Language models can handle a variety of natural language tasks, such as question-answering and conversational conversation. Language model technology is a key path to artificial intelligence. To be able to perform a variety of natural language tasks, language models require sufficient, comprehensive, and complete training data as support. However, training data may be incomplete, and the inability to provide effective training data supplementation recommendations can affect the effectiveness of language model training. Summary of the Invention

[0003] In view of this, the present application proposes a training data evaluation method, device, electronic device and storage medium, which can provide effective training data supplementation suggestions when the training data is incomplete.

[0004] The technical solutions proposed in this application are as follows:

[0005] In a first aspect, the present application provides a method for evaluating training data, comprising:

[0006] Calculate the similarity between the pre-collected question data and the training data;

[0007] If there is at least one target training data in the training data, and the similarity between the target training data and the question data is greater than a set similarity threshold, then it is determined that the question data is successfully answered;

[0008] If the question answering rate of all question data is less than the set question answering rate threshold, the first supplementary data indication information of the training data is generated and output based on all question data that failed to be answered; the question answering rate represents the proportion of the number of question data that were successfully answered in all question data.

[0009] In a second aspect, the present application provides a training data evaluation device, comprising:

[0010] A calculation module, used to calculate the similarity between pre-collected question data and training data;

[0011] A first determination module is configured to determine that the question data is successfully answered if there is at least one target training data in the training data and the similarity between the target training data and the question data is greater than a set similarity threshold;

[0012] The second determination module is used to generate and output the first supplementary data indication information of the training data based on all the question data that failed to be answered if the question answering rate of all the question data is less than the set question answering rate threshold; the question answering rate represents the proportion of the number of question data that have been successfully answered in all the question data.

[0013] In a third aspect, the present application provides an electronic device, comprising:

[0014] memory and processor;

[0015] Wherein, the memory is used to store programs;

[0016] The processor is configured to implement any one of the above methods by running the program in the memory.

[0017] In a fourth aspect, the present application provides a storage medium having a computer program stored thereon, and when the computer program is executed by a processor, any of the methods described above is implemented.

[0018] The training data evaluation method proposed in this application calculates the similarity between pre-collected question data and training data. If there is at least one target training data in the training data whose similarity with the question data is greater than a set similarity threshold, it is determined that the question data has been successfully answered. If the question answering rate of all question data is less than the set question answering rate threshold, the first supplementary data indication information of the training data is generated and output based on all question data that failed to be answered, wherein the question answering rate represents the proportion of the number of question data that were successfully answered in all question data. This allows the training data to be supplemented according to the first supplementary data indication information, so that the integrity of the training data meets the training requirements of the speech model, thereby improving the model training effect and training efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0020] Figure 1 1 is a flow chart of a method for evaluating training data provided in an embodiment of the present application;

[0021] Figure 2 1 is a flow chart of another method for evaluating training data provided in an embodiment of the present application;

[0022] Figure 3 This is a flow chart of calculating the similarity between problem data and training data provided by an embodiment of the present application;

[0023] Figure 4 is a schematic diagram of another process for calculating the similarity between problem data and training data provided by an embodiment of the present application;

[0024] Figure 5 Schematic diagram of a training data evaluation device provided in an embodiment of the present application;

[0025] Figure 6 This is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] The technical solution of the embodiment of the present application is applicable to application scenarios for evaluating training data. By adopting the technical solution of the embodiment of the present application, when it is determined that the training data is incomplete, supplementary suggestions for the training data can be given to ensure that the integrity of the training data meets the training requirements of the speech model.

[0027] Language models, such as large language models (LLMs), possess rich functionality and can handle a variety of natural language tasks, such as question-answering and conversational conversation. To be able to perform these tasks, language models require sufficient, comprehensive, and complete training data. However, the integrity of training data cannot be judged by human standards. Even if humans believe the training data is complete, after inputting it into a language model for training, the model may still be unable to effectively answer questions due to missing knowledge.

[0028] Based on this, the present application proposes a training data evaluation method, device, electronic device and storage medium. This technical solution can calculate the similarity between pre-collected problem data and training data, and determine the supplementary data indication information of the training data through the similarity, thereby solving the problem in the existing technology that it is impossible to provide effective training data supplementation suggestions, affecting the training effect of the language model.

[0029] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0030] The present application embodiment provides a method for evaluating training data, which can be executed by an electronic device. The electronic device can be any device with data and instruction processing functions, such as a computer, an intelligent terminal, a server, etc. Figure 1 As shown, the method includes:

[0031] S101: Calculate the similarity between pre-collected question data and training data.

[0032] The pre-collected question data refers to any one of multiple pieces of question data collected in advance from a predetermined URL. The predetermined URL can be any public and decrypted URL, and this embodiment does not limit this. Training data refers to data used to train a language model, and is generally unstructured data in the form of plain text.

[0033] In an embodiment of the present application, the similarity between the question data and the training data can be calculated. Specifically, based on the actual setting, the training data for which similarity needs to be calculated can be determined, and the similarity between the question data and each piece of training data for which similarity needs to be calculated can be calculated. The training data for which similarity needs to be calculated can be all of the training data or a set portion of the training data, which is not limited in this embodiment.

[0034] In some embodiments, when calculating the similarity between the problem data and each piece of training data for which similarity needs to be calculated, the problem data can be converted into a problem data vector, the training data can be converted into a training data vector, the dot product similarity between the problem data vector and the training data vector is calculated, and the dot product similarity between the problem data vector and the training data vector is used as the similarity between the corresponding problem data and the training data.

[0035] In some other embodiments, when calculating the similarity between the problem data and each piece of training data for which similarity needs to be calculated, the problem data can also be converted into a problem data vector, the training data can be converted into a training data vector, and then the cosine similarity between the problem data vector and the training data vector can be calculated, and the cosine similarity between the problem data vector and the training data vector can be used as the similarity between the corresponding problem data and the training data.

[0036] S102: Check whether there is at least one target training data in the training data, and the similarity between the target training data and the problem data is greater than the set similarity threshold; if there is at least one target training data in the training data, execute step S103; if there is no target training data in the training data, execute step S104.

[0037] The target training data refers to all the training data for which similarity calculation is required, and the data whose similarity with the problem data is greater than the set similarity threshold. It should be noted that the similarity threshold can be set according to actual conditions. For example, the similarity threshold can be set to 80% or 90%, which is not limited in this embodiment.

[0038] Specifically, the similarity value is compared with a set similarity threshold. The similarity value refers to the similarity between the pre-collected question data and each training data for which similarity is to be calculated. Training data whose similarity to the question data exceeds the set similarity threshold is marked as target training data.

[0039] If it is detected that there is at least one target training data in the training data whose similarity with the above-mentioned problem data is greater than the set similarity threshold, step S103 is executed; if it is detected that there is no target training data in the training data whose similarity with the above-mentioned problem data is greater than the set similarity threshold, step S104 is executed.

[0040] S103: Determine whether the above question data is successfully answered.

[0041] If it is detected that there is at least one target training data in the training data whose similarity with the above question data is greater than the set similarity threshold, it is determined that the answer corresponding to the above question data exists in the training data, and the above question data is successfully solved.

[0042] S104: Determine that the above problem data has failed to be solved.

[0043] If it is detected that there is no target training data in the training data whose similarity with the above question data is greater than the set similarity threshold, it is determined that the answer corresponding to the above question data does not exist in the training data, and the answer to the above question data fails.

[0044] S105. Check whether the question answering rate of all question data is less than the set question answering rate threshold; if the question answering rate of all question data is greater than or equal to the set question answering rate threshold, execute step S106; if the question answering rate of all question data is less than the set question answering rate threshold, execute step S107.

[0045] All the above-mentioned question data refers to all the question data for which similarity needs to be calculated. Specifically, the question data for which similarity needs to be calculated can be determined based on the actual setting, and then the similarity between the training data and each question data for which similarity needs to be calculated can be calculated according to the above steps. Among them, the question data for which similarity needs to be calculated can be all the question data or a set part of the question data, which is not limited in this embodiment.

[0046] The above question answering rate represents the ratio of successfully answered question data to all question data. The formula is as follows:

[0047]

[0048] In some embodiments, a question answering rate threshold may be preset. When the question answering rate of all question data is greater than or equal to the question answering rate threshold, step S105 may be executed; when the question answering rate of all question data is less than the question answering rate threshold, step S106 may be executed.

[0049] The question answering rate threshold can be set according to actual conditions, for example, it can be set to 90% or 95%, which is not limited in this embodiment.

[0050] S106: Determine whether the training data is complete training data.

[0051] All question data comes from various public and de-identified websites, covering a wide range of question types and fields. In this case, if the training data's answer rate for all question data exceeds the question answer rate threshold, it means that the majority of questions in the training data can be answered within the training data, that the training data contains knowledge data from most categories and fields on the internet, and that the training data is highly complete and meets the requirements. Therefore, the training data can be considered complete.

[0052] By using such data to train the language model, the language model can learn data from most categories and fields on the Internet, and the probability of knowledge gaps is low.

[0053] S107: Generate and output first supplementary data indication information of training data based on all question data that fail to be answered.

[0054] When the question answering rate of all question data is less than the question answering rate threshold, it means that a large portion of the question data cannot be answered in the training data, and the training data does not contain knowledge data in the field where this type of question is located. The integrity of the training data is low and data supplementation is required.

[0055] Furthermore, first supplementary data indication information for the training data can be generated and output based on all unanswered question data. Specifically, the knowledge domain to which all unanswered question data belongs can be analyzed, and then the knowledge domain to which all unanswered question data belongs can be output as the first supplementary data indication information, so that relevant staff can supplement the data in that knowledge domain into the training data to make the training data more complete.

[0056] For example, the knowledge domain to which the unanswered question data belongs can be determined based on a pre-trained question classification model. Specifically, all unanswered question data can be input into the trained question classification model to obtain the knowledge domain to which each unanswered question data belongs as output by the question classification model.

[0057] A large amount of sample question data can be obtained as training samples for the question classification model, and the knowledge domain to which the sample question data belongs can be marked as training labels to train the question classification model. The training process is as follows: the training samples are input into the question classification model to obtain the prediction results output by the question classification model, and the loss value of the question classification model is determined by comparing the prediction results output by the question classification model with the training labels. The parameters of the question classification model are adjusted with the goal of reducing the loss value of the question classification model, and then the above training process is repeated until the loss value of the question classification model is less than the set value and the training is complete. The above set value can be set according to the actual situation and is not limited in this embodiment.

[0058] It should be noted that the question classification model can adopt a deep learning classification model as a basic model, such as a multilayer perceptron (MLP), which is not limited in this embodiment.

[0059] In the above embodiment, the similarity between the pre-collected question data and the training data can be calculated. If there is at least one target training data in the training data whose similarity to the question data is greater than a set similarity threshold, it is determined that the question data has been successfully answered. If the question answering rate of all question data is less than the set question answering rate threshold, the first supplementary data indication information of the training data is generated and output based on all the question data that failed to be answered, wherein the question answering rate represents the proportion of the number of question data that were successfully answered in all the question data. This allows the training data to be supplemented according to the first supplementary data indication information, so that the integrity of the training data meets the training requirements of the speech model.

[0060] Furthermore, currently, existing training data is input into the language model for training, and the trained language model is then applied to real-world scenarios to solve real-world problems. This actual application test determines whether the language model has been adequately and reasonably trained, and further determines whether the training data used to train the language model is complete. If the training data is determined to be incomplete, new training data is constructed and the model is trained again. This current approach involves passively verifying the training data after the language model is trained, which may result in multiple repetitions of the language model training, resulting in a long model training cycle and low efficiency.

[0061] The technical solution provided by the embodiments of this application assesses the integrity of the training data before model training. If the training data is incomplete, relevant staff can be prompted to supplement the missing data. Once the training data integrity meets the training requirements of the speech model, the training data can be used to train the language model. This setup eliminates the need for repeated language model training, thereby improving model training effectiveness and efficiency.

[0062] As an optional implementation, such as Figure 2 As shown, in another embodiment of the present application, the steps of the above embodiment for calculating the similarity between the pre-collected question data and the training data may further include the following steps:

[0063] S201: Calculate the similarity between the problem data and the training data in a specific dimension.

[0064] It should be noted that all question data includes question data of multiple dimensions, and the dimensions are determined according to the domain of the question. The domain of the question data refers to the domain of knowledge related to the question data. That is, in the embodiments of the present application, all question data is divided into different dimensions according to the different domains of the question data.

[0065] The division of fields can be determined based on the application scenario of the language model, which is not limited in this embodiment. For example, if the language model is applied to a medical question-and-answer scenario, the question data may include fields in the medical scenario, such as disease fields, medication fields, surgery fields, and recipe fields. If the question data is "What are the effects of ibuprofen?", it can be determined that the knowledge involved in the question data is medication knowledge, and it is determined that the question data belongs to the dimension corresponding to the medication field.

[0066] The specific dimensions mentioned above refer to the dimensions that require data integrity assessment. The dimensions that require data integrity assessment can be all dimensions involved in all problematic data, or a pre-set subset of all dimensions involved in all problematic data, which is not limited in this embodiment.

[0067] In an embodiment of the present application, all question data are first divided into different dimensions, and then the similarity between the question data of a specific dimension and the training data is calculated.

[0068] Among them, the method of calculating the similarity between the problem data of a specific dimension and the training data is the same as the method of calculating the similarity between the problem data and the training data in the above embodiment. Technical personnel in this field can refer to the records of the above embodiment to calculate the similarity between the problem data of a specific dimension and the training data, which will not be repeated here.

[0069] Further, such as Figure 2As shown, after the above step S201, the following steps may also be included:

[0070] S202: Check whether there is at least one target training data in the training data, and the similarity between the target training data and the problem data is greater than the set similarity threshold; if there is at least one target training data in the training data, execute step S203; if there is no target training data in the training data, execute step S204.

[0071] S203: Determine whether the above question data is successfully answered.

[0072] S204: Determine that the above problem data has failed to be solved.

[0073] Among them, the specific execution method of steps S202 to S204 is the same as the specific execution method of steps S102 to S104 in the above embodiment. Those skilled in the art can refer to the description of the above embodiment, and no further details are given here.

[0074] Furthermore, the steps of the above embodiment determine the integrity assessment result of the training data based on the question answering rate of all question data, which specifically includes the following steps:

[0075] S205. Check whether the question answering rate of all question data of a specific dimension is less than the set question answering rate threshold of the specific dimension; if the question answering rate of all question data of a specific dimension is greater than or equal to the question answering rate threshold of the specific dimension, execute step S206; if the question answering rate of all question data of a specific dimension is less than the question answering rate threshold of the specific dimension, execute step S207.

[0076] All the question data in the above-mentioned specific dimension refers to all the question data under the specific dimension for which similarity needs to be calculated. Specifically, the question data for which similarity needs to be calculated under the specific dimension can be determined according to the actual setting situation, and then the similarity between the training data and each question data for which similarity needs to be calculated under the specific dimension can be calculated according to the above steps. Among them, the question data for which similarity needs to be calculated under the specific dimension can be all the question data under the specific dimension, or it can be part of the question data set under the specific dimension, which is not limited in this embodiment.

[0077] The question answering rate of question data of a specific dimension indicates the proportion of successfully answered question data among all question data of the specific dimension.

[0078] A specific dimension question answering rate threshold corresponding to the specific dimension can be set in advance. When the question answering rate of all question data of the specific dimension is greater than or equal to the specific dimension question answering rate threshold, step S206 can be executed; when the question answering rate of all question data of the specific dimension is less than the specific dimension question answering rate threshold, step S207 can be executed.

[0079] The threshold value of the answer rate of the above-mentioned specific dimension questions can be set according to actual conditions, for example, it can be set to 90% or 95%, which is not limited in this embodiment.

[0080] S206: Determine that the training data belonging to the specific dimension is complete training data.

[0081] If the question answering rate of the training data for all question data of a specific dimension is greater than the specific dimension question answering rate threshold corresponding to the specific dimension, it means that most of the questions in the question data of the specific dimension can be answered in the training data, the training data contains various types of data of the specific dimension, and the training data of the specific dimension has high integrity and meets the requirements. Then, the training data belonging to the above-mentioned specific dimension can be determined as complete training data.

[0082] S207: Generate and output first supplementary data indication information of training data based on all question data of a specific dimension that fail to be answered.

[0083] When the question answering rate of all question data in a specific dimension is less than the specific dimension question answering rate threshold corresponding to the specific dimension, it means that among all the question data in the specific dimension, a large portion of the question data cannot be answered in the training data, and the training data does not contain knowledge data in the field where this type of question is located. The integrity of the training data is low and data supplementation is required.

[0084] Furthermore, first supplementary data indication information for training data can be generated and output based on all unanswered question data in all question data of a specific dimension. Specifically, the knowledge domain to which all unanswered question data in all question data of a specific dimension belong can be analyzed, and then the knowledge domain to which all unanswered question data in all question data of the specific dimension belong can be output as the first supplementary data indication information, so that relevant staff can supplement the data in this knowledge domain to the training data, making the training data more complete and achieving the purpose of ensuring that the integrity of the training data meets the training requirements of the speech model.

[0085] In the above embodiments, it is possible to calculate whether the training data of each dimension is complete, and then output corresponding supplementary data indication information on the premise that the training data of a specific dimension is incomplete, so that relevant staff can perform targeted data supplementation according to the corresponding supplementary data indication information to make the training data more complete.

[0086] As an optional implementation, another embodiment of the present application discloses that before calculating the similarity between the question data of a specific dimension and the training data in the steps of the above embodiment, the following steps may also be included:

[0087] Collect problem data and divide all problem data into problem data of multiple dimensions.

[0088] Specifically, in this step, problem data can be collected from the set website, and then all problem data can be divided into problem data of multiple dimensions according to the field where the problem data is located, so as to determine the data integrity assessment results of the training data corresponding to each dimension. When the training data of a specific dimension is incomplete, the first supplementary data indication information of the training data is generated and output, so that relevant staff can supplement the training data under this dimension in a targeted manner.

[0089] As an optional implementation, another embodiment of the present application discloses that the steps of collecting problem data in the above embodiment may specifically include the following steps:

[0090] Initial question data is obtained from a setting web page; and knowledge-related question data is filtered out from the initial question data.

[0091] The set webpage is the webpage corresponding to the set URL. The set URL can be any public and anonymized URL, and this embodiment does not limit this. Specifically, the initial question data can be collected from webpages corresponding to various public and anonymized URLs. When collecting the initial question data, the quality of the initial question data can be ignored, and the focus is on ensuring that the collected initial question data covers multiple fields.

[0092] After obtaining the initial question data from the setup webpage, you can filter the initial question data to obtain question data. Specifically, when filtering the initial question data, retain knowledge-related questions and remove reasoning, calculation, analysis, and other questions to obtain all question data. Furthermore, in some application scenarios, the language model may only require training data in a specific field. In this case, when filtering the initial question data, it is also necessary to retain the question data corresponding to that field.

[0093] In some embodiments, the purpose of screening the initial question data can be achieved by training a screening model. Specifically, the initial question data obtained from the setting webpage can be input into the trained screening model so that the screening model can screen out knowledge-related question data from the initial question data.

[0094] A large amount of sample initial question data can be obtained as training samples for the screening model, and the question data that needs to be retained in the sample initial question data can be marked as training labels. During training, the training samples are input into the screening model to obtain the prediction results output by the screening model. By comparing the prediction results output by the screening model and the training labels, the loss value of the screening model is determined. The parameters of the screening model are adjusted with the goal of reducing the loss value of the screening model, and then the above training process is repeated until the loss value of the screening model is less than the set value and the training is complete. The above set value can be set according to actual conditions and is not limited in this embodiment.

[0095] Among them, the screening model can use a deep learning model as a basic model, such as a Transformer model, which is not limited in this embodiment.

[0096] In the above embodiment, after obtaining the initial question data from the setting webpage, the initial question data can be screened to obtain valuable question data, thereby reducing the amount of calculation in the similarity calculation process and improving the evaluation speed of the training data integrity.

[0097] As an optional implementation, another embodiment of the present application discloses that the steps of the above embodiment divide all question data into question data of multiple dimensions, which may specifically include the following steps:

[0098] All question data are input into a pre-trained dimensional classification model so that the classification model divides all question data into question data of multiple preset dimensions; and / or, all question data are clustered to determine question data belonging to the same category as question data of the same dimension.

[0099] In some embodiments, if it is clear that the evaluation dimensions of the training data integrity are multiple preset dimensions, all problem data can be input into the pre-trained dimension classification model with the help of a pre-trained dimension classification model. The dimension classification model can divide all problem data into problem data of multiple preset dimensions, so as to facilitate the evaluation of the data integrity of the training data under the above preset dimensions.

[0100] A large amount of sample question data can be obtained as training samples for the dimensional classification model, and the dimensions to which the sample question data belong can be marked as training labels, wherein the sample question data includes question data in the above-mentioned multiple preset dimensions, so that the dimensional classification model can recognize the above-mentioned multiple preset dimensions and the question data in the above-mentioned multiple preset dimensions after training is completed.

[0101] The training process of the dimensional classification model is the same as the training process of the screening model in the above embodiment. The specific training process of the dimensional classification model can be referred to by those skilled in the art in the art in the above embodiment, and will not be described in detail here. It should be noted that the dimensional classification model can use a deep learning classification model as the basic model, such as a multilayer perceptron (MLP), which is not limited in this embodiment.

[0102] In some embodiments, if the evaluation dimension of the training data integrity cannot be clearly defined, an unsupervised clustering algorithm can be used to cluster all problem data, such as using agglomerative hierarchical clustering, K-Means clustering, etc., to determine problem data belonging to the same category as problem data of the same dimension.

[0103] In some embodiments, if it is clear that some evaluation dimensions of the training data integrity include multiple preset dimensions, and it is impossible to clearly define all evaluation dimensions of the training data integrity, all problem data can be divided into two parts, one part of which uses a classification model to divide the problem data into problem data of multiple preset dimensions, and the other part is clustered to determine problem data belonging to the same category as problem data of the same dimension, thereby obtaining problem data of multiple dimensions.

[0104] In the above embodiments, all problem data can be divided into multiple dimensions to facilitate evaluation of the data integrity of the training data in each dimension.

[0105] As an optional implementation, another embodiment of the present application discloses that the method of the above embodiment may further include the following steps:

[0106] Detect the number of target training data corresponding to the problem data in the training data; if the number of target training data is greater than a set first target training data number threshold, determine that the training data is valid training data for the problem data; if the number of target training data is less than a set second target training data number threshold, determine that the training data is invalid training data for the problem data.

[0107] Specifically, the number of target training data corresponding to any problem data in the training data may be detected. If the number of target training data is greater than a set first target training data number threshold, the training data is determined to be valid training data for the problem data.

[0108] For all question data that need to calculate similarity, if the number of target training data corresponding to all question data that need to calculate similarity is greater than the set first target training data number threshold, then the training data can be considered as valid training data for all question data. The training data is of high quality and can be used to train the language model.

[0109] On the contrary, if the amount of the target training data is less than the set second target training data amount threshold, the training data is determined to be invalid training data for the problem data.

[0110] For all question data that need to calculate similarity, if there is at least one question data that needs to calculate similarity, and the number of corresponding target training data is less than the set second target training data number threshold, then it can be considered that the training data is not valid training data for all question data, and the training data is of low quality. New training data needs to be supplemented or replaced to improve the overall quality of the training data, thereby improving the effect and training efficiency of the language model.

[0111] The need to supplement or replace new training data refers to the training data corresponding to the problem data whose amount of target training data is less than a set second target training data amount threshold.

[0112] It should be noted that the first target training data quantity threshold and the second target training data quantity threshold can be set according to actual conditions, and the first target training data quantity threshold and the second target training data quantity threshold can be the same value or different values, for example, both are 500 or 600, etc., which is not limited in this embodiment.

[0113] In some other embodiments, the number of target training data corresponding to all question data in a specific dimension can be calculated. If the number of target training data corresponding to all question data in a specific dimension is greater than a set first target training data number threshold, then the training data quality of the dimension can be considered to be high; if the number of target training data corresponding to all question data in a specific dimension is less than a set second target training data number threshold, then the training data quality of the dimension can be considered to be low, and a supplementary reminder can be output to remind relevant staff to supplement the training data of the dimension to improve the overall quality of the training data, thereby improving the effectiveness and training efficiency of the language model.

[0114] In the above embodiment, the dot product similarity between the problem data vector and the training data vector is calculated, and the dot product similarity between the problem data vector and the training data vector is used as the similarity between the corresponding problem data and the training data. Although this method has certain advantages, the problem data vector and the training data vector can be calculated offline and used repeatedly, and the calculation efficiency is very high, it also has very obvious disadvantages. This method separates the problem data and the training data, ignores the intrinsic relationship between the two, and will result in a significantly low value when calculating the similarity.

[0115] In order to solve the above technical problems, as an optional implementation method, Figure 3 As shown, in another embodiment of the present application, the steps of the above embodiment for calculating the similarity between the pre-collected question data and the training data may specifically include the following steps:

[0116] S301: Calculate the dot product similarity between the first question data and each piece of training data.

[0117] The first question data mentioned above refers to each piece of question data for which similarity needs to be calculated in the above embodiments. The training data mentioned above refers to each piece of training data for which similarity needs to be calculated. In the embodiments of the present application, when calculating the similarity between the first question data and the training data, the dot product similarity between the first question data and each piece of training data is first calculated.

[0118] Specifically, the first question data can be converted into a first question data vector, the training data can be converted into a training data vector, and the dot product similarity between the first question data vector and the training data vector can be calculated as the corresponding dot product similarity between the first question data and the training data.

[0119] In some embodiments, all question data are divided into multiple dimensions. When determining whether the training data is complete in a specific dimension, the dot product similarity between the first question data in the specific dimension and each piece of training data can be calculated.

[0120] S302: Select K pieces of training data with the highest dot product similarity to the first problem data from the training data to form a training data set.

[0121] The above K is a positive integer. The value of K can be set according to actual conditions and is not limited in this embodiment.

[0122] In this step, K pieces of training data with the highest dot product similarity to the first problem data are selected from the training data for which similarity calculation is performed, to form a training data set.

[0123] In some embodiments, the training data for which similarity calculation is performed may be arranged in descending order of similarity to form a training data sequence, and then the first K pieces of training data in the training data sequence are selected to form a training data set. In still other embodiments, the training data for which similarity calculation is performed may be arranged in ascending order of similarity to form a training data sequence, and then the last K pieces of training data in the training data sequence are selected to form a training data set.

[0124] S303 , concatenating each piece of training data in the training data set with the first question data to obtain concatenated data, and calculating the dot product similarity between each piece of concatenated data and the first question data.

[0125] Since this step selects the K training data pieces with the highest dot product similarity, the target training data whose similarity to the first question data exceeds the set similarity threshold should be among these K training data pieces. To reduce computational complexity and improve computational speed, each piece of training data in the training data set can be concatenated with the first question data to generate concatenated data. The dot product similarity between each piece of concatenated data and the first question data is then calculated as the similarity between the training data contained in the concatenated data and the first question data.

[0126] Each piece of training data in the training data set is concatenated with the first question data to obtain concatenated data. For example, if the first question data is X1 and the training data are X2, X3, and X4 respectively, the concatenated data obtained are X1X2, X1X3, and X1X4.

[0127] Next, calculate the dot product similarity between each concatenated data point and the first question data point. As in the example above, if the first question data point is X1, and the concatenated data points are X1X2, X1X3, and X1X4, calculate the dot product similarity between X1 and X1X2, X1 and X1X3, and X1 and X1X4, respectively.

[0128] When calculating the dot product similarity between each piece of spliced data and the first question data, each piece of spliced data can be uniformly encoded to obtain a spliced data vector. The dot product similarity between the spliced data vector and the first question data vector is calculated as the dot product similarity between the corresponding spliced data and the first question data.

[0129] like Figure 4 As shown, that is, the embodiment of the present application mainly includes the following two stages when calculating the similarity between the first question data and the training data:

[0130] In the first stage, the dot product similarity between the first problem data vector and the training data vector is calculated, and then the K training data with the highest dot product similarity are selected to form the training data set.

[0131] In the second stage, each piece of training data in the training data set is spliced with the first question data to obtain spliced data, and the dot product similarity between each spliced data vector and the first question data vector is calculated.

[0132] It should be noted that, in the above embodiments, when converting the spliced data, the first question data and the training data into corresponding vectors, an encoder can be used, that is, the spliced data, the first question data or the training data can be input into the encoder, and the encoder encodes the corresponding data and outputs the corresponding vector.

[0133] In the above embodiment, calculating the similarity between the problem data and the training data through the above two stages can not only overcome the problem that the problem data and the training data are separated, resulting in a significantly low value when calculating the similarity, but also ensure a faster calculation speed.

[0134] As an optional implementation, another embodiment of the present application discloses that before calculating the dot product similarity between the first question data and each training data in the steps of the above embodiment, the following steps may be further included:

[0135] All question data are divided into question data of multiple dimensions; and first question data are obtained by sampling from all question data of a specific dimension.

[0136] In this step, all question data can be divided into question data of multiple dimensions. For the specific method of dividing all question data into question data of multiple dimensions, those skilled in the art can refer to the description of the above embodiment, which will not be described in detail here.

[0137] In some embodiments, in the context of large-scale language model training, the volume of all question data and training data is in the billions. A full-data analysis approach is computationally intensive and inefficient for evaluating training data. Therefore, in this embodiment, a sampling question approach is used for evaluation and analysis.

[0138] Specifically, the first question data can be sampled from all question data of a specific dimension, so that when calculating the similarity between the question data of the specific dimension and the training data, the similarity between the sampled first question data and the training data is calculated, thereby reducing the amount of calculation. In some embodiments, the question data of each specific dimension can be sampled in the same proportion.

[0139] However, the sampling method is quite random. To reduce the bias in the evaluation results, we can sample all the question data for a specific dimension N times, calculate the question answering rate for each sampling, and take the average of the N sampling question answering rates as the question answering rate for all the question data for that specific dimension. Where N is a positive integer.

[0140] In the above embodiments, the amount of data calculation can be reduced and the evaluation speed of training data can be increased by sampling the problem data.

[0141] As an optional implementation, another embodiment of the present application discloses that the training data evaluation method of the above embodiment may further include the following steps:

[0142] The ratio between the first data and the second data is calculated. If the difference between the ratio and the question answering rate is greater than a set difference threshold, the second supplementary data indication information of the training data is generated and output based on all the second question data.

[0143] Among them, the first data is the sum of the number of first training data corresponding to each problem data, and the second data is the product of the number of pre-collected problem data and K; the first training data corresponding to the problem data is the training data among the K training data with the greatest similarity to the problem data, and the similarity with the problem data is greater than the set similarity threshold, and K is a positive integer.

[0144] If the ratio between the first data and the second data is defined as richness, then the calculation formula of richness is as follows:

[0145]

[0146] In some embodiments, the dot product similarity between the problem data and each piece of training data can be calculated according to the steps described in the above embodiments. K pieces of training data with the highest dot product similarity to the problem data are selected from the training data to form a training data set. Then, each piece of training data in the training data set is concatenated with the problem data to obtain concatenated data. The dot product similarity between each concatenated data piece and the problem data is calculated as the similarity between the training data contained in each concatenated data piece and the problem data. For any piece of problem data, the training data among the K pieces of training data whose similarity to the problem data is greater than a set similarity threshold is used as the first training data corresponding to the problem data.

[0147] The more first training data corresponding to each question data, the more content in the training data related to the question data, the richer the content of the training data, and the higher the corresponding richness. When all of the K training data are first training data, the numerator and denominator of the richness calculation formula are equal, and the richness is 100%. When none of the K training data are first training data, the numerator of the richness is 0, and the richness value is 0. In other words, the richness is a number between 0 and 1.

[0148] In this embodiment, the richness is less than or equal to the question answering rate. When the richness is closer to the question answering rate, it means that the richness of the collected training data is higher, and vice versa.

[0149] If the difference between the ratio of the first data and the second data and the question-answering rate is greater than a set difference threshold, it indicates that the difference between the richness and the question-answering rate is large, the richness of the training data is low, and additional training data is needed. The set difference threshold can be set according to actual circumstances, for example, to 30% or 20%, and this embodiment is not limited thereto. Specifically, second supplementary data indication information for the training data can be generated and output based on all the second question data.

[0150] The second question data refers to question data for which the corresponding first training data is less than the set first training data threshold. The first training data threshold is the minimum amount of first training data allowed and can be set according to actual conditions, for example, it can be set to 200 or 500, etc., and this embodiment does not limit this. In other words, the second question data is data with fewer answers in the training data. This is because the training data corresponding to the second question data is not rich enough, resulting in the overall lack of richness of the training data. Therefore, the knowledge domain to which all second question data belongs can be output as second supplementary data indication information to enable relevant staff to supplement the data of this knowledge domain to the training data, making the training data richer.

[0151] The knowledge domain to which the second question data belongs may be determined based on the question classification model of the above embodiment.

[0152] In some embodiments, the richness of the training data corresponding to a specific dimension may be calculated so as to remind relevant staff to supplement the training data corresponding to the specific dimension when the richness of the training data corresponding to the specific dimension is low.

[0153] Specifically, when calculating the richness of the training data corresponding to a specific dimension, the first data is the sum of the number of first training data corresponding to each problem data of the specific dimension, and the second data is the product of the number of problem data of the specific dimension collected in advance and K. The first training data corresponding to the problem data of the specific dimension is the training data among the K training data with the greatest similarity to the problem data of the specific dimension, and the similarity with the problem data of the specific dimension is greater than the set similarity threshold, and K is a positive integer.

[0154] In a specific dimension, the more first training data corresponding to each question data, the richer the content of the training data in that specific dimension, and the greater the richness value. In this embodiment, the richness is always less than or equal to the question answering rate. When the richness is closer to the question answering rate, it means that the richness of the training data collected in that specific dimension is higher, and vice versa. In this specific dimension, if the ratio between the first data and the second data and the difference between the question answering rate is greater than the set difference threshold, it is necessary to remind the relevant staff to supplement the data corresponding to the specific dimension.

[0155] In the above embodiments, when the training data is not rich enough, relevant staff can be reminded to supplement the training data, thereby improving the model training effect and training efficiency.

[0156] In a specific embodiment, a richness threshold and a question answering rate threshold can be set for a specific dimension. When the question answering rate in the training data for questions under a specific dimension is greater than the question answering rate threshold, and the difference between the richness and the question answering rate is less than the richness threshold, it means that under the specific dimension, the training data meets the training requirements.

[0157] When the question answering rate in the training data is less than or equal to the question answering rate threshold, and / or the difference between the richness and the question answering rate is greater than or equal to the richness threshold, it means that the training data does not meet the training requirements in this specific dimension, and the staff can be reminded to supplement the training data in this specific dimension.

[0158] After the training data is supplemented, the question answering rate and richness can be retested according to the description of the above embodiment until the question answering rate and richness meet the training requirements, and the training data can be used to train the model.

[0159] Corresponding to the above-mentioned training data evaluation method, the present application embodiment also discloses a training data evaluation device, see Figure 5 As shown, the device includes:

[0160] A calculation module 100 is used to calculate the similarity between the pre-collected question data and the training data;

[0161] A first determination module 110 is configured to determine that the question data is successfully solved if there is at least one target training data in the training data and the similarity between the target training data and the question data is greater than a set similarity threshold;

[0162] The second determination module 120 is used to generate and output first supplementary data indication information of the training data based on all the question data that failed to be answered if the question answering rate of all the question data is less than the set question answering rate threshold; the question answering rate represents the proportion of the number of question data that have been successfully answered in all the question data.

[0163] As an optional implementation, another embodiment of the present application discloses that all question data includes question data of multiple dimensions, and the dimensions are determined according to the domain of the question;

[0164] The calculation module 100 of the above embodiment, when calculating the similarity between the pre-collected question data and the training data, is specifically used to: calculate the similarity between the question data of a specific dimension and the training data;

[0165] The second determination module 120 of the above embodiment is specifically used to generate and output the first supplementary data indication information of the training data based on all the question data that failed to be answered if the question answering rate of all the question data is less than the set question answering rate threshold of the specific dimension, and is used to generate and output the first supplementary data indication information of the training data based on all the question data that failed to be answered if the question answering rate of all the question data of the specific dimension is less than the set question answering rate threshold of the specific dimension.

[0166] As an optional implementation, another embodiment of the present application discloses that the apparatus of the above embodiment further includes:

[0167] The collection module is used to collect question data and divide all question data into question data of multiple dimensions before calculating the similarity between the question data of a specific dimension and the training data.

[0168] As an optional implementation, another embodiment of the present application discloses that the collection module of the above embodiment, when collecting problem data, is specifically used to:

[0169] Initial question data is obtained from a setting web page; and knowledge-related question data is filtered out from the initial question data.

[0170] As an optional implementation, another embodiment of the present application discloses that the collection module of the above embodiment, when dividing all problem data into problem data of multiple dimensions, is specifically used to:

[0171] All question data are input into a pre-trained dimensional classification model so that the classification model divides all question data into question data of multiple preset dimensions; and / or, all question data are clustered to determine question data belonging to the same category as question data of the same dimension.

[0172] As an optional implementation, another embodiment of the present application discloses that the apparatus of the above embodiment further includes:

[0173] The third determination module is used to detect the number of target training data corresponding to the problem data in the training data; if the number of target training data is greater than a set first target training data number threshold, the training data is determined to be valid training data for the problem data; if the number of target training data is less than a set second target training data number threshold, the training data is determined to be invalid training data for the problem data.

[0174] As an optional implementation, another embodiment of the present application discloses that the calculation module 100 of the above embodiment, when calculating the similarity between the pre-collected question data and the training data, is specifically used to:

[0175] Calculate the dot product similarity between the first problem data and each training data; select K training data with the highest dot product similarity to the first problem data from the training data to form a training data set; where K is a positive integer; splice each training data in the training data set with the first problem data to obtain spliced data, and calculate the dot product similarity between each spliced data and the first problem data.

[0176] As an optional implementation, another embodiment of the present application discloses that the apparatus of the above embodiment further includes:

[0177] The sampling module is used to divide all question data into question data of multiple dimensions before calculating the point product similarity between the first question data and each training data; and sample the first question data from all question data of a specific dimension.

[0178] As an optional implementation, another embodiment of the present application discloses that the apparatus of the above embodiment further includes:

[0179] The fourth determination module is used to calculate the ratio between the first data and the second data; the first data is the sum of the number of first training data corresponding to each problem data, and the second data is the product of the number of pre-collected problem data and K; the first training data corresponding to the problem data is the training data with the greatest similarity to the problem data among the K training data, and the similarity to the problem data is greater than the set similarity threshold, K is a positive integer; if the difference between the ratio and the problem answering rate is greater than the set difference threshold, then based on all the second problem data, second supplementary data indication information of the training data is generated and output; the second problem data is the problem data whose corresponding first training data is less than the set first training data threshold.

[0180] Specifically, for the specific working contents of each unit of the above-mentioned training data evaluation device, please refer to the contents of the above-mentioned method embodiment, which will not be repeated here.

[0181] An embodiment of the present application further provides a control device, which includes a processor and an interface circuit. The processor in the control device is connected to an input and output component through the interface circuit of the control device.

[0182] The input and output components specifically refer to hardware components that enable users to input information and output information to users, such as microphones, keyboards, handwriting tablets, touch screens, displays, speakers, printers, etc.

[0183] The above-mentioned interface circuit can be any interface circuit that can realize data communication function, for example, it can be a USB interface circuit, a Type-C interface circuit, a serial port circuit, a PCIE circuit, etc.

[0184] The processor in the control device is a circuit with signal processing capability, which executes any one of the training data evaluation methods introduced in the above embodiments.

[0185] When the control device is applied to a device with human-computer interaction function, the input and output components of the control device can be the input components and output components on the device, such as a microphone, keyboard, handwriting tablet, touch screen, display, audio player, etc. At the same time, the processor of the control device can be the CPU or GPU of the device, etc., and the interface circuit of the control device can be the interface circuit between the information input component of the device and the processor such as the CPU or GPU.

[0186] Corresponding to the above-mentioned training data evaluation method, the present application embodiment also discloses an electronic device, see Figure 6 As shown, the electronic device includes:

[0187] Memory 200 and processor 210;

[0188] The memory 200 is connected to the processor 210 and is used to store programs;

[0189] The processor 210 is configured to implement the training data evaluation method disclosed in any of the above embodiments by running the program stored in the memory 200 .

[0190] Specifically, the electronic device may further include: a bus, a communication interface 220 , an input device 230 and an output device 240 .

[0191] The processor 210, the memory 200, the communication interface 220, the input device 230 and the output device 240 are interconnected via a bus.

[0192] A bus may include a pathway that transfers information between components of a computer system.

[0193] Processor 210 can be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, or the like, or an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the present application. It can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component.

[0194] The processor 210 may include a main processor, and may also include a baseband chip, a modem, and the like.

[0195] The memory 200 stores a program for executing the technical solution of the present application, and may also store an operating system and other key services. Specifically, the program may include program code, and the program code includes computer operating instructions. More specifically, the memory 200 may include a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM), other types of dynamic storage devices that can store information and instructions, disk storage, flash, etc.

[0196] The input device 230 may include a device for receiving data and information input by a user, such as a keyboard, a mouse, a camera, a scanner, a light pen, a voice input device, a touch screen, a pedometer, or a gravity sensor.

[0197] Output device 240 may include devices that allow information to be output to a user, such as a display screen, printer, speakers, etc.

[0198] The communication interface 220 may include any device such as a transceiver to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0199] The processor 210 executes the program stored in the memory 200 and calls other devices, which can be used to implement each step of the training data evaluation method provided in the above embodiment of the present application.

[0200] In addition to the above methods and devices, embodiments of the present application may also be computer program products, which include computer program instructions. When the computer program instructions are executed by a processor, the processor executes the various steps of the training data evaluation method provided in the above embodiments.

[0201] The computer program product may be written in any combination of one or more programming languages to implement the program code of the embodiments of the present application, including object-oriented programming languages such as Java, C++, and conventional procedural programming languages such as C or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0202] In addition, an embodiment of the present application may also be a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the processor executes each step of the training data evaluation method provided in the above embodiment.

[0203] The computer-readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0204] For the sake of simplicity, the aforementioned method embodiments are described as a series of action combinations. However, those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required for this application.

[0205] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between the various embodiments can be referred to in conjunction with each other. For device embodiments, since they are generally similar to method embodiments, their description is relatively simplified. For relevant parts, refer to the description of the method embodiments.

[0206] The steps in the methods of each embodiment of the present application can be adjusted in sequence, merged, and deleted according to actual needs, and the technical features recorded in each embodiment can be replaced or combined.

[0207] The modules and sub-modules in the devices and terminals in the various embodiments of the present application can be merged, divided, and deleted according to actual needs.

[0208] In the several embodiments provided in this application, it should be understood that the disclosed terminals, devices, and methods can be implemented in other ways. For example, the terminal embodiments described above are merely illustrative. For example, the division of modules or submodules is merely a logical function division. In actual implementation, there may be other division methods, such as multiple submodules or modules can be combined or integrated into another module, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or module, which can be electrical, mechanical or other forms.

[0209] The modules or submodules described as separate components may or may not be physically separate, and the components of the modules or submodules may or may not be physical modules or submodules, that is, they may be located in one place or distributed across multiple network modules or submodules. Some or all of the modules or submodules may be selected to achieve the purpose of this embodiment according to actual needs.

[0210] In addition, each functional module or submodule in each embodiment of the present application may be integrated into a processing module, or each module or submodule may exist physically separately, or two or more modules or submodules may be integrated into a single module. The above-mentioned integrated modules or submodules may be implemented in the form of hardware or software functional modules or submodules.

[0211] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0212] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, software units executed by a processor, or a combination of the two. The software units may be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art.

[0213] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus comprising the element.

[0214] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for evaluating training data, characterized in that: include: Calculating similarity between pre-collected question data and training data, where the training data refers to text data used to train a language model, where the language model is used to process a natural language task, including at least one of a question-answering task and a conversation task; If there is at least one target training data in the training data, and the similarity between the target training data and the question data is greater than a set similarity threshold, then it is determined that the question data is successfully answered; If the question answering rate of all question data is less than the set question answering rate threshold, then the first supplementary data indication information of the training data is generated and output based on all question data that have failed to be answered. If the question answering rate of all question data is greater than or equal to the question answering rate threshold, then the training data is determined to be complete data; the question answering rate represents the proportion of the number of question data that have been successfully answered in all question data.

2. The method according to claim 1, characterized in that All question data include question data of multiple dimensions, where the dimensions are determined according to the domain of the question; The calculating of the similarity between the pre-collected question data and the training data includes: Calculating the similarity between the problem data of a specific dimension and the training data; If the question answering rate of all question data is less than a set question answering rate threshold, generating and outputting first supplementary data indication information of the training data based on all question data that failed to be answered, including: If the question answering rate of all question data of a specific dimension is less than the set question answering rate threshold of the specific dimension, the first supplementary data indication information of the training data is generated and output based on all question data of the specific dimension that failed to be answered.

3. The method according to claim 2, characterized in that Before calculating the similarity between the problem data and the training data in a specific dimension, the method further includes: Collect problem data and divide all problem data into problem data of multiple dimensions.

4. The method according to claim 3, characterized in that The collected problem data includes: Get the initial question data from the settings web page; Filter out knowledge-related question data from the initial question data.

5. The method according to claim 3, characterized in that The problem data is divided into multiple dimensions, including: All question data are input into a pre-trained dimensional classification model so that the classification model divides all question data into question data of multiple preset dimensions; and / or, all question data are clustered to determine question data belonging to the same category as question data of the same dimension.

6. The method according to claim 1, characterized in that Also includes: Detecting the number of target training data corresponding to the problem data in the training data; If the amount of the target training data is greater than a set first target training data amount threshold, determining that the training data is valid training data for the problem data; If the amount of the target training data is less than a set second target training data amount threshold, the training data is determined to be invalid training data for the problem data.

7. The method according to claim 1, characterized in that The calculating of the similarity between the pre-collected question data and the training data includes: Calculate the dot product similarity between the first question data and each training data; From the training data, select K pieces of training data with the highest dot product similarity to the first problem data to form a training data set; where K is a positive integer; Each piece of training data in the training data set is spliced with the first question data to obtain spliced data, and a dot product similarity between each piece of spliced data and the first question data is calculated.

8. The method according to claim 7, characterized in that Before calculating the dot product similarity between the first question data and each piece of training data, the method further includes: Divide all problem data into problem data of multiple dimensions; The first question data is sampled from all question data of a specific dimension.

9. The method according to claim 1, characterized in that Also includes: calculating a ratio between the first data and the second data; The first data is the sum of the number of first training data corresponding to each question data, and the second data is the product of the number of pre-collected question data and K; the first training data corresponding to the question data is the training data with the greatest similarity to the question data among the K training data, and the similarity to the question data is greater than a set similarity threshold, where K is a positive integer; If the difference between the ratio and the question answering rate is greater than a set difference threshold, generating and outputting second supplementary data indication information of the training data based on all the second question data; The second problem data is problem data whose corresponding first training data is smaller than a set first training data threshold.

10. A training data evaluation device, characterized in that: include: a calculation module for calculating the similarity between pre-collected question data and training data, where the training data refers to text data used to train a language model used to process natural language tasks, including at least one of a question-answering task and a dialogue task; A first determination module is configured to determine that the question data is successfully answered if there is at least one target training data in the training data and the similarity between the target training data and the question data is greater than a set similarity threshold; The second determination module is used to generate and output the first supplementary data indication information of the training data based on all the question data that failed to be answered if the question answering rate of all the question data is less than the set question answering rate threshold; if the question answering rate of all the question data is greater than or equal to the question answering rate threshold, the training data is determined to be complete data; the question answering rate represents the proportion of the number of question data that have been successfully answered in all the question data.

11. An electronic device, characterized in that: include: memory and processor; Wherein, the memory is used to store programs; The processor is configured to implement the method according to any one of claims 1 to 9 by running the program in the memory.

12. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images